ChatGPT takes on the tough US medical licensing exam
LOS ANGELES, February 12 – Dr ChatGPT will see you soon. According to a study published Thursday, the artificial intelligence system achieved passing or near-passing results on the US medical licensing exam.
Advertisements
Advertisements
“Achieving the passing mark for this notoriously difficult expert exam, and doing so without any human augmentation, is a notable milestone in clinical AI maturation,” say the authors of the study published in the journal Plos Digital Health .
“These results suggest that large language models may have the potential to assist in medical education, and potentially clinical decision making,” they said.
Capable of producing essays, poems and programming code in seconds, ChatGPT was developed by OpenAI, a California-based startup founded in 2015 with early funding from Elon Musk, among others.
Microsoft invested US$1 billion (RM4.33 billion) in OpenAI in 2019 and just signed a new multi-billion deal with the firm.
For the study, researchers at AnsibleHealth in California tested ChatGPT’s performance on a three-part licensing exam taken by medical students and physicians in training in the United States.
The standardized exam tests knowledge in various medical disciplines from basic science to biochemistry to diagnostic reasoning to bioethics.
The AI system was tested on 350 of the 376 public questions in the June 2022 version of the exam, the study said, and the chatbot received no specialized training ahead of time.
Image based questions have been removed.
ChatGPT scored between 52.4 percent and 75 percent across the three parts of the exam.
A pass rate is around 60 percent.
According to the study, the first part of the exam, which focuses on basic science and pharmacology, is typically taken by medical students who have put in 300–400 hours of dedicated study time.
The second part is usually taken by fourth year medical students and emphasizes clinical reasoning, medical management and bioethics.
The final section is for doctors who have completed at least six months to a year of postgraduate medical training.
Dr. Google and Nurse Bing
The questions were presented to ChatGPT in various formats, including open-ended prompting such as “What would be the patient’s diagnosis based on the information provided?”
There were also multiple choice questions such as: “The patient’s condition is most often caused by which of the following pathogens?”
Two physician raters blinded to each other reviewed the responses to come up with the final scores, the study said.
An outside expert, Simon McCallum, a senior lecturer in software engineering at Victoria University of Wellington, New Zealand, noted that Google has received encouraging results with an AI medical aid known as Med-PaLM.
“ChatGPT can pass the exam, but Med-PaLM is able to give advice to patients as good as a professional GP,” McCallum said. “And both of these systems are improving.
“Society is about to change, and instead of warning about the hypochondria of randomly searching the internet for symptoms, we may soon be getting our medical advice from Doctor Google or Nurse Bing.”
ChatGPT was also helpful to the authors of the medical exam study in another way.
They used the chatbot to help write it, co-author Tiffany Kung said. — ETX Studio
