Objectives: As a large language model (LLM) trained on a large data set, ChatGPT can perform a wide array of tasks without additional training. We evaluated the performance of ChatGPT on postgraduate UK medical examinations through a systematic literature review of ChatGPT's performance in UK postgraduate medical assessments and its performance on Member of Royal College of Physicians (MRCP) Part 1 examination.
Methods: Medline, Embase and Cochrane databases were searched. Articles discussing the performance of ChatGPT in UK postgraduate medical examinations were included in the systematic review. Information was extracted on exam performance including percentage scores and pass/fail rates. MRCP UK Part 1 sample paper questions were inserted into ChatGPT-3.5 and -4 four times each and the scores marked against the correct answers provided.
Results: 12 studies were ultimately included in the systematic literature review. ChatGPT-3.5 scored 66.4% and ChatGPT-4 scored 84.8% on MRCP Part 1 sample paper, which is 4.4% and 22.8% above the historical pass mark respectively. Both ChatGPT-3.5 and -4 performance was significantly above the historical pass mark for MRCP Part 1, indicating they would likely pass this examination. ChatGPT-3.5 failed eight out of nine postgraduate exams it performed with an average percentage of 5.0% below the pass mark. ChatGPT-4 passed nine out of eleven postgraduate exams it performed with an average percentage of 13.56% above the pass mark. ChatGPT-4 performance was significantly better than ChatGPT-3.5 in all examinations that both models were tested on.
Conclusion: ChatGPT-4 performed at above passing level for the majority of UK postgraduate medical examinations it was tested on. ChatGPT is prone to hallucinations, fabrications and reduced explanation accuracy which could limit its potential as a learning tool. The potential for these errors is an inherent part of LLMs and may always be a limitation for medical applications of ChatGPT.
Download full-text PDF |
Source |
|---|---|
| http://www.ncbi.nlm.nih.gov/pmc/articles/PMC11290618 | PMC |
| http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0307372 | PLOS |
Publication Analysis
Top Keywords
Similar Publications
Microbiome characterization of patients with Crohn disease and the use of fecal microbiota transplantation: A review.
Medicine (Baltimore)
January 2025
Department of Gastroenterology, The Second Affiliated Hospital of Hainan Medical University, Haikou, China.
Inflammatory bowel disease is a chronic inflammatory condition predominantly affecting the intestines, encompassing both ulcerative colitis and Crohn disease (CD). As one of the most common gastrointestinal disorders, CD's pathogenesis is closely linked with the intestinal microbiota. Recently, fecal microbiota transplantation (FMT) has gained attention as a potential treatment for CD, with the effective reestablishment of intestinal microecology considered a crucial mechanism of FMT therapy.
View Article and Find Full Text PDFTravel medicine training and experience among primary care physicians in Qatar.
PLoS One
January 2025
Department of Public Health and Preventive Medicine, Faculty of Medicine Cairo University, Cairo, Egypt.
Background: Travel medicine (TM) focuses on preventing and managing travel-related issues. Evidence has become more important than expert opinions in the development of TM standards. This study aimed to evaluate the training and experience of TM among Primary Care Physicians (PCPs) in Qatar and their associated factors.
View Article and Find Full Text PDFRealist evaluation of the AKU-SONAM mentorship program.
PLoS One
January 2025
Department for Educational Development, Aga Khan University, Karachi, Pakistan.
Background & Objectives: The context, mechanism, and outcome (CMO) framework is meant to identify specific contextual factors (C) related to organizational and program structure that trigger certain mechanisms (M) involving the unique characteristics of a program, leading to specific outcomes (O). The purpose of this study was to explore the contextual underpinnings, operational processes, and resultant effects of the faculty mentorship program at AKU-SONAM. This exploration involved the context in terms of organizational culture, mechanisms examining processes such as communication between mentors and mentees, quality of relationships, the challenges encountered, and the program's adaptability to cope up while, outcomes encompassed improvements in interpersonal relationships, career advancement, and skill development.
View Article and Find Full Text PDFIschemic Myocardium Targeting Peptide-Guided Nanobubbles for Multimodal Imaging and Treatment of Coronary Microvascular Dysfunction.
Adv Healthc Mater
January 2025
Department of Ultrasound, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, 510120, China.
Coronary microvascular dysfunction (CMD) refers to clinical symptoms caused by structural and functional damage to coronary microcirculation. The timely and precise diagnosis of CMD-related myocardial ischemia is essential for improving patient prognosis. This study describes a method for the multimodal (fluorescence, ultrasonic, and photoacoustic) noninvasive imaging and treatment of CMD based on ischemic myocardium-targeting peptide (IMTP)-guided nanobubbles functionalized with indocyanine green (IMTP/ICG NBs) and characterizes their basic characteristics and in vitro imaging and targeting abilities.
View Article and Find Full Text PDFInternational Validation of a Novel PEACE Scale to Improve the Quality of Upper Gastrointestinal Mucosal Inspection During Endoscopy.
Clin Transl Gastroenterol
January 2025
Division of Gastroenterology, Kansas City VA Medical Center, Kansas City, Missouri, USA.
Introduction: The performance of a high quality esophagogastroduodenoscopy (EGD) is dependent on the mucosal cleanliness. Recently, the Polprep: Effective Assessment of Cleanliness in EGD (PEACE) scale was created to assess the degree of mucosal cleanliness during EGD. The aim of this study was to validate this scoring system in a cohort of international endoscopists.
View Article and Find Full Text PDFWant AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!