Background: ChatGPT is a novel tool that allows people to engage in conversations with an advanced machine learning model. ChatGPT's performance in the US Medical Licensing Examination is comparable with a successful candidate's performance. However, its performance in the nephrology field remains undetermined. This study assessed ChatGPT's capabilities in answering nephrology test questions.
Methods: Questions sourced from Nephrology Self-Assessment Program and Kidney Self-Assessment Program were used, each with multiple-choice single-answer questions. Questions containing visual elements were excluded. Each question bank was run twice using GPT-3.5 and GPT-4. Total accuracy rate, defined as the percentage of correct answers obtained by ChatGPT in either the first or second run, and the total concordance, defined as the percentage of identical answers provided by ChatGPT during both runs, regardless of their correctness, were used to assess its performance.
Results: A comprehensive assessment was conducted on a set of 975 questions, comprising 508 questions from Nephrology Self-Assessment Program and 467 from Kidney Self-Assessment Program. GPT-3.5 resulted in a total accuracy rate of 51%. Notably, the employment of Nephrology Self-Assessment Program yielded a higher accuracy rate compared with Kidney Self-Assessment Program (58% versus 44%; P < 0.001). The total concordance rate across all questions was 78%, with correct answers exhibiting a higher concordance rate (84%) compared with incorrect answers (73%) ( P < 0.001). When examining various nephrology subfields, the total accuracy rates were relatively lower in electrolyte and acid-base disorder, glomerular disease, and kidney-related bone and stone disorders. The total accuracy rate of GPT-4's response was 74%, higher than GPT-3.5 ( P < 0.001) but remained below the passing threshold and average scores of nephrology examinees (77%).
Conclusions: ChatGPT exhibited limitations regarding accuracy and repeatability when addressing nephrology-related questions. Variations in performance were evident across various subfields.
Download full-text PDF |
Source |
|---|---|
| http://www.ncbi.nlm.nih.gov/pmc/articles/PMC10843340 | PMC |
| http://dx.doi.org/10.2215/CJN.0000000000000330 | DOI Listing |
Publication Analysis
Top Keywords
Similar Publications
Orthopedic Resident Intern Foundations: A Cross-Institutional, Month-Long Skills Training Boot Camp for PGY-1 Residents.
J Surg Educ
January 2025
Washington University of St. Louis, Department of Orthopaedic Surgery, St. Louis, Missouri.
Objective: Orthopedic residents are tasked with rapidly acquiring clinical and surgical skills, especially during their PGY-1 year. However, resource constraints and other factors frequently cause skills training to fall short of established guidelines. We aimed to design and evaluate a cross-institutional, month-long curriculum aimed at pooling resources to optimize training.
View Article and Find Full Text PDFDevelopment and Examination of an Educational Program Combining E-Learning and Face-to-Face Training That Nurtures Inflammatory Bowel Disease Nurse Specialists.
Inflamm Intest Dis
December 2024
Faculty of Nursing and Graduate School of Nursing, Kansai Medical University, Hirakata, Japan.
Introduction: The number of patients with inflammatory bowel disease (IBD) in Japan has continued to increase, leading to diverse and complex patient backgrounds. Despite these challenges, the education of IBD nurse specialists has not kept pace with the evolving circumstances. Therefore, our research aimed to develop and validate an educational program for the training of IBD nurse specialists.
View Article and Find Full Text PDFBackground: Being overweight/having obesity is a prevalent condition not only among the general population but also among individuals with special occupations such as police officers, where fitness is often a necessity. The present study's aim was to assess how much a psychoeducational intervention based on social cognitive theory (SCT) would be helpful for encouraging weight loss behaviors among police officers.
Methods: In a randomized control trial, 102 police officers who were overweight or had obesity voluntarily registered for a weight loss program and were assigned to either an intervention or control group.
Development and Outcomes of a Provider-Driven, Online Continuing Education Program on Integrative Palliative Care: Randomized Controlled Trial.
Glob Adv Integr Med Health
January 2025
Optimal Data Analysis, Kouts, IN, USA.
Background: Integrative therapies are increasingly in demand for both symptom management and quality of life in palliative care (PC) populations. Multidisciplinary PC professionals need continuing education/continuing medical education (CE/CME) to keep current on the evidence-informed use of integrative therapies in PC planning.
Objectives: (1) Elicit input from multidisciplinary PC providers on needs for CE/CME content on integrative care, and indicators of implementation for use in impact assessment.
Effects of the Early Start Denver Model on emotional dysregulation and behavior problems in children with Autism spectrum disorder : Effects of the early start Denver model on emotional dysregulation and behavior problems in Chinese children with Autism spectrum disorder.
BMC Pediatr
January 2025
Department of Developmental and Behavioral Pediatrics, Children's Medical Center, The First Hospital of Jilin University, Changchun, China.
Background: Most previous studies have focused on the clinical efficacy after intervention of ESDM, particularly in core symptoms. However, only a few have paid attention to the effectiveness of ESDM on emotional dysregulation and behavior problems in children with ASD. This study aimed to explore the effect of the ESDM on addressing emotional dysregulation and behavior problems in children with ASD in China, as well as its correlation with core symptoms of ASD.
View Article and Find Full Text PDFWant AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!