Objective: To analyze the accuracy of ChatGPT-generated responses to common rhinologic patient questions.
Methods: Ten common questions from rhinology patients were compiled by a panel of 4 rhinology fellowship-trained surgeons based on clinical patient experience. This panel (Panel 1) developed consensus "expert" responses to each question. Questions were individually posed to ChatGPT (version 3.5) and its responses recorded. ChatGPT-generated responses were individually graded by Panel 1 on a scale of 0 (incorrect) to 3 (correct and exceeding the quality of expert responses). A 2nd panel was given the consensus and ChatGPT responses to each question and asked to guess which response corresponded to which source. They then graded ChatGPT responses using the same criteria as Panel 1. Question-specific and overall mean grades for ChatGPT responses, as well as interclass correlation coefficient (ICC) as a measure of interrater reliability, were calculated.
Results: The overall mean grade for ChatGPT responses was 1.65/3. For 2 out of 10 questions, ChatGPT responses were equal to or better than expert responses. However, for 8 out of 10 questions, ChatGPT provided responses that were incorrect, false, or incomplete based on mean rater grades. Overall ICC was 0.526, indicating moderate reliability among raters of ChatGPT responses. Reviewers were able to discern ChatGPT from human responses with 97.5% accuracy.
Conclusion: This preliminary study demonstrates overall near-complete and variably accurate responses provided by ChatGPT to common rhinologic questions, demonstrating important limitations in nuanced subspecialty fields.
Download full-text PDF |
Source |
|---|---|
| http://www.ncbi.nlm.nih.gov/pmc/articles/PMC11705442 | PMC |
| http://dx.doi.org/10.1002/oto2.70065 | DOI Listing |
Publication Analysis
Top Keywords
Similar Publications
Dementia Care Research and Psychosocial Factors.
Alzheimers Dement
December 2024
Michigan Alzheimer's Disease Research Center, Ann Arbor, MI, USA.
Background: The integration of Large Language Models (LLMs) like ChatGPT-4 in clinical settings offers potential enhancements in medical practice, particularly in neurology and dementia care. There is rising public usage of ChatGPT-4 for preliminary information gathering. This study aims to evaluate the effectiveness of ChatGPT-4 in responding to neurology-focused queries, with an emphasis on Alzheimer's Disease (AD).
View Article and Find Full Text PDFImproving readability in AI-generated medical information on fragility fractures: the role of prompt wording on ChatGPT's responses.
Osteoporos Int
January 2025
Division of Occupational Therapy, Faculty of Health Sciences, Kutahya Health Sciences University, Kutahya, Turkey.
Unlabelled: Understanding how the questions used when interacting with chatbots impact the readability of the generated text is essential for effective health communication. Using descriptive queries instead of just keywords during interaction with ChatGPT results in more readable and understandable answers about fragility fractures.
Purpose: Large language models like ChatGPT can enhance patients' understanding of medical information, making health decisions more accessible.
Objective: To analyze the accuracy of ChatGPT-generated responses to common rhinologic patient questions.
Methods: Ten common questions from rhinology patients were compiled by a panel of 4 rhinology fellowship-trained surgeons based on clinical patient experience. This panel (Panel 1) developed consensus "expert" responses to each question.
Purpose: Caregivers in pediatric oncology need accurate and understandable information about their child's condition, treatment, and side effects. This study assesses the performance of publicly accessible large language model (LLM)-supported tools in providing valuable and reliable information to caregivers of children with cancer.
Methods: In this cross-sectional study, we evaluated the performance of the four LLM-supported tools-ChatGPT (GPT-4), Google Bard (Gemini Pro), Microsoft Bing Chat, and Google SGE-against a set of frequently asked questions (FAQs) derived from the Children's Oncology Group Family Handbook and expert input (In total, 26 FAQs and 104 generated responses).
Want AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!