Analyzing the Performance of ChatGPT About Osteoporosis.

Cureus

Department of Interventional Physiatry, Biruni University, Istanbul, TUR.

Published: September 2023

Introduction: This study evaluates the knowledge of ChatGPT about osteoporosis.

Methods: Osteoporosis-related frequently asked questions (FAQs) created by examining the websites frequently visited by patients, the official websites of hospitals, and social media. Questions based on these scientific data have been prepared in accordance with National Osteoporosis Guideline Group guides. Rater scored all ChatGPT answers between 1 and 4 (1 stated that the information was completely correct, 2 stated that the information was correct but insufficient, 3 stated that although some of the information was correct, there was incorrect information in the answer, and 4 stated that the answer consisted of completely incorrect information). The reproducibility of ChatGPT responses on osteoporosis was assessed by asking each question twice. The repeatability of the ChatGPT answer was considered as getting the same score twice.

Results: ChatGPT responded to 72 FAQs with an accuracy rate of 80.6%. The highest accuracy in ChatGPT's answers about osteoporosis was in the prevention category, 91.7%, and in the general knowledge category, 85.8%. Only 19 of the 31 (61.3%) questions prepared according to the National Osteoporosis Guideline Group guidelines were answered correctly by ChatGPT, and two answers (6.4%) were categorized as grade 4. The reproducibility rate of ChatGPT answers on 72 FAQs was 86.1% and the reproducibility rate of ChatGPT answers on National Osteoporosis Guideline Group guidelines was 83.9%.

Conclusion: Present study outcomes for the first time showed that ChatGPT provided adequate answers to more than 80% of FAQs about osteoporosis. However, the accuracy of ChatGPT's answers to inquiries based on National Osteoporosis Guideline Group guidelines was decreased to 61.3%.

Download full-text PDF

Source
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC10599213PMC
http://dx.doi.org/10.7759/cureus.45890DOI Listing

Publication Analysis

Top Keywords

national osteoporosis
16
osteoporosis guideline
16
guideline group
16
chatgpt answers
16
group guidelines
12
chatgpt
10
osteoporosis
8
prepared national
8
stated correct
8
accuracy chatgpt's
8

Similar Publications

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!