Assessment and Comparison of Artificial Intelligence generated Information Regarding Shoulder Arthroplasty from Multiple Interfaces.

Purpose: This study aims to analyze and compare the quality, accuracy, and readability of information regarding anatomic Total Shoulder Arthroplasty (aTSA) and reverse Total Shoulder Arthroplasty (rTSA) provided by various AI interfaces (Open AI's ChatGPT and Microsoft's Copilot).

Methods: Thirty commonly asked questions (categorized by Rothwell criteria into Fact, Policy, and Value) by patients were inputted into ChatGPT 3.5 and Copilot. Responses were assessed with the DISCERN scale, JAMA benchmark criteria, Flesch-Kincaid Reading Ease Score (FRES), and Grade Level (FKGL). The sources of citations provided by CoPilot were further analyzed.

Results: Both AI interfaces generated DISCERN scores >50 (aTSA and rTSA ChatGPT 57 (Fact), 61 (Policy), 58 (Value), aTSA and rTSA Copilot 68 (Fact), 72 (Policy), 70 (Value)), demonstrating "good" quality of information provided, except for the Policy questions by CoPilot which were scored as "excellent" (>70). CoPilot's higher JAMA score (3 vs. 0) and FRES scores above 30 indicated more reliable, accessible responses, which required a minimum of 12th grade education to read the same. In comparison, the ChatGPT generated more complex texts, with the majority of the FRES scores <20, and FKGL score signifying complexity of academic level text. Finally, CoPilot provided citations and demonstrated the highest percentage of Academic sources (31.1% for rTSA and 26.7% for aTSA) suggesting reliable sources of information.

Conclusion: Overall, the information provided by both AI interfaces ChatGPT and CoPilot was scored as a "good" source of information for commonly asked patient questions regarding shoulder arthroplasty. But the answers to questions pertaining to shoulder arthroplasty provided by CoPilot proved to be more reliable (p=0.0061), less complex, easier to read (p=0.0031), and referenced information from reliable resources including Academic sources, Journal articles, and Medical sites. Although answers provided by CoPilot were "easier" to read, they still required a 12th grade education, which may be too complex for most patients, posing a challenge for patient comprehension. There were a substantial amount of non-medical media sites, and Commercial sources that were cited for both aTSA and rTSA questions by CoPilot. Critically, answers from both AI interfaces should serve as supplementary resources rather than primary sources on perioperative conditions pertaining to shoulder arthroplasty.