Human Reviewers' Ability to Differentiate Human-Authored or Artificial Intelligence-Generated Medical Manuscripts: A Randomized Survey Study.

Objective: To assess the ability of humans to differentiate human-authored vs artificial intelligence (AI)-generated medical manuscripts.

Methods: This is a prospective randomized survey study from October 1, 2023, to December 1, 2023, from a single academic center. Artificial intelligence-generated medical manuscripts were created using ChatGPT 3.5 and were evaluated alongside randomly selected human-authored manuscripts. Participants, who were blinded from manuscript selection and creation, were randomized to receive three manuscripts that were either human-authored or AI-generated and had to fill out a survey questionnaire after review regarding who authored the manuscript. The primary outcome was accuracy of human reviewers in differentiating manuscript authors. Secondary outcomes were to identify factors that influenced prediction accuracy.

Results: Fifty-one physicians were included in the study, including 12 post-doctorates, 19 assistant professors, and 20 associate or full professors. The overall specificity of 55.6% (95% CI, 30.8% to 78.5%), sensitivity of 31.2% (95% CI,11.0% to 58.7%), positive predictive value of 38.5% (95% CI,13.9% to 68.4%) and negative predictive value of 47.6% (95% CI, 25.7% to 70.2%). A stratified analysis of human-authored manuscripts indicated that high-impact factor manuscripts were identified with higher accuracy than low-impact factor ones (P=.037). For individual-level data, neither academic rank nor prior manuscript review experience significantly predicted the accuracy. The frequency of AI interaction was a significant factor, with occasional (odds ratio [OR], 8.20; P=.016), fairly frequent (OR, 7.13; P=.033), and very frequent (OR, 8.36; P=.030) use associated with correct identification. Further analysis revealed no significant predictors among the papers' qualities.

Conclusion: Generative AI such as ChatGPT could create medical manuscripts that could not be differentiated from human-authored manuscripts.