Objective: To determine if reviewer experience impacts the ability to discriminate between human-written and ChatGPT-written abstracts.
Methods: Thirty reviewers (10 seniors, 10 juniors, and 10 residents) were asked to differentiate between 10 ChatGPT-written and 10 human-written (fabricated) abstracts. For the study, 10 gynecologic oncology abstracts were fabricated by the authors. For each human-written abstract we generated a ChatGPT matching abstract by using the same title and the fabricated results of each of the human generated abstracts. A web-based questionnaire was used to gather demographic data and to record the reviewers' evaluation of the 20 abstracts. Comparative statistics and multivariable regression were used to identify factors associated with a higher correct identification rate.
Results: The 30 reviewers discriminated 20 abstracts, giving a total of 600 abstract evaluations. The reviewers were able to correctly identify 300/600 (50%) of the abstracts: 139/300 (46.3%) of the ChatGPT-generated abstracts and 161/300 (53.7%) of the human-written abstracts (p=0.07). Human-written abstracts had a higher rate of correct identification (median (IQR) 56.7% (49.2-64.1%) vs 45.0% (43.2-48.3%), p=0.023). Senior reviewers had a higher correct identification rate (60%) than junior reviewers and residents (45% each; p=0.043 and p=0.002, respectively). In a linear regression model including the experience level of the reviewers, familiarity with artificial intelligence (AI) and the country in which the majority of medical training was achieved (English speaking vs non-English speaking), the experience of the reviewer (β=10.2 (95% CI 1.8 to 18.7)) and familiarity with AI (β=7.78 (95% CI 0.6 to 15.0)) were independently associated with the correct identification rate (p=0.019 and p=0.035, respectively). In a correlation analysis the number of publications by the reviewer was positively correlated with the correct identification rate (r28)=0.61, p<0.001.
Conclusion: A total of 46.3% of abstracts written by ChatGPT were detected by reviewers. The correct identification rate increased with reviewer and publication experience.
Download full-text PDF |
Source |
|---|---|
| http://dx.doi.org/10.1136/ijgc-2023-005162 | DOI Listing |
Publication Analysis
Top Keywords
Similar Publications
A proposal of a fecal scoring system based on physicochemical analyses of suckling pig feces.
Porcine Health Manag
January 2025
Departament de Sanitat i Anatomia Animals, Facultat de Veterinària, Campus de la Universitat Autònoma de Barcelona (UAB), 08193, Bellaterra, Barcelona, Spain.
Background: Digestive disorders are one of the main health problems in suckling piglets. The correct visual identification of feces in suckling piglets is an important tool for the diagnosis of enteric diseases. The aim of the present observational study was to analyze different physicochemical parameters of the feces of suckling piglets aged 0 to 21 days: visual appearance (color and consistency), fecal dry matter (FDM) content and pH.
View Article and Find Full Text PDFA scaling law to model the effectiveness of identification techniques.
Nat Commun
January 2025
Data Science Institute, Imperial College London, London, UK.
AI techniques are increasingly being used to identify individuals both offline and online. However, quantifying their effectiveness at scale and, by extension, the risks they pose remains a significant challenge. Here, we propose a two-parameter Bayesian model for exact matching techniques and derive an analytical expression for correctness (κ), the fraction of people accurately identified in a population.
View Article and Find Full Text PDF[Management of risks linked to the sterilization of reusable medical devices in dentistry].
Ann Pharm Fr
January 2025
Service de la pharmacie, pharmacologie et pharmacotechnie hospitalière. Hôpital militaire principal d'instruction de Tunis, Tunisia; Faculté de pharmacie de Monastir. Université de Monastir, Tunisia.
Objective: The aim of this study was to analyze the risks associated with the sterilization process for reusable medical devices (RMD) in stomatology, by applying the FMECA method, with a view to implementing the necessary corrective and preventive actions necessary to secure this process.
Methods: The study, which was descriptive, took place between June and July 2024 in the medicine and dental surgery department of our hospital and concerned the moist heat sterilization process of RMD. The study began by defining its scope and the formation of the work team, followed by the functional analysis of the process, the identification of the failure modes (FM), the definition of the rating scales, the rating of the FM and finally the calculation of the criticality index and the development of the action plan.
Improving the Depth and Reliability of Glycopeptide Identification Using Protein Prospector.
Mol Cell Proteomics
January 2025
Department of Pharmaceutical Chemistry, University of California, San Francisco.
Glycosylation is the most common and diverse modification of proteins. It can affect protein function and stability and is associated with many diseases. While proteomic methods to study most post-translational modifications are now quite mature, glycopeptide analysis is still a challenge, particularly from the aspect of data analysis.
View Article and Find Full Text PDFThe acquisition of chopstick skills is considered essential for child development and etiquette in many Asian cultures. However, a decline in chopstick education has been observed in Japan, and the underlying causes of this phenomenon remain elusive. This study aims to investigate children's chopstick skills and develop an objective method to evaluate them using a hand posture estimation model.
View Article and Find Full Text PDFWant AI Summaries of new PubMed Abstracts delivered to your In-box?
Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!