Performance of GPT-4 on Chinese Nursing Examination: Potentials for AI-Assisted Nursing Education Using Large Language Models.

Nurse Educ

School of Nursing, Capital Medical University, Beijing, China (Drs Miao, Luo, Zhao, Li, Liu, Wang, and Wu); and School of Nursing, Johns Hopkins University, Baltimore, USA (Dr Chen).

Published: October 2024

AI Article Synopsis

  • A study was conducted to evaluate GPT-4's performance in nursing examinations specific to the Chinese context, focusing on both multiple-choice and open-ended questions.
  • GPT-4 scored 71% accuracy on multiple-choice questions but showed moderate performance in open-ended questions based on factors like logical consistency and information quality.
  • The findings suggest that while GPT-4 is effective for basic knowledge questions, it has limitations with more complex queries, prompting nursing educators to consider its potential and challenges for educational use.

Article Abstract

Background: The performance of GPT-4 in nursing examinations within the Chinese context has not yet been thoroughly evaluated.

Objective: To assess the performance of GPT-4 on multiple-choice and open-ended questions derived from nursing examinations in the Chinese context.

Methods: The data sets of the Chinese National Nursing Licensure Examination spanning 2021 to 2023 were used to evaluate the accuracy of GPT-4 in multiple-choice questions. The performance of GPT-4 on open-ended questions was examined using 18 case-based questions.

Results: For multiple-choice questions, GPT-4 achieved an accuracy of 71.0% (511/720). For open-ended questions, the responses were evaluated for cosine similarity, logical consistency, and information quality, all of which were found to be at a moderate level.

Conclusion: GPT-4 performed well at addressing queries on basic knowledge. However, it has notable limitations in answering open-ended questions. Nursing educators should weigh the benefits and challenges of GPT-4 for integration into nursing education.

Download full-text PDF

Source
http://dx.doi.org/10.1097/NNE.0000000000001679DOI Listing

Publication Analysis

Top Keywords

performance gpt-4
16
open-ended questions
16
nursing education
8
nursing examinations
8
examinations chinese
8
gpt-4 multiple-choice
8
multiple-choice questions
8
nursing
7
gpt-4
7
questions
6

Similar Publications

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!