Predicting Performance on Clerkship Examinations and USMLE Step 1: What Is the Value of Open-Ended Question Examination?

Purpose: Despite advances in learning sciences that highlight the efficacy of elaborative interrogation, in which students explain and elaborate on concepts in their own words, assessment techniques in medical education have commonly employed multiple-choice questions (MCQs). Educators' reluctance to consider alternatives such as open-ended questions (OEQs) stems from practical advantages of MCQs and the lack of empirical data on the predictability of OEQs for performance on other high-stakes assessments. In this study, the authors compared the predictive value of preclerkship assessments using OEQs for the outcomes of clerkship examinations and United States Medical Licensing Examination (USMLE) Step 1.

Method: The authors compared outcomes of 2 assessment formats using multiyear performance data (2015 and 2016 cohorts) on preclerkship MCQ versus OEQ examinations for predicting students' subsequent performance on 6 clerkship examinations and USMLE Step 1. The authors conducted a regression analysis to compare the predictability of MCQs and OEQs by using clerkship exam scores and Step 1 scores as dependent variables and performance on MCQs and OEQs as predictors in the models.

Results: Regression models with OEQs were consistently higher for predicting clerkship exam (NBME shelf-exam) scores, except for one clerkship, compared with models using MCQs. For Step 1, R-square using MCQs was higher with 59% of the variance explained compared with 46% with OEQs, but the OEQ cohort scored significantly higher on Step 1.

Conclusions: OEQ examinations predict performance on subsequent high-stakes MCQ examinations. Given the predictive value and closer alignment with scientific principles of effective learning, OEQ examinations are an examination format worthy of consideration in preclerkship medical education programs.