Advancing the objective structured clinical examination: sequential testing in theory and practice.

Context: Models of short-term remediation for failing students are typically associated with improvements in candidate performance at retest. However, the process is costly to deliver, particularly for performance retests with objective structured clinical examinations (OSCEs), and there is increasing evidence that these traditional models are associated with the longitudinal underperformance of candidates.

Methods: Rather than a traditional OSCE model, sequential testing involves a shorter 'screening' format, with an additional 'sequential' test for candidates who fail to meet the screening standard. For those tested twice, overall pass/fail decisions are then based on results on the full sequence of tests. In this study, the impacts of sequential assessment on student performance, cost of assessment delivery and overall reliability were modelled using data sourced from a final graduating OSCE in an undergraduate medical degree programme.

Results: Initial modelling using pre-existing OSCE data predicted significant improvements in reliability in the critical area, reflected in pilot results: 13.5% of students (n = 228) were required to sit the sequential OSCE. One student (0.4%) was identified as representing a false positive result (i.e. under the previous system this student would have passed the OSCE but failed on extended testing). Nine students (3.9%) who would have required OSCE retests under the prior system passed the full sequence and were therefore able to graduate at the normal time without loss of earnings. Overall reliability was estimated as 0.79 for the full test sequence. Significant cost savings were realised.

Conclusions: Sequential testing in OSCEs increases reliability for borderline students because the increased number of observations implies that 'observed' student marks are closer to 'true' marks. However, the station-level quality of the assessment needs to be sufficiently high for the full benefits in terms of reliability to be achieved. The introduction of such a system has financial benefits, good validity inferences and has proved acceptable to students and other stakeholders.