Assessment of diagnostic skills in specialist examinations should include lessons learnt from misdiagnosis.

Background: Poor authenticity in high stake clinical exams adversely effects validity. We propose including known misleading diagnostic factors and contextual biases in the assessment of diagnostic skills amongst advanced specialty trainees. We hypothesise that this strategy offers a more realistic and critical assessment of diagnostic skill than strategies in which candidates are presented with directive, bias free information, allowing for assumptions which cannot be made in real life.

Methods: Eleven patient based practice clinical exam stations were presented to nine advanced ophthalmology trainees. Four patients had a history of misdiagnosis or near misdiagnosis of key ophthalmic findings, presumed to result from identifiable biases and misleading information. In those four stations, candidates were presented with authentic, file based information and were asked authentic questions, similar to those with which the patients presented. If the candidates were unsuccessful in identifying key findings, the questions were converted into directive questions about the same key findings (i.e. "examine the patient's eyelids, what is your diagnosis?"), and the candidates re-assessed the patient and re-answered.

Results: Ninety-eight doctor-patient encounters took place. Of those, 35 encounters were analysed for the purpose of this study. In 63% of those encounters, key findings were missed when the question included authentic biases or misleading background information, but rephrasing the question to a directive exam format led to their correct identification (Fail converted to pass). Key findings were detected despite contextual biases or misleading background information in only 23% of encounters. In 14% the findings were missed with either question phrasing.

Conclusions: Presentation authentic questions provide a more realistic and less forgiving measure of diagnostic skills than directive exam questions. Given the prevalence of diagnostic errors and their importance to patient outcomes, known mechanisms contributing to diagnostic errors should be used as one of the assessment tools of advanced speciality trainees.