Are clinicians correct when they believe they are correct? Implications for medical decision support.

The process of clinical decision support is linked to the validity of clinicians' confidence in their judgments. Clinicians who are appropriately confident-highly confident when they are correct and less confident when they are incorrect-will access computer-based and other information resources only when they are needed. Clinicians who are consistently underconfident will rely on external resources when they are not needed. Those who are overconfident, who believe they are correct when in fact they are not, will be prone to medical errors. An extensive literature indicates a general tendency toward overconfidence in human judgment. This study explores the relationship between confidence and "correctness", across three levels of clinical experience, in the task domain of diagnosis in internal medicine. We created detailed synopses of 36 diagnostically challenging cases and divided them into four equivalent sets of nine cases each. We asked 216 subjects at three experience levels (72 senior medical students, 72 senior medical residents, and 72 faculty attendings) to generate a differential diagnosis for each of the nine cases in one randomly-assigned set, and simultaneously to indicate their level of confidence in each of their diagnoses. We then examined the relationship between the correctness of these diagnoses (the appearance of the correct diagnosis anywhere in the hypothesis list) and these confidence judgments, for all subjects and separately for subjects at each experience level. Results indicate a small but statistically significant relationship associating correctness with higher confidence for all subjects (Kendall's tau b =.-106;p <.0001). This statistical relationship is strongest for the students ( tau b =.-121;p <.001), somewhat lesser but still significant for the faculty-level attendings ( tau b =.-103;p <.005), and non-significant ( tau b =.-041 ) for the residents. (The negative correlations are a coding artifact.) Subjects in this study showed a tendency toward underconfidence: they had low confidence in correct diagnoses more often than they had high confidence when wrong. Nonetheless, they were overconfident and thus "error prone" for 17% of cases overall. The medical students were possibly overmatched by the difficulty of the cases, so their concordance between confidence and correctness may have resulted from an awareness that they were often guessing. The relatively low concordance seen in the residents and attendings makes a strong argument that decision support systems to reduce medical errors should include both "push" and "pull" models. In sum, these results indicate that medical decision support systems cannot rely exclusively on clinicians' perceptions of their information needs, as such perceptions will frequently be incorrect.