The Fault in Our Sets: A Mixed Methods Analysis of Clinical Value Set Errors.

Objective: To characterize clinical value set issues and identify common patterns of errors.

Materials And Methods: We conducted semi-structured interviews with 26 value set experts and performed root cause analyses of errors identified in electronic health records (EHRs). We also analyzed a random sample of user-reported issues from the Value Set Authority Center (VSAC), developing a categorization scheme for value set errors. Additionally, we audited medication value sets from three sources and assessed the impact of value set variations on a clinical quality measure within Vanderbilt's Epic system.

Results: Interviews highlighted ongoing difficulties in value set identification, creation, and maintenance, with significant consequences for clinical decision support (CDS), quality measurement, and patient care. Content analysis indicated that 42% of errors involved missing codes, 14% included extraneous codes, and 40% arose from misinterpretations of value set intent; 72% of these errors were present at creation. The audit revealed errors in 50% of medication value sets, predominantly omissions. The impact analysis demonstrated that value set selection altered a clinical quality measure's outcome by 3- to 30-fold.

Discussion: Value set errors are widespread and arise from a delineable set of causes. Characterizing patterns of errors allowed us to identify best practices and potential solutions to minimize their frequency.

Conclusion: Better tools for finding, authoring, auditing and monitoring value sets are urgently needed.