Using Clinical Vignettes to Evaluate VTE Protocol Adherence.

Background: Venous thromboembolism (VTE) prophylaxis is underutilized in hospitalized medical patients. Underutilization might occur as a result of resident practice variation incurred by using a complex risk assessment tool.

Objective: To examine what impact repetitive exposure to an electronic point-based VTE risk assessment tool has on resident inter-rater reliability and protocol adherence.

Design: Pre and post intervention cross-sectional cohort study.

Setting: Single academic center.

Patients: Convenience samples of Internal Medicine residents.

Interventions: Residents completed clinical vignettes before and after any exposure to an electronic risk assessment tool and reminder alert. They were asked to make three determinations using a point-based VTE risk assessment tool: risk stratification, identify contraindications, and VTE prevention strategy.

Measurements: Inter-rater reliability for risk assessment, contraindications, and VTE prophylaxis strategy and protocol adherence.

Results: Kappa scores for VTE risk assessment did not change, but improved for VTE plan increasing from 0.28 to 0.37. Protocol adherence improved from 71% in 2008 to 79% (P = 0.06). There was a significant decrease in under-prophylaxis (22% to 6%, P < 0.0001) but a significant increase in over-prophylaxis (7% to 16%, P = 0.001).

Conclusions: Using clinical vignettes, we determined that daily exposure to an electronic risk assessment tool did not improve the inter-rater reliability of a point-based risk assessment tool when used by medical residents. This might be due to inexperienced providers using a complex point-based tool. Overall, adherence improved, and under-prophylaxis decreased, but over-prophylaxis increased. Clinical vignettes are a generalizable method to monitor resident prophylaxis practices and way to identify educational and process improvement opportunities.

Keywords: Resident; Inter-rater reliability; Venous thromboembolism; Agreement; Risk assessment score.