Diabetes healthcare quality report cards: how accurate are the grades?

Objective: To evaluate the accuracy and precision of random sampling in identifying healthcare system outliers in diabetes performance measures.

Study Design: Cross-sectional analysis of 79 Veterans Health Administration facilities serving 250 317 patients with diabetes mellitus between October 1, 1999, and September 30, 2000.

Methods: Primary outcome measures were poor glycosylated hemoglobin (A1C) control and good low-density lipoprotein cholesterol (LDL-C) and blood pressure (BP) control. Facility performance for each measure was calculated using 150 separate random samples and was compared with results using the bootstrap method as the criterion standard for determining outlier status (defined as a >/=5% difference from the mean, within the 10th or 90th percentile, or >/=2 SDs from the mean).

Results: The study population was largely male (97.4%), with 54.0% of subjects being 65 years or older. The facility-level mean performances were 22.8% for poor A1C control, 53.1% for good LDL-C control, and 55.3% for good BP control. Comparing the random sampling method with the bootstrap method, the sensitivity ranged between 0.64 and 0.83 for the 3 outcome measures, positive predictive values ranged between 0.55 and 0.88, and specificity and negative predictive values ranged between 0.88 and 0.99.

Conclusions: The specificity and negative predictive value of the random sampling method in identifying nonoutliers in performance were generally high, while its sensitivity and positive predictive value were moderate. The use of random sampling to determine performance for individual outcome measures may be most appropriate for internal quality improvement rather than for public reporting.