Reliability testing of a case-leveling framework for assigning level of difficulty of pharmacist's initial patient medication assessments.

Objective: To develop and test the reliability of a case-leveling framework for assigning level of difficulty of the pharmacist's task for initial medication assessments conducted by pharmacists integrated into family physician offices.

Design: Descriptive study.

Setting: Seven family practice sites in Ontario from June 2004 to July 2006.

Patients: Patients referred by their family physician for pharmacist assessment.

Intervention: Individual medication assessments, monitoring, and follow-up by pharmacists. A case-leveling framework was developed with three levels of complexity (graded as I, II, or III) including specific descriptors and practice-based examples. Reliability was assessed between two standardized assessors and between one assessor and project pharmacists. Project pharmacist feedback was elicited through an e-mail survey. Reliability is reported using the kappa statistic.

Main Outcome Measures: Reliability of a case-leveling framework and helpfulness of the framework as reported by pharmacists.

Results: 53 patient cases were evaluated for interrater reliability between standardized assessors. The mean (+/- SD) case level assigned was 1.8 +/- 0.68, and the kappa was 0.62 (95% CI 0.44-0.79), indicating a substantial strength of agreement between raters. For the second reliability test, 52 cases were rated, with a level of agreement between project pharmacists and the external assessor of 0.46 (95% CI 0.27-0.65), indicating moderate agreement. Feedback resulted in slight revisions to the original framework.

Conclusion: The case-leveling framework was a reliable method and can be used to determine the level of difficulty of patient cases in primary care.