Assessment of inter-rater reliability of screening tools to identify patients at risk of medication-related problems across the emergency department continuum of care.

Background: Following a national multicentre study, two emergency department (ED) screening tools were developed to determine risk of medication-related problems; one for use at ED presentation and another at ED discharge to the community. This study aimed to determine the inter-rater reliability amongst ED health professionals when applying these screening tools to a series of case scenarios.

Methods: A prospective, cross-sectional study was undertaken in the ED of a major metropolitan hospital. Twelve case scenarios were developed following ED observation of a range of patients, which were incorporated into a questionnaire and distributed to 50 health professionals. Inter-rater reliabilities of each explanatory variable of the screening tools and overall assessment were calculated using Fleiss' multi-rater kappa.

Results: The questionnaire was completed by 15 doctors, 19 nurses and 16 pharmacists. Fleiss' kappa showed an overall inter-rater reliability for the ED presentation tool of 0.83 (95% CI 0.83-0.84), indicating near perfect agreement. Fleiss' kappa for the ED discharge tool was 0.83 (95% CI 0.83-0.85), which also showed near perfect agreement.

Conclusions: The screening tools produced favourable inter-rater reliability amongst ED health professionals. These results have important implications for ensuring consistency of ED decision-making in screening patients at risk of developing medication-related problems.