Background: Behavioural marker systems are used across several healthcare disciplines to assess behavioural (non-technical) skills, but rater training is variable, and inter-rater reliability is generally poor. Inter-rater reliability provides data about the tool, but not the competence of individual raters. This study aimed to test the inter-rater reliability of a new behavioural marker system (PhaBS - pharmacists' behavioural skills) with clinically experienced faculty raters and near-peer raters. It also aimed to assess rater competence when using PhaBS after brief familiarisation, by assessing completeness, agreement with an expert rater, ability to rank performance, stringency or leniency, and avoidance of the halo effect.

Methods: Clinically experienced faculty raters and near-peer raters attended a 30-min PhaBS familiarisation session. This was immediately followed by a marking session in which they rated a trainee pharmacist's behavioural skills in three scripted immersive acute care simulated scenarios, demonstrating good, mediocre, and poor performances respectively. Inter-rater reliability in each group was calculated using the two-way random, absolute agreement single-measures intra-class correlation co-efficient (ICC). Differences in individual rater competence in each domain were compared using Pearson's chi-squared test.

Results: The ICC for experienced faculty raters was good at 0.60 (0.48-0.72) and for near-peer raters was poor at 0.38 (0.27-0.54). Of experienced faculty raters, 5/9 were competent in all domains versus 2/13 near-peer raters (difference not statistically significant). There was no statistically significant difference between the abilities of clinically experienced versus near-peer raters in agreement with an expert rater, ability to rank performance, stringency or leniency, or avoidance of the halo effect. The only statistically significant difference between groups was ability to compete the assessment (9/9 experienced faculty raters versus 6/13 near-peer raters, p = 0.0077).

Conclusions: Experienced faculty have acceptable inter-rater reliability when using PhaBS, consistent with other behaviour marker systems; however, not all raters are competent. Competence measures for other assessments can be helpfully applied to behavioural marker systems. When using behavioural marker systems for assessment, educators must start using such rater competence frameworks. This is important to ensure fair and accurate assessments for learners, to provide educators with information about rater training programmes, and to provide individual raters with meaningful feedback.

Download full-text PDF

Source
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC11687013PMC
http://dx.doi.org/10.1186/s41077-024-00329-9DOI Listing

Publication Analysis

Top Keywords

experienced faculty
24
near-peer raters
24
behavioural marker
20
inter-rater reliability
20
faculty raters
20
rater competence
16
marker systems
16
raters
14
clinically experienced
12
rater
8

Similar Publications

Study Objectives: Cancer-related fatigue is one of the most common symptoms in cancer survivors. Cognitive behavioural therapy for insomnia (CBT-I) can improve fatigue, but mechanisms are unclear. This secondary analysis of a randomized controlled trial evaluated whether CBT-I led to a significant improvement in fatigue, accounting for change in comorbid symptoms of insomnia, perceived cognitive impairment (PCI), anxiety, and depression.

View Article and Find Full Text PDF

Introduction: This prospective, single-arm pharmacodynamic study assessed the effect of colchicine (COLC) [Strides Pharma UK Ltd, Watford, Hertfordshire, England] 0.5 mg administered orally once daily for 14 days on platelet reactivity with respect to aspirin reaction units (ARUs) and P2Y reaction units (PRUs).

Methods: Twenty-two patients with stable coronary artery disease (CAD) on dual antiplatelet therapy (DAPT) with daily maintenance aspirin and clopidogrel were recruited.

View Article and Find Full Text PDF

Assessment Implementation Measures (AIM): The Journey Towards the Tool Validity.

J Coll Physicians Surg Pak

January 2025

Department of Oral and Maxillofacial Surgery, Frontier Medical and Dental College, Abbottabad, Pakistan.

Objective: To establish the construct validity of the Assessment Implementation Measures (AIM) tool to accurately assess faculty perspectives on implemented assessment systems, facilitating the alignment with set standards.

Study Design: Qualitative Study. Place and Duration of the Study: The study was carried out at RIPHAH International University and data were collected from participants (senior faculty members) involved in teaching and assessment of undergraduates from various medical and dental colleges in Pakistan.

View Article and Find Full Text PDF

A surge in female condom distribution during the COVID-19 pandemic in Gauteng province, South Africa.

Contracept Reprod Med

January 2025

School of Public Health, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa.

Background: Female Condoms are 90-95% effective against HIV transmission when correctly and consistently used and are also cost-effective. In general, condoms prevent sexually transmitted infections (STIs) and unwanted pregnancies. Although the COVID-19 pandemic had the potential to undermine routine healthcare services delivery and utilisation, there is limited evidence about the pandemic's effect on Female Condom uptake in Gauteng, one of the hardest-hit provinces in South Africa.

View Article and Find Full Text PDF

Objectives: Palliative care (PC) is an interdisciplinary approach aimed at improving the physical, psychological, and spiritual well-being of patients and families affected by life-threatening diseases. This study aimed to investigate the need for PC among critically ill patients and their quality of life (QOL) in low-income groups in Bangladesh.

Methods: This cross-sectional study was conducted at four healthcare facilities from March to April 2023, involving 553 registered patients with advanced chronic conditions.

View Article and Find Full Text PDF

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!