Assessment of clinical competence is complex and inference based. Trustworthy and defensible assessment processes must have favourable evidence of validity, particularly where decisions are considered high stakes. We aimed to organize, collect and interpret validity evidence for a high stakes simulation based assessment strategy for certifying paramedics, using Kane's validity framework, which some report as challenging to implement. We describe our experience using the framework, identifying challenges, decisions points, interpretations and lessons learned. We considered data related to four inferences (scoring, generalization, extrapolation, implications) occurring during assessment and treated validity as a series of assumptions we must evaluate, resulting in several hypotheses and proposed analyses. We then interpreted our findings across the four inferences, judging if the evidence supported or refuted our proposed uses of the assessment data. Data evaluating "Scoring" included: (a) desirable tool characteristics, with acceptable inter-item correlations (b) strong item-total correlations (c) low error variance for items and raters, and (d) strong inter-rater reliability. Data evaluating "Generalizability" included: (a) a robust sampling strategy capturing the majority of relevant medical directives, skills and national competencies, and good overall and inter-station reliability. Data evaluating "Extrapolation" included: low correlations between assessment scores by dimension and clinical errors in practice. Data evaluating "Implications" included low error rates in practice. Interpreting our findings according to Kane's framework, we suggest the evidence for scoring, generalization and implications supports use of our simulation-based paramedic assessment strategy as a certifying exam; however, the extrapolation evidence was weak, suggesting exam scores did not predict clinical error rates. Our analysis represents a worked example others can follow when using Kane's validity framework to evaluate, and iteratively develop and refine assessment strategies.

Download full-text PDF

Source
http://dx.doi.org/10.1007/s10459-017-9800-3DOI Listing

Publication Analysis

Top Keywords

data evaluating
16
kane's validity
12
validity framework
12
assessment
9
simulation based
8
based assessment
8
assessment clinical
8
clinical competence
8
high stakes
8
assessment strategy
8

Similar Publications

Objective: To examine the medical students' awareness of laparoscopic surgery as well as assess the perceived importance of laparoscopic simulation training, and its impact on students' confidence, career aspirations, proficiency, spatial skills, and physical tolerance.

Design: Descriptive and comparative study using pre- and post-training assessments.

Setting: Simulation training sessions centred on laparoscopic surgery techniques.

View Article and Find Full Text PDF

Background: To develop and validate a clinical-radiomics model for preoperative prediction of lymphovascular invasion (LVI) in rectal cancer.

Methods: This retrospective study included data from 239 patients with pathologically confirmed rectal adenocarcinoma from two centers, all of whom underwent MRI examinations. Cases from the first center (n = 189) were randomly divided into a training set and an internal validation set at a 7:3 ratio, while cases from the second center (n = 50) constituted the external validation set.

View Article and Find Full Text PDF

Background And Importance: Traumatic intracranial hemorrhage (tICH) after mild traumatic brain injury (mTBI) is not uncommon in the elderly. Often, these patients are admitted to the hospital for observation. The necessity of admission in the absence of clinically important intracranial injuries is however unclear.

View Article and Find Full Text PDF

Stoma reversal surgery is known for relatively high complication rates. While Enhanced Recovery After Surgery (ERAS) protocols are extensively validated for colorectal surgery, their use in stoma reversal remains underexplored. This systematic review and meta-analysis evaluates clinical outcomes of stoma reversal surgery under ERAS protocols compared to standard care (SC) practices.

View Article and Find Full Text PDF

Purpose: Heart failure (HF) is a disease that leads to approximately 300,000 fatalities annually in Europe and 250,000 deaths each year in the United States. Type 2 Diabetes Mellitus (T2DM) is a significant risk factor for HF, and testing for N-terminal (NT)-pro hormone BNP (NT-proBNP) can aid in early detection of HF in T2DM patients. We therefore developed and validated the HFriskT2DM-HScore, an algorithm to predict the risk of HF in T2DM patients, so guiding NT-proBNP investigation in a primary care setting.

View Article and Find Full Text PDF

Want AI Summaries of new PubMed Abstracts delivered to your In-box?

Enter search terms and have AI summaries delivered each week - change queries or unsubscribe any time!