How Consistent Is Competent? Examining Variance in Psychomotor Skills Assessment.

Purpose: Direct assessment of trainee performance across time is a core tenet of competency-based medical education. Unlike variability of psychomotor skills across levels of expertise, performance variability exhibited by a particular trainee across time remains unexplored. The goal of this study was to document the consistency of individual surgeons' technical skill performance.

Method: A secondary analysis of assessment data (collected in 2010-2012, originally published in 2015) generated by a prospective cohort of participants at Montreal Children's Hospital with differing levels of expertise was conducted in 2017. Trained raters scored blinded recordings of a myringotomy and tube insertion performed 4 times by junior and senior residents and attending surgeons over a 6-month period using a previously reported assessment tool. Descriptive exploratory analyses and univariate comparison of standard deviations (SDs) were conducted to document variability within individuals across time and across training levels.

Results: Thirty-six assessments from 9 participants were analyzed. The SD of scores for junior residents was highly variable (5.8 out of a scale of 30 compared with 1.8 for both senior residents and attendings [F(2,19) = 5.68, P < 0.05]). For a given individual, the range of scores was twice as large for junior residents than for senior residents and attendings.

Conclusions: Surgical residents may display highly variable performances across time, and individual variability appears to decrease with increasing expertise. Operative skill variability could be underrepresented in direct observation assessment; emphasis on an adequate amount of repetitive evaluations for junior residents may be needed to support judgments of competence or entrustment.