How Consistent Is Competent? Examining Variance in Psychomotor Skills Assessment.

Abstract

Direct assessment of trainee performance across time is a core tenet of competency-based medical education. Unlike variability of psychomotor skills across levels of expertise, performance variability exhibited by a particular trainee across time remains unexplored. The goal of this study was to document the consistency of individual surgeons' technical skill performance. A secondary analysis of assessment data (collected in 2010-2012, originally published in 2015) generated by a prospective cohort of participants at Montreal Children's Hospital with differing levels of expertise was conducted in 2017. Trained raters scored blinded recordings of a myringotomy and tube insertion performed 4 times by junior and senior residents and attending surgeons over a 6-month period using a previously reported assessment tool. Descriptive exploratory analyses and univariate comparison of standard deviations (SDs) were conducted to document variability within individuals across time and across training levels. Thirty-six assessments from 9 participants were analyzed. The SD of scores for junior residents was highly variable (5.8 out of a scale of 30 compared with 1.8 for both senior residents and attendings [F(2,19) = 5.68, P < 0.05]). For a given individual, the range of scores was twice as large for junior residents than for senior residents and attendings. Surgical residents may display highly variable performances across time, and individual variability appears to decrease with increasing expertise. Operative skill variability could be underrepresented in direct observation assessment; emphasis on an adequate amount of repetitive evaluations for junior residents may be needed to support judgments of competence or entrustment.