Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

Abstract

Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method. In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment. Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74). EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.