Differences in Resident Self-Evaluation and Clinical Competency Committee Evaluation Using ACGME Milestone Versions 1.0 and 2.0

Abstract

Intentionally self-driven professional development of surgical resident physicians is a hallmark of surgical training and is expected to gain further traction as Entrustable Professional Activities (EPAs) become the new paradigm for surgical education. We aimed to analyze how surgical residents rate themselves as compared to the evaluation of the Clinical Competency Committee using ACGME Milestones Version 1 (M1.0) and Version 2 (M2.0). We asked 22 general surgical trainees for self-evaluation of Milestones (both M1.0 and M2.0) from 2017 semiannually to 2022. ACGME-required Milestone evaluations by the Clinical Competency Committee (CCC) were independently performed after the time window for resident self-evaluation. Neither trainees nor CCC were aware of the other party's evaluations. There were 1552 paired data available for evaluating individual competencies by both trainees and CCC. Paired Wilcoxon signed-rank tests were then performed among the corresponding pairs. MercyOne Des Moines Medical Center, Des Moines, IA; Teaching tertiary referral center. Twenty-two general surgical trainees at this hospital and 28 faculty surgeons participated in this study. The average self-evaluation of surgical residents was lower in the M1.0 cohort compared to the corresponding CCC evaluation (1.96 ± 0.72 vs. 2.11 ± 0.67; p < 0.001). M1.0 self-assessments and CCC-assessments were statistically similar for ICS (p = 0.548) and PROF (p = 0.554) competencies and differed for MK (p < 0.001), PBLI (p < 0.001), PC (p < 0.001), SBP (p = 0.008). On the contrary, the M2.0 cohort demonstrated higher average self-evaluation of surgical residents compared to the corresponding CCC evaluation (2.75 ± 0.87 vs. 2.12 ± 0.97; p < 0.001). Significant differences were observed for all 6 ACGME competencies using M2.0 self-assessments and CCC-assessments (all p < 0.001). Multivariate regression modeling (p < 0.001, R2 = 0.255) predicted the degree of discordance between self-assessment and CCC-assessed achievement of competencies with a significant effect of gender (baseline male: coef = -0.232, p < 0.001), PGY level (-0.083 per year, p < 0.001) and Milestone version (0.831, p < 0.001). A significant interaction exists for all gender/Milestone combinations except for the female trainees with M1.0. The difference between self-evaluated Milestone achievement and faculty-driven CCC evaluation of surgical resident physician performance is more evident in Milestones 2.0 than in Milestones 1.0. Residents self-evaluate higher compared to faculty using Milestones 2.0. This discrepancy is seen among both genders and is more pronounced among male residents overestimating core competencies with M2.0 self-evaluation than formal CCC assessment.