Impact of an entrustable professional activities-based assessment system

Abstract

Background: Beginning in 2014, all Accreditation Counsel of Graduate Medical Education (ACGME) accredited residency programs were required to move to a Milestones-based system for biannual resident assessment. The resident assessment system for the Virginia Tech-Carilion Obstetrics and Gynecology (OB/GYN) residency program was re-designed to meet this requirement in July, 2014. The ACGME Milestones based assessment tool was identified on multiple faculty surveys as an area for improvement. To address this issue, an entrustable professional activities (EPA) based assessment system was designed and implemented for assessment of all OB/GYN rotations. Objective: To evaluate the impact of an EPA based resident assessment system on faculty member’s evaluation of resident assessment tools. Methods: In this prospective quality improvement study, a survey was sent to all faculty members prior to the implementation of the EPA-based assessment system. The same survey was performed three months after the implementation of the new system. To facilitate analysis, each level of agreement was assigned a numerical value (1-5). The results were aggregated, and were analyzed using t-tests, assuming unequal variances. Results: Sixty-eight percent of the faculty responded to the first survey, and 67% responded to the follow up survey. Statistically significant (p<.05) improvements were noted in most measures of the EPA based assessment tool including “ease of use” (2.2 vs 4.4, p< 0.001) and “accurate representation of resident performance” (2.5 vs 3.9, p <0.001). Conclusion: An EPA based resident evaluation system significantly improved teaching faculty’s impression of most domains of our OB/GYN resident assessment tools.