A Question of Scale? Comparison of Generalizability in Ottawa and Chen Scales When Used to Formulate Ad Hoc Entrustment Decisions for the Core EPAs.

Abstract

Purpose: Assessment of the Core Entrustable Professional Activities (Core EPAs) is based on observations of supervisors throughout a medical student’s progression toward entrustment. 1,2 In our previous work, we examined performance of the Ottawa Clinic Assessment Tool (Ottawa) 3 when used to measure medical student performance of the Core EPAs in the workplace setting. 4 The findings from that study demonstrated poor generalizability of the Ottawa scale. 4 The primary purpose of the present study was to compare performance of the Ottawa scale with a second scale—the undergraduate medical education (UME) supervisory scale proposed by Chen and colleagues (Chen) 5 in terms of reliability and generalizability. A secondary aim was to determine the impact of frequent assessors on the validity and reliability of the data. Methods: For the 2019–2020 academic year, the Virginia Commonwealth University School of Medicine modified a previously described, 4 student-initiated, workplace-based assessment (WBA) system developed to provide formative feedback for the Core EPAs across clerkships. The WBA scored students’ performance using both the modified Ottawa and the modified Chen scales. Generalizability and decision studies were performed to determine the reliability of each scale. Secondary analysis explored whether faculty who frequently assess the EPAs demonstrated better reliability. Results: A total of 923 raters completed 7,277 WBAs on 208 medical students across all clerkships. Using Ottawa, variability attributable to the student ranged from 0.8% to 6.5%. For Chen, variability attributable to the student ranged from 1.8% to 7.1%. These findings indicate that the majority of the variation for EPA ratings was due to the rater (42.8%–61.3%) and other unexplained factors. A range of 28 to 127 assessments were required to obtain a Phi coefficient of 0.70. For 2 EPAs, using only faculty who frequently assess the EPA improved generalizability—requiring only 5 and 13 assessments for the Chen scale. Discussion: Both the Ottawa and Chen scales performed poorly in terms of variance attributed to the learner. The frequent assessor model seemed to increase variance attributed to the learner for the Chen scale in only 2 Core EPAs. Overall, these findings were similar to our previous study 4 involving only the Ottawa scale. Based on these findings in conjunction with prior evidence, we suggest that the root cause analysis for challenges associated with WBAs for the Core EPAs involves the lack of a true competency-based curriculum in UME; the choice of scale alone does not appear to impact performance. Significance: This study adds to the emerging literature around WBAs specific to the Core EPAs in UME. Based on these findings as well as those from our prior work, we feel there is a need to reconsider the workflow, scale, and investment of learners and faculty to best assess the Core EPAs in the UME setting. Acknowledgments: The authors would like to thank Joel Browning (former director of academic information systems at Virginia Commonwealth University School of Medicine [VCU-SOM]), who worked with Brie Dubinsky, MS, to develop the workplace-based assessment system described in this manuscript. In addition, the authors would like to thank Yoon Soo Park, PhD, for his assistance with statistical methodology.