ICRE Top Research Abstracts

ICRE Top Research Abstracts

Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade.

Competency-Based Medical Training in Anesthesiology: Has It Delivered on the Promise of Better Education?

Introduction: The specialist Emergency Medicine (EM) postgraduate training program at Queens University implemented a new Competency-Based Medical Education (CBME) model on July 1 2017. This occurred one year ahead of the national EM cohort, in the model of Competence By Design (CBD) as outlined by the Royal College of Physicians and Surgeons of Canada (RCPSC). This presents an opportunity to identify critical steps, successes, and challenges in the implementation process to inform ongoing national CBME implementation efforts. Methods: A case-study methodology with Rapid Cycle Evaluation was used to explore the lived experience of implementing CBME in EM at Queens, and capture evidence of behavioural change. Data was collected at 3- and 6- months post-implementation via multiple sources and methods, including: field observations, document analysis, and interviews with key stakeholders: residents, faculty, program director, CBME lead, academic advisors, and competence committee members. Qualitative findings have been triangulated with available quantitative electronic assessment data. Results: The critical processes of implementation have been outlined in 3 domain categories: administrative transition, resident transition, and faculty transition. Multiple themes emerged from stakeholder interviews including: need for holistic assessment beyond Entrustable Professional Activity (EPA) assessments, concerns about the utility of milestones in workplace based assessment by front-line faculty, trepidation that CBME is adding to, rather than replacing, old processes, and a need for effective data visualisation and filtering for assessment decisions by competency committees. We identified a need for administrative direction and faculty development related to: new roles and responsibilities, shared mental models of EPAs and entrustment scoring. Quantitative data indicates that the targeted number of assessments per EPA and stage of training may be too high. Conclusion: Exploring the lived experience of implementing CBME from the perspectives of all stakeholders has provided early insights regarding the successes and challenges of operationalizing CBME on the ground. Our findings will inform ongoing local implementation and higher-level national planning by the Canadian EM Specialty Committee and other programs who will be implementing CBME in the near future.

A key component of competency-based medical education (CBME) is direct observation of trainees. Direct observation has been emphasized as integral to workplace-based assessment (WBA) yet previously identified challenges may limit its successful implementation. Given these challenges, it is imperative to fully understand the value of direct observation within a CBME program of assessment. Specifically, it is not known whether the quality of WBA documentation is influenced by observation type (direct or indirect). The objective of this study was to determine the influence of observation type (direct or indirect) on quality of entrustable professional activity (EPA) assessment documentation within a CBME program. EPA assessments were scored by four raters using the Quality of Assessment for Learning (QuAL) instrument, a previously published three-item quantitative measure of the quality of written comments associated with a single clinical performance score. An analysis of variance was performed to compare mean QuAL scores among the direct and indirect observation groups. The reliability of the QuAL instrument for EPA assessments was calculated using a generalizability analysis. A total of 244 EPA assessments (122 direct observation, 122 indirect observation) were rated for quality using the QuAL instrument. No difference in mean QuAL score was identified between the direct and indirect observation groups (p=0.17). The reliability of the QuAL instrument for EPA assessments was 0.84. Observation type (direct or indirect) did not influence the quality of EPA assessment documentation. This finding raises the question of how direct and indirect observation truly differ and the implications for meta-raters such as competence committees responsible for making judgments related to trainee promotion.

Entrustable professional activities (EPAs) have become a cornerstone of assessment in competency-based medical education (CBME). Increasingly, EPAs are being adopted that do not conform to EPA standards. This study aimed to develop and validate a scoring rubric to evaluate EPAs for alignment with their purpose, and to identify substandard EPAs. The EQual rubric was developed and revised by a team of education scholars with expertise in EPAs. It was then applied by four residency program directors/CBME leads (PDs) and four nonclinician support staff to 31 stage-specific EPAs developed for internal medicine in the Royal College of Physicians and Surgeons of Canada's Competency by Design framework. Results were analyzed using a generalizability study to evaluate overall reliability, with the EPAs as the object of measurement. Item-level analysis was performed to determine reliability and discrimination value for each item. Scores from the PDs were also compared with decisions about revisions made independently by the education scholars group. The EQual rubric demonstrated high reliability in the G-study with a phi-coefficient of 0.84 when applied by the PDs, and moderate reliability when applied by the support staff at 0.67. Item-level analysis identified three items that performed poorly with low item discrimination and low interrater reliability indices. Scores from support staff only moderately correlated with PDs. Using the preestablished cut score, PDs identified 9 of 10 EPAs deemed to require major revision. EQual rubric scores reliably measured alignment of EPAs with literature-described standards. Further, its application accurately identified EPAs requiring major revisions.

A key component of competency-based medical education is workplace-based assessment, which includes observation (direct or indirect) of residents. Direct observation has been emphasized as an ideal form of assessment yet challenges have been identified that may limit its adoption. At present, it remains unclear how often direct and indirect observation are being used within the clinical setting. The objective of this study was to describe patterns of observation in an emergency medicine competency-based program 2years postimplementation. Emergency medicine residents (n=19) recorded the type of observation they received (direct or indirect) following workplace-based entrustable professional activity (EPA) assessments from December 15, 2019, to April 30, 2020. Assessment forms were reviewed and analyzed to describe patters of observation. Assessments were collected on all 19 eligible residents (100% participation). A total of 1,070 EPA assessments were completed during the study period, of which 798 (74.6%) had the type of observation recorded. Of these recorded observations, 546 (68.4%) were directly observed and 252 (31.6%) were indirectly observed. The length of written comments contained within assessments following direct and indirect observation did not differ significantly. There was no significant association between resident gender and observation type or resident stage of training and observation type. Certain EPA assessments showed a clear preference toward either direct or indirect observation. To the best of our knowledge, this study is the first to report patterns of observation in a competency-based residency program. The results suggest that direct observation can be quickly adopted as the primary means of workplace-based assessment. Indirect observation comprised a sizeable minority of observations and may be an underrecognized contributor to workplace-based assessment. The preference toward either direct or indirect observation for certain EPA assessments suggests that the entrustable professional activity itself may influence the type of observation.

Establishing a Competency Based Medical Education Curriculum for International Medical Graduates Pursuing Pediatric Surgery Training in High-income Countries.

The Importance of Competency-Based Programmatic Assessment in Graduate Medical Education.

Real-world impact of the COVID-19 pandemic on the assessment of anaesthesiology residents

Order entry, entrustable professional activity (EPA) 4, is one of several EPAs that residency program directors identify as a weakness for PGY 1 residents. A multispecialty survey of program directors indicated that only 69% of interns could be trusted to enter and discuss orders and prescriptions without supervision. To address this gap, we developed a formative workshop for fourth-year medical students. Prior to the start of their subinternships, 366 fourth-year medical students engaged in an order entry workshop. Students performed chart reviews on electronic standardized patients within an educational electronic health record (EHR), placed admission orders, customized order sets, responded to safety alerts, utilized decision support tools, and incorporated high-value care considerations. Students used expert-validated rubrics to assess the quality of their admission orders and participated in a facilitated group discussion on key learning points. Finally, students participated in order entry, with all orders requiring cosignature by a supervising physician, during their clinical rotations. Students reported their confidence with order entry before and after the workshop and after the clinical rotation. One hundred seventeen students completed the pre- and postworkshop surveys, and 99 went on to complete the postcourse evaluation. Students showed a statistically significant increase in their confidence level following the workshop. Order entry is a critical, complex skill that requires deliberate instruction. This curriculum, which leverages the features of an educational EHR, can facilitate instruction, practice, and confidence gains regarding order entry prior to further application of these skills in the clinical environment.

The Referral and Consultation Entrustable Professional Activity: Defining the Components in Order to Develop a Curriculum for Pediatric Residents

Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings. An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees' endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes. Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions "comfortably" (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making. Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters' criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.

Entrustable Professional Activities (EPAs) has recently been the buzzword in medical education and in the realm of the healthcare professions. Medical professional bodies and medical educators worldwide envision the development of EPAs as a promising approach to overcome the difficulties in assessing their trainee’s competencies in a work-integrated learning (WIL) clinical environment. Albeit numerous research has been done on how EPAs can be used as a stopwatch type of competencies assessment method helping to ensure medical trainees are judged to be the best based on well-tested as well as agreed-upon meaningful standards, the prime and essential question of how best EPAs could drive students’ daily self-directed learning for their continuous professional development of expertise remains largely unanswered. Amid this leading-edge development in the competency-based medical education (CBME) along with the important role healthcare has to play in the 21st century’s global society, there is an increasing need to understand current processes of change and the impact this will have on preparing medical professionals and on healthcare in the coming future. With this endeavour in mind, this paper provides an update on the progress of this gradually establishing approach to competency-based medical education, identifies the essence of this emerging novel workplace-based assessment method in relations to a review of the contemporary competency-based medical education, and finally explores how Jesuit’s wisdom could be used in furthering EPAs’ potential impact on attainment of medical education’s mission.

Creating a Competency-Based Medical Education Curriculum for Canadian Diagnostic Radiology Residency (Queen's Fundamental Innovations in Residency Education)-Part 2: Core of Discipline Stage.