EPA-based Assessment Research Articles

Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process.

Entrustable professional activities (EPAs) will be used for initial certification by the American Board of Pediatrics by 2028. Less than half of pediatric fellowships currently use EPAs for assessment, yetall will need to adopt them. Our objectives were to identify facilitators and barriers to the implementation of EPAs to assess pediatric fellows and to determine fellowship program directors' (FPD) perceptions of EPAs and Milestones. We conducted a survey of FPDs from 15 pediatric subspecialties. EPA users were asked about their implementation of EPAs, barriers encountered, and perceptions of EPAs. Nonusers were queried about deterrents to using EPAs. Both groups were asked about potential facilitators of implementation and their perceptions of Milestones. The response rate was 65% (575/883). Of these, 344 (59.8%) were EPA users and 231 (40.2%) were nonusers. Both groups indicated work burden as a barrier to implementation. Nonusers reported more barriers than users (mean [SD]: 7 [3.8] vs 5.8 [3.4], P < .001). Both groups identified training materials and premade assessment forms as facilitators to implementation. Users felt that EPAs were easier to understand than Milestones (89%) and better reflected what it meant to be a practicing subspecialty physician (90%). In contrast, nonusers felt that Milestones were easy to understand (57%) and reflected what it meant to be a practicing subspecialist (58%). Implementing EPA-based assessment will require a substantial investment by FPDs, facilitated by guidance and easily accessible resources provided by multiple organizations. Perceived barriers to be addressed include FPD time constraints, a need for additional assessment tools, and outcomes data.

A Modified Delphi Consensus Approach to Define Entrustable Professional Activities for Neurocritical Care Advanced Practice Providers.

To define consensus entrustable professional activities (EPAs) for neurocritical care (NCC) advanced practice providers (APPs), establish validity evidence for the EPAs, and evaluate factors that inform entrustment expectations of NCC APP supervisors. A three-round modified Delphi consensus process followed by application of the EQual rubric and assessment of generalizability by clinicians not affiliated with academic medical centers. Electronic surveys. NCC APPs ( n = 18) and physicians ( n = 12) in the United States with experience in education scholarship or APP program leadership. None. The steering committee generated an initial list of 61 possible EPAs. The panel proposed 30 additional EPAs. A total of 47 unique nested EPAs were retained by consensus opinion. The steering committee defined six core EPAs addressing medical knowledge, procedural competencies, and communication proficiency which encompassed the nested EPAs. All core EPAs were retained and subsequently met the previously described cut score for quality and structure using the EQual rubric. Most clinicians who were not affiliated with academic medical centers rated each of the six core EPAs as very important or mandatory. Entrustment expectations did not vary by prespecified groups. Expert consensus was used to create EPAs for NCC APPs that reached a predefined quality standard and were important to most clinicians in different practice settings. We did not identify variables that significantly predicted entrustment expectations. These EPAs may aid in curricular design for an EPA-based assessment of new NCC APPs and may inform the development of EPAs for APPs in other critical care subspecialties.

Validity and Educational Impact of Clinical Entrustable Professional Activities Linked to the Nutrition Care Process for Work-Based Assessment of Entry-Level Dietetics Students: Evaluation of a3-Year Implementation Study in Australia.

Entrustable professional activities (EPAs) describe key workplace activities and are increasingly being used for student work-based assessment. An EPA-based work-based assessment tool offers potential to increase understanding of dietetics student skill development and opportunity for standardised work-based student assessment. To determine construct validity and educational impact of an EPA work-based assessment tool for dietetics placement students in clinical settings. Using a time series design, supervisor and student self-assessment data collected from an EPA-based assessment tool from three cohorts of dietetics students and supervisors from 2017-2019 was analyzed. Dietetics students (n= 145) from an accredited dietetic training program in Australia and affiliated metropolitan and rural hospitals. Construct validity was determined through analysis of supervisor evaluation of student performance against EPAs over time. Educational impact was determined through comparing student performance across EPAs to predicted milestones to identify areas of least entrustment. Student performance over time and differences between student self-assessment and supervisor assessment for each EPA were investigated using Wilcoxon signed-rank tests and linear mixed-model analysis. Descriptive statistics summarised student performance against each EPA. Performance significantly increased over time in 35 out of 37 EPAs. Significant differences between supervisor and student self-assessment were evident in 9 out of 37 EPAs. Dietetics student performance varied across EPAs with 88.2% of students achieving entrustment for nutrition management EPAs compared with 100% for professionalism EPAs. The tool's construct validity was established. EPAs identified areas of Nutrition Care Process skills development requiring additional support. Data collected by the tool can inform teaching.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose. This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Implications of Entrustable Professional Activities for Motivation and Learning.

This study sought to explore the experience of entrustable professional activity-based assessment and its effect on self-determination, a well-supported measure of motivation. The goal was to understand EPA-based assessment's implications for student motivation, learning, and wellbeing. A qualitative case study design was utilized through a standardized simulation activity with EPA-based assessment. An analysis of interviews and reflection journals was completed by the coding of emergent themes and significant statements with a subsequent thematic analysis framed in self-determination theory (SDT). Entrustment was found to be an affectively and interpersonally impactful metric that provided the learner with a clear understanding of clinical performance. Quality entrustment feedback supported participants' sense of competence and increased their confidence in their ability to be autonomous as well as a valuable part of the healthcare team. EPAs appeared to support learners' sense of competence and to provide a bridge to support autonomy and the ability to relate to others. To the extent that entrustment supported learners' sense of competence, other SDT propositions could be subsequently bolstered as well. Self-determination and intrinsic motivation were uniquely supported by EPA-based assessment, if implemented effectively. Further research is needed to better define effective implementation, faculty development, and feedback strategies.

A Tool to Assess Competence in Critical Care Ultrasound Based on Entrustable Professional Activities.

Existing assessment tools for competence in critical care ultrasound (CCUS) have limited scope and interrupt clinical workflow. The framework of entrustable professional activities (EPAs) is well suited to developing an assessment tool that is comprehensive and readily integrated into the intensive care unit (ICU) training environment. This study sought to design an EPA-based tool to assess competence in CCUS for pulmonary and critical care fellows and to assess the validity and reliability of the tool. Eight experts in CCUS met to define the core EPAs for CCUS. A nominal group technique was used to reach consensus. An assessment tool was created based on the EPAs with a modified Ottawa entrustability scale. Trained faculty evaluated pulmonary and critical care fellows using this tool in the ICU over a 6-month study period at a single institution. An assessment of validity of the EPA-based tool is made with four sources of validity evidence: content, response process, reliability, and relation to other variables. Reliability and response process data were generated using generalizability theory analysis to estimate sources of variance in entrustment scores. Analysis of response process validity and validity by relation to other variables was performed using regression models. Fifty-four assessments were recorded during the study period, conducted on 23 trainees by 13 faculty. Content validity of the tool was demonstrated using expert consensus and published guidelines from critical care societies to define the EPAs. Response process validity was demonstrated by the low variance in entrustment scores due to evaluators (0.086 or 6%) and high agreement between score and trainee self-assessment (regression coefficient, 0.82; P < 0.0001). Reliability was demonstrated by the high "true" variance in entrustment score attributable to the trainee: 0.674 or 45%. Validity by relation to other variables was demonstrated using regression analysis to show correlation between entrustment score and the number of times a fellow has performed an EPA (regression coefficient, 0.023; P < 0.0001). An EPA-based assessment tool for competence in CCUS was created. We obtained sufficient validity evidence on three of the diagnostic EPAs. Procedural EPAs were infrequently assessed, limiting generalizability in this subgroup.

Viewing Readiness-for-Residency through Binoculars: Mapping Competency-Based Assessments to the AAMC’s 13 Core Entrustable Professional Activities (EPAs)

Construct: The construct being assessed is readiness-for-residency of graduating medical students, as measured through two assessment frameworks. Background: Readiness-for-residency of near-graduate medical students should be but is not consistently assessed. To address this, the Association of American Medical Colleges (AAMC), in 2014, identified and described 13 core Entrustable Professional Activities (EPAs), which are tasks that all residents should be able to perform unsupervised upon entering residency. However, the AAMC did not initially provide measurement guidelines or propose standardized assessments. We designed Night-onCall (NOC), an immersive simulation for our near-graduating medical students to assess and address their readiness-for-residency, framed around tasks suggested by the AAMC’s core EPAs. In adopting this EPA assessment framework, we began by building upon an established program of competency-based clinical skills assessments, repurposing competency-based checklists to measure components of the EPAs where possible, and designing new checklists, when necessary. This resulted in a blended suite of 14 checklists, which theoretically provide substantive assessment of all 13 core EPAs. In this paper, we describe the consensus-based mapping process conducted to ensure we understood the relationship between competency and EPA-based assessment lenses and could therefore report meaningful feedback on both to transitioning students in the NOC exercise. Approach: Between January-November 2017, five clinician and two non-clinician health professions educators at NYU Grossman School of Medicine conducted a rigorous consensus-based mapping process, which included each rater mapping each of the 310 NOC competency-based checklist items to lists of entrustable behaviors expected of learners according to the AAMC 13 core EPAs. Findings: All EPAs were captured to varying degrees by the 14 NOC checklists (overall Intraclass Correlation Coefficient (ICC) = 0.77). Consensus meetings resolved discrepancies and improved ICC values for three (EPA-9, EPA-10, EPA-12) of the four EPAs that initially showed poor reliability. Conclusions: Findings suggest that with some limitations (e.g., EPA-7 “form clinical questions/retrieve evidence”) established competency-based assessments can be repurposed to measure readiness-for-residency through an EPA lens and both can be reported to learners and faculty.

Implementing an entrustable professional activities programmatic assessments for nurse practitioner training in emergency care: A pilot study

BackgroundEntrustable professional activities (EPAs), as new methods used to operationalize competency-based education, are gaining acceptance in medical professions. However, no EPA-based framework exists to evaluate emergency care nurse practitioners' competencies in clinical practice. ObjectivesTo develop, implement, and evaluate an EPA-based framework for emergency care nurse practitioners. DesignProspective observational study. SettingAn accredited nurse practitioner training program in the emergency department of a University-affiliated Hospital. ParticipantsEight novice nurse practitioners and eighteen clinical instructors. MethodsA working group of academic educators, nurse practitioner leaders, and clinical instructors developed a set of EPAs using a participatory design method. We scheduled weekly ad hoc evaluations for the nurse practitioners in emergency care, and we collected several observations from multiple assessors for analysis. ResultsFour nested EPAs were developed that adopted a 1–5 supervision–entrustment scale to measure performance. The results of the 20-month assessments illustrated that the average scores of EPA1, which measured the assessment and management of patients with acute medical presentations in the emergency department, differed significantly but were not positively correlated with the nurse practitioners' increasing subspecialty and department seniority levels (Kruskal–Wallis test P = .011 and 0.006, respectively). In addition, the most time required for the nurse practitioners to achieve a stable level 5 entrustability score for all EPAs for at least 3 consecutive months was 18 months. Clinical instructors noted that recognizing limits is a skill that is essential for a nurse practitioner to earn clinical instructors' trust. ConclusionsEntrustability scores may not correlate with nurse practitioners' increasing seniority, and patient care requires the longest training time (18 months) for emergency care nurse practitioners to achieve autonomy. Program directors should adjust the supervision–entrustment level expected at various stages of training according to the complexity of individual EPAs. An EPA-based assessment framework in a nurse practitioner training program may assist in overseeing the development of competencies for emergency care nurse practitioners.

Feasibility, and validity aspects of Entrustable Professional Activity (EPA)-based assessment in general practice training

ABSTRACT Introduction Entrustable Professional Activities (EPAs) are developed to support the practical application of competency frameworks in postgraduate medical education (PGME) programmes. EPAs are used for the assessment of the trainees’ competence development, which takes place by means of an entrustment decision, aiming to stimulate learning and independent practice in trainees. In this pilot study, we explore the feasibility and validity of EPA-based assessment in a General Practice (GP) training programme. Methods We used questionnaires to evaluate trainers’ and trainees’ experiences with the use of six EPAs for trainee learning, assessment and independent practice at the Out-of-Hours GP Center. Data were analysed quantitatively and qualitatively. Additionally, we examined the inter-item correlation between scores on EPA-based assessment and competency-based assessment using Spearman’s Rho. Results EPA-based assessment provided opportunities for giving concrete feedback and substantiating competency-based assessment. No consistent correlation between EPA-based assessment and competency-based assessment could be detected. Only later in the course of the training programme a correlation was found between the EPA scores and the degree of independence of trainees. Discussion Results of this pilot study confirm the theories behind EPAs, as well as earlier research on EPAs in the workplace regarding trainee learning, assessment and independent practice. An important limitation of this study was the COVID-19 pandemic, as it influenced the results through reduced inclusion and follow-up, and through the impact on the workplace and trainee learning possibilities. Further research is needed to determine how EPAs support independent practice of trainees, as well as the assessment of trainee competency development.

Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade.

Entrustable Professional Activities: Reflecting on Where We Are to Define a Path for the Next Decade.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended.

Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention. A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively. After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments. Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

Entrustable Professional Activity-Based Summative Performance Assessment in the Surgery Clerkship

The objectives of this study were to 1) assess the performance Entrustable Professional Activities (EPAs) when integrated into the summative assessment of third-year medical students on the surgery clerkship and 2) to compare EPAs to traditional clinical performance assessment tools. EPA assessments were collected prospectively from a minimum of 4 evaluators at the completion of each surgical clerkship rotation from November 2019 to June 2019. Overall EPA-based clinical performance scores were calculated as the sum of the mean EPA score from each evaluator. A rating of overall clinical performance called the clinical performance appraisal (CPA) was also collected. EPA ratings were compared to the CPA score, National Board of Medical Examiners exam score, objective structured clinical exam scores, and final clerkship grade. Northwestern Memorial Hospital, a tertiary care teaching institution in Chicago, IL. Overall, 446 evaluations (111 students) were included in the analysis. The aggregate EPA scores ranged from 11.6-24.0 (mean 19.9 ± 2.0), and the CPA scores ranged from 4.4-9.0 (mean 7.6 ± 0.7). The variance among learners in EPA scores was significantly higher than CPA scores (p < 0.001). The aggregate EPA scores correlated well with CPA scores (Spearman's rho 0.803) but had lesser, positive correlations with the objective structured clinical exam (rho 0.153) and National Board of Medical Examiners (rho 0.265) scores. When all EPA scores were included in ordinal logistic regression, only EPA 6, oral presentation of patients, was independently associated with students' final grades (OR: 10.05, 95%CI 1.41-71.80; p = 0.02). Integration of EPAs for use in clinical performance assessment of medical students is feasible within a surgery clerkship. Compared to a global clinical performance assessment, EPA-based assessment provided better discrimination of clinical performance among learners. Use of EPAs may better identify advanced learners and those that need additional time.

EPA-based assessment: Clinical teachers’ challenges when transitioning to a prospective entrustment-supervision scale

Background This study explores the challenges clinical teachers face when first using a prospective entrustment-supervision (ES) scale in a curriculum based on Entrustable Professional Activities (EPAs). A prospective ES scale has the purpose to estimate at which level of supervision a student will be ready to perform an activity in subsequent encounters. Methods We studied the transition to prospective assessment of medical students in clerkships via semi-structured interviews with twelve purposefully sampled clinical teachers, shortly after the introduction of a new undergraduate EPA-based curriculum and EPA-based assessment employing a prospective ES scale. Results While some clinical teachers showed a correct interpretation, rating strategies also appeared to be affected by the target supervision level for completion of the clerkship. Instructions to estimate readiness for a supervision level in the future were not always understood. Further, teachers' interpretation of the scale anchors relied heavily on the phrasing. Discussion Prospective assessment asks clinical teachers to make an extra inference step in their judgement process from reporting observed performance to estimating future level of supervision. This requires a change in mindset when coming from a retrospective, performance-oriented assessment method, i.e., reporting what was observed. Our findings suggest optimizing the ES-scale wordings and improving faculty development.

Facilitators and Inhibitors to Assessing Entrustable Professional Activities in Pediatric Residency.

Research on entrustable professional activities (EPAs) has focused on EPA development with little attention paid to implementation experiences. This constructivist grounded theory study sought to begin filling this gap by exploring the experiences of pediatric residency programs with implementing EPA-based assessment. Interviews with 19 program leader and clinical competency committee participants from 13 sites were held between January and July 2019. Participants were asked about their experiences with implementing EPA-based assessment. Data collection and analysis were iterative. Participants described a range of facilitators and inhibitors that influenced their efforts to implement EPA-based assessment. These fell into 4 thematic areas: 1) alignment of EPA construct with local views of performance and assessment, 2) assessing EPAs illuminates holes in the residency curriculum, 3) clinical competency committee structure and process impacts EPA-based assessment, and 4) faculty engagement and development drives ability to assess EPAs. Areas described as facilitators by some participants were noted to be inhibitors for others. The sum of a program's facilitators and inhibitors led to more or less ability to assess EPAs on the whole. Finally, the first area functions differently from the others; it can shift the entire balance toward or away from the ability to assess EPAs overall. This study helps fill a void in implementation evidence for EPA-based assessment through better understanding of facilitators and inhibitors to such efforts.

73. ETIOLOGY OF UNABLE TO ASSESS ENTRUSTABLE PROFESSIONAL ACTIVITIES IN A NATIONAL STUDY

Background Assessment using entrustable professional activities (EPAs) is currently under study as a joint effort of the American Board of Pediatrics and program director community. However, limited evidence exists for the feasibility of EPA-based assessment in pediatrics. Aim Among pediatric residency programs that implemented EPA-based assessment over 3 academic years (2015-16, 2016-17, 2017-18), determine how often programs were “unable to assess” a resident on a given EPA. Methods 23 programs reported clinical competency committee (CCC) determined EPA supervision level assignments for all residents in their program for a subset of 5-6 of the 17 general pediatrics EPAs biannually (Fall and Spring), except interns who were only assessed at the end of the academic year (Spring). They indicated “unable to assess” whenever applicable. Results Across the 5 data collection cycles, there were 29,643 opportunities to assign an EPA supervision level to 1987 residents from all three post-graduate years. Of these, 4140 assignments (14%) were deemed “unable to assess.” Across all EPAs, this determination was made less often in mid-year reports than year-end reports (p=.005). Post-graduate year 3 (PGY3) residents and PGY2 residents had significantly lower frequencies of “unable to assess” assignments when compared with PGY1 residents (Figure). EPAs with the lowest rates of “unable to assess” included: EPA 4 (patients with acute, common dx) at 0.4%, EPAs 10 (initial resuscitation/stabilization) and 16 (handovers) at 0.9%, EPA 11 (manage information) at 4%, and EPA 13 (practice management) at 5%. EPAs with the highest rates of “unable to assess” included EPA 8 (transition to adult care) at 43%, EPA 17 (procedures) at 27%, EPA 14 (quality improvement) at 20%, and EPAs 12 (make referrals) and 9 (behavioral and mental health) at 14%. Conclusions “Unable to assess” assignments are related to the given EPA, training year, and timing of assessment within the academic year. CCCs may be less likely to assign EPA supervision levels at timepoints viewed as higher stakes, such as transition from one training year to the next. Future work should explore this further and consider the etiology behind high levels of inability to assess some EPAs in particular.

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

In 2018, Canadian postgraduate emergency medicine (EM) programs began implementing a competency-based medical education (CBME) assessment program. Studies evaluating these programs have focused on broad outcomes using data from national bodies and lack data to support program-specific improvement. We evaluated the implementation of a CBME assessment program within and across programs to identify successes and opportunities for improvement at the local and national levels. Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of entrustable professional activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages were compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Data from 15 of 17 (88%) programs containing 9842 EPA observations from 68 of 77 (88%) EM residents in the 2018 cohort were analyzed. Average numbers of EPAs observed per resident in each program varied from 92.5 to 229.6, correlating with the number of blocks spent on EM and pediatric EM (r = 0.83, P < .001). Relative to the specialty committee's guidelines, residents were promoted later than expected (eg, one-third of residents had a 2-month delay to promotion from the first to second stage) and with fewer EPA observations than suggested. There was demonstrable variation in EPA-based assessment numbers and promotion timelines between programs and with national guidelines.

Constructing Approaches to Entrustable Professional Activity Development that Deliver Valid Descriptions of Professional Practice

Issue: Entrustable Professional Activities (EPAs) describe the core tasks health professionals must be competent performing prior to promotion and/or moving into unsupervised practice. When used for learner assessment, they serve as gateways to increased responsibility and autonomy. It follows that identifying and describing EPAs is a high-stakes form of work analysis aiming to describe the core work of a profession. However, hasty creation and adoption of EPAs without rigorous attention to content threatens the quality of judgments subsequently made from using EPA-based assessment tools. There is a clear need for approaches to identify validity evidence for EPAs themselves prior to their deployment in workplace-based assessment. Evidence: For EPAs to realize their potential in health professions education, they must first be constructed to reflect accurately the work of that profession or specialty. If the EPAs fail to do so, they cannot predict a graduate’s readiness for or future performance in professional practice. Evaluating the methods used for identification, description, and adoption of EPAs through a construct validity lens helps give leaders and stakeholders of EPA development confidence that the EPAs constructed are, in fact, an accurate representation of the profession’s work. Implications: Application of a construct validity lens to EPA development impacts all five commonly followed steps in EPA development: selection of experts; identification of candidate EPAs; iterative revisions; evaluation of proposed EPAs; and formal adoption of EPAs into curricula. It allows curricular developers to avoid pitfalls, bias, and common mistakes. Further, construct validity evidence for EPA development provides assurance that the EPAs adopted are appropriate for use in workplace-based assessment and entrustment decision-making.

MP17: Evaluation of a national competency-based assessment system in emergency medicine: A CanDREAM study

Introduction: In 2018, Canadian postgraduate specialist Emergency Medicine (EM) programs began implementing a competency-based medical education (CBME) assessment system. To support improvement of this assessment program, we sought to evaluate its short-term educational outcomes nationally and within individual programs. Methods: Program-level data from the 2018 resident cohort were amalgamated and analyzed. The number of Entrustable Professional Activity (EPA) assessments (overall and for each EPA) and the timing of resident promotion through program stages was compared between programs and to the guidelines provided by the national EM specialty committee. Total EPA observations from each program were correlated with the number of EM and pediatric EM rotations. Results: Data from 15 of 17 (88.2%) EM programs containing 9,842 EPA observations from 68 of the 77 (88.3%) Canadian EM specialist residents in the 2018 cohort were analyzed. The average number of EPAs observed per resident in each program varied from 92.5 to 229.6 and correlated strongly with the number of blocks spent on EM and pediatric EM (r = 0.83, p &lt; 0.001). Relative to the guidelines outlined by the specialty committee, residents were promoted later than expected and with fewer EPA observations than suggested. Conclusion: We present a new approach to the amalgamation of national and program-level assessment data. There was demonstrable variation in both EPA-based assessment numbers and promotion timelines between programs and with national guidelines. This evaluation data will inform the revision of local programs and national guidelines and serve as a starting point for further reaching outcome evaluation. This process could be replicated by other national assessment programs.

Developing an Entrustment Process: Insights from the AAMC CoreEPA Pilot.

One of the main goals of the CoreEPA pilot has been to determine the feasibility of developing a process to make summative entrustment decisions regarding entrustable professional activities (EPAs). Five years into the pilot, we report results of a research study we conducted to explore approaches to the entrustment process undertaken by our ten participating schools. We sought to identify the choices that participating schools made regarding the entrustment process and why these decisions were made. We are sharing these results, highlighting ongoing challenges that were identified with the intent of helping other medical schools that are moving toward EPA-based assessment. We conducted semi-structured interviews with representatives of all 10 medical schools in the CoreEPA pilot to understand their choices in designing the entrustment process. Additional information was obtained through follow-up communication to ensure completeness and accuracy of the findings. Several common themes are described. Our results indicate that, while approaches to the entrustment process vary considerably, all schools demonstrated consistent adherence to the guiding principles of the pilot. Several common barriers to the entrustment process emerged, and there was a consensus that more experience is needed with the process before consequential entrustment decisions can be made. The CoreEPA pilot schools continue to address challenges identified in implementing entrustment processes and making entrustment decisions for our students graduating in the Class of 2020.

The impact of entrustment assessments on feedback and learning: Trainee perspectives.

Assessment for and of learning in workplace settings is at the heart of competency-based medical education. In postgraduate medical education (PGME), entrustable professional activities (EPAs) and entrustment scales are increasingly used to assess competence. However, the educational impacts of these assessment approaches remain unknown. Therefore, this study aimed to explore trainee perceptions regarding the impacts of EPAs and entrustment scales on feedback and learning processes in the clinical setting. Four focus groups were conducted with postgraduate trainees in anaesthesia, emergency medicine, general internal medicine and nephrology at McMaster University in Hamilton, Ontario, Canada. Data collection and analysis were informed by principles of constructivist grounded theory. Entrustable professional activities representing well-defined tasks are perceived as potentially effective drivers for feedback and learning. Use of EPAs and entrustment scales, however, may augment existing tensions between developmental (for learning) and decision-making (of learning) assessment functions. Three key dilemmas seem to influence the impact of EPA-based assessment approaches on residents' learning: (a) standardisation of outcomes versus flexibility in assessment to align with individual learning experiences; (b) assessment tasks focusing on performance standards versus opportunities for learning, and (c) feedback focusing on numeric entrustment scores versus narrative and dialogue. Use of entrustment as an assessment outcome may impact trainees' motivation and feelings of self-efficacy, further enhancing tensions between learning and performance. Entrustable professional activities and entrustment scales may support assessment for learning in PGME. However, their successful implementation requires the careful management of dilemmas that arise in EPA-based assessment in order to support competence development.