EPA-based Assessment Research Articles

Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities

Entrustable professional activities (EPAs) are an emerging workplace-based, patient-oriented assessment approach with limited empirical evidence. To measure the development of pediatric trainees' clinical skills over time using EPA-based assessment data. Prospective cohort study of categorical pediatric residents over 3 academic years (2015-2016, 2016-2017, and 2017-2018) assessed on 17 American Board of Pediatrics EPAs. Residents in training at 23 pediatric residency programs in the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network were included. Assessment was conducted by clinical competency committee members, who made summative assessment decisions regarding levels of supervision required for each resident and each EPA. Data were collected from May 2016 to November 2018 and analyzed from November to December 2018. Longitudinal, prospective assessment using EPAs. Trajectories of supervision levels by EPA during residency training and how often graduating residents were deemed ready for unsupervised practice in each EPA. Across the 5 data collection cycles, 1987 residents from all 3 postgraduate years in 23 residency programs were assigned 25 503 supervision level reports for the 17 general pediatrics EPAs. The 4 EPAs that required the most supervision across training were EPA 14 (quality improvement) on the 5-level scale (estimated mean level at graduation, 3.7; 95% CI, 3.6-3.7) and EPAs 8 (transition to adult care; mean, 7.0; 95% CI, 7.0-7.1), 9 (behavioral and mental health; mean, 6.6; 95% CI, 6.5-6.6), and 10 (resuscitate and stabilize; mean, 6.9; 95% CI, 6.8-7.0) on the expanded 5-level scale. At the time of graduation (36 months), the percentage of trainees who were rated at a supervision level corresponding to "unsupervised practice" varied by EPA from 53% to 98%. If performance standards were set to align with 90% of trainees achieving the level of unsupervised practice, this standard would be met for only 8 of the 17 EPAs (although 89% met this standard for EPA 17, performing the common procedures of the general pediatrician). This study presents initial evidence for empirically derived practice readiness and sets the stage for identifying curricular gaps that contribute to discrepancy between observed practice readiness and standards needed to produce physicians able to meet the health needs of the patient populations they serve. Future work should compare these findings with postgraduation outcomes data as a means of seeking validity evidence.

Using an Entrustable Professional Activity to Assess Consultation Requests Called on an Internal Medicine Teaching Service.

The Accreditation Council for Graduate Medical Education's milestones require internal medicine residents to have competency in calling consults. Based on a literature review, we developed an Entrustable Professional Activity (EPA) to delineate the knowledge, skills, and attitudes required for a consultation request and, building on the EPA, implemented an assessment instrument to provide feedback to interns calling consultation requests and assess the quality of their consult questions and the level of supervision required in performing this milestone. Assessments were done on internal medicine inpatient teaching services. Consultation requests were performed by interns and observed by residents using the assessment instrument. Feedback was provided to the interns. Interns then completed a self-reflection instrument based on the feedback. Twenty-six paired observations were collected over three 1-month rotations. There was a moderate positive correlation (r = .43) comparing resident and intern responses to how they felt about the intern's ability to make a consultation request. There was a strong positive correlation (r = .65) comparing resident opinion of how strong the intern's ability in calling a consult to how well the consult question used the PICO (patient, intervention, comparators, outcomes of interest) framework. Twenty-five out of 28 interns (89%) said they would make a change during their next consultation request due to feedback from their resident. Our EPA-based assessment instrument provided an opportunity to give interns feedback and to assess the quality of the consultation requests they made.

Introducing an assessment tool based on a full set of end-of-training EPAs to capture the workplace performance of final-year medical students

BackgroundWhile literature on the theoretical value of entrustable professional activities (EPAs) for assessment is rapidly expanding, little experience exists on its application. The aims of this study are to develop and explore the utility of an EPA-based assessment tool for capturing the workplace performance of final-year medical students based on a full set of end-of-training EPAs.MethodsThe tool was developed in a systematic iterative process. Twelve 12 end-of-undergraduate medical training EPAs were nested into 72 smaller EPAs and cross-mapped onto a 6-point supervision level scale, both adjusted to the context of final-year clerkships. One version was created for students’ self-assessment of their ability to carry out tasks and their history of carrying out tasks, and another version was created for supervisors’ assessment of students’ ability to carry out tasks. The tool was administered to final-year clerkship students and their clinical supervisors to explore its utility as an assessment approach. The results were analysed using descriptive and interferential statistics.ResultsWe enrolled a total of 60 final-year medical students. For 33 students, ratings were provided from one supervisor and for 27 students from two supervisors. With regard to the reliability and validity of the tool, students’ and supervisors’ ratings showed an overall good internal consistency as well as variability between and within the EPAs. Over the full EPA range, students rated their ability to perform a task slightly higher than their task performance history and slightly lower than the supervisors’ ratings. Students’ self-ratings of their ability to perform a task correlated with their history in performing the task. Supervisors’ ratings correlated among supervisors and not with students’ ratings. Concerning educational outcomes, supervisors’ average rating of students’ ability to perform the EPAs without direct supervision was 64%, and key findings being double-checked.ConclusionsThis study introduces a tool that is adjusted to the final-year clerkship context and can assess the workplace performance of trainees based on a full set of end-of-training EPAs. Its utility characteristics suggest that the tool may be employed as a formative and outcome-aligned approach to the assessment of final-year students before entering into residency.

Impact of an entrustable professional activities-based assessment system

Background: Beginning in 2014, all Accreditation Counsel of Graduate Medical Education (ACGME) accredited residency programs were required to move to a Milestones-based system for biannual resident assessment. The resident assessment system for the Virginia Tech-Carilion Obstetrics and Gynecology (OB/GYN) residency program was re-designed to meet this requirement in July, 2014. The ACGME Milestones based assessment tool was identified on multiple faculty surveys as an area for improvement. To address this issue, an entrustable professional activities (EPA) based assessment system was designed and implemented for assessment of all OB/GYN rotations. Objective: To evaluate the impact of an EPA based resident assessment system on faculty member’s evaluation of resident assessment tools. Methods: In this prospective quality improvement study, a survey was sent to all faculty members prior to the implementation of the EPA-based assessment system. The same survey was performed three months after the implementation of the new system. To facilitate analysis, each level of agreement was assigned a numerical value (1-5). The results were aggregated, and were analyzed using t-tests, assuming unequal variances. Results: Sixty-eight percent of the faculty responded to the first survey, and 67% responded to the follow up survey. Statistically significant (p<.05) improvements were noted in most measures of the EPA based assessment tool including “ease of use” (2.2 vs 4.4, p< 0.001) and “accurate representation of resident performance” (2.5 vs 3.9, p <0.001). Conclusion: An EPA based resident evaluation system significantly improved teaching faculty’s impression of most domains of our OB/GYN resident assessment tools.

Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional Activities.

Entrustable professional activities (EPAs) represent the routine and essential activities that physicians perform in practice. Although some level of supervision scales have been proposed, they have not been validated. In this study, the investigators created level of supervision scales for EPAs common to the pediatric subspecialties and then examined their validity in a study conducted by the Subspecialty Pediatrics Investigator Network (SPIN). SPIN Steering Committee members used a modified Delphi process to develop unique scales for six of the seven common EPAs. The investigators sought validity evidence in a multisubspecialty study in which pediatric fellowship program directors and Clinical Competency Committees used the scales to evaluate fellows in fall 2014 and spring 2015. Separate scales for the six EPAs, each with five levels of progressive entrustment, were created. In both fall and spring, more than 300 fellows in each year of training from over 200 programs were assessed. In both periods and for each EPA, there was a progressive increase in entrustment levels, with second-year fellows rated higher than first-year fellows (P < .001) and third-year fellows rated higher than second-year fellows (P < .001). For each EPA, spring ratings were higher (P < .001) than those in the fall. Interrater reliability was high (Janson and Olsson's iota = 0.73). The supervision scales developed for these six common pediatric subspecialty EPAs demonstrated strong validity evidence for use in EPA-based assessment of pediatric fellows. They may also inform the development of scales in other specialties.

Entrustable professional activities: A useful concept for neurology education.

Medical education is currently undergoing a paradigm shift from process-based to competency-based education, focused on measuring the desired competence of a physician. In an attempt to improve the assessment framework used for medical education, the concept of entrustable professional activities (EPAs) has gained traction. EPAs are defined as professional activities that can be entrusted to an individual in a clinical context. The Association of American Medical Colleges (AAMC) defined a set of 13 such EPAs to define the core of what all students should be able to do on day 1 of residency, regardless of specialty choice. The AAMC is currently piloting these EPAs with 10 medical schools to determine if EPAs can be used as a way to observe, measure, and entrust medical students with core clinical activities by the end of the clinical immersion experiences of the third year. The specialty of pediatrics is piloting the use of specialty-specific EPAs at 5 medical schools to assess readiness for transitions from medical school into pediatric residency training and practice. To date, no neurology-specific EPAs have been published for use in neurology clerkships or neurology residencies. This article introduces the concept of EPAs in the context of competency-based medical education and describes how EPAs might be relevant and applicable in neurologic education across the continuum. The Undergraduate Education Subcommittee of the American Academy of Neurology advocates for a proactive approach to incorporating core EPAs in undergraduate medical education and to considering an EPA-based specialty-specific assessment framework for neurology.

Using the Entrustable Professional Activities Framework in the Assessment of Procedural Skills.

The entrustable professional activity (EPA) framework has been identified as a useful approach to assessment in competency-based education. To apply an EPA framework for assessment, essential skills necessary for entrustment to occur must first be identified. Using an EPA framework, our study sought to (1) define the essential skills required for entrustment for 7 bedside procedures expected of graduates of Canadian internal medicine (IM) residency programs, and (2) develop rubrics for the assessment of these procedural skills. An initial list of essential skills was defined for each procedural EPA by focus groups of experts at 4 academic centers using the nominal group technique. These lists were subsequently vetted by representatives from all Canadian IM training programs through a web-based survey. Consensus (more than 80% agreement) about inclusion of each item was sought using a modified Delphi exercise. Qualitative survey data were analyzed using a framework approach to inform final assessment rubrics for each procedure. Initial lists of essential skills for procedural EPAs ranged from 10 to 24 items. A total of 111 experts completed the national survey. After 2 iterations, consensus was reached on all items. Following qualitative analysis, final rubrics were created, which included 6 to 10 items per procedure. These EPA-based assessment rubrics represent a national consensus by Canadian IM clinician educators. They provide a practical guide for the assessment of procedural skills in a competency-based education model, and a robust foundation for future research on their implementation and evaluation.

The ABCs of entrustable professional activities: an overview of 'entrustable professional activities' in medical education.

Consultants regularly need to decide whether a trainee can be entrusted to perform a clinical activity independently. 'Entrustable professional activities' (EPA) provide a framework for justifying and better utilising supervisor entrustment decisions for trainee feedback and assessment in the workplace. Since being proposed by Olle ten Cate in 2005, EPA are emerging as an integral part of many international medical curricula, and are being considered by the Royal Australasian College of Physicians in the current review of physician training. EPA are defined as tasks or responsibilities that can be entrusted to a trainee once sufficient competence is reached to allow for unsupervised practice. An example might be to entrust a trainee to 'Initiate and co-ordinate care of the palliative patient' with only off-site or indirect supervision. Rather than attempting to measure directly each of the many separate competencies required to undertake such a complex task, EPA direct the trainee and supervisor's attention to the trainee's performance in a limited number of selected, representative, important day-to-day activities. EPA-based assessment is gaining momentum, amongst significant concerns regarding feasibility of implementation. While the optimal process for designing and implementing EPA remains to be determined, it is an assessment strategy where the over-arching goal of optimal patient care remains in clear sight. This review explores the central role of trust in medical training, the case for EPA and potential barriers to implementing EPA-based assessment.

Building the Plane As We Fly It.

Building the Plane As We Fly It.

Developing Entrustable Professional Activities as the Basis for Assessment of Competence in an Internal Medicine Residency: A Feasibility Study

Graduate medical education programs assess trainees' performance to determine readiness for unsupervised practice. Entrustable professional activities (EPAs) are a novel approach for assessing performance of core professional tasks. To describe a pilot and feasibility evaluation of two EPAs for competency-based assessment in internal medicine (IM) residency. Post-graduate year-1 interns (PGY-1s) and attendings at a large internal medicine (IM) residency program. Two Entrustable professional activities (EPA) assessments (Discharge, Family Meeting) were piloted. Twenty-eight out of 43 (65.1 %) PGY-1 s and 32/43 (74.4 %) attendings completed surveys about the Discharge EPA experience. Most who completed the EPA assessment (10/12, 83.8 %, PGY-1s; 9/11, 83.3 %, attendings) agreed it facilitated useful feedback discussions. For the Family Meeting EPA, 16/26 (61.5 %) PGY-1s completed surveys, and most who participated (9/12 PGY1s, 75 %) reported it improved attention to family meeting education, although only half recommended continuing the EPA assessment. From piloting two EPA assessments in a large IM residency, we recognized our reminder systems and time dedicated for completing EPA requirements as inadequate. Collaboration around patient safety and palliative care with relevant clinical services has enhanced implementation and buy-in. We will evaluate how well EPA-based assessment serves the intended purpose of capturing trainees' trustworthiness to conduct activities unsupervised.