Assessment Of Entrustable Professional Activities Research Articles

American Dental Education Association Compendium Entrustable Professional Activities Workgroup report.

Entrustable professional activities (EPAs) are discrete clinical tasks that can be evaluated to help define readiness for independent practice in the health professions and are intended to increase trust in the dental graduate. EPAs provide a framework that bridges competencies to clinical practice. This report describes the work of the American Dental Education Association (ADEA) Compendium EPA Workgroup to develop a list of EPAs for dental education and supportive resources, including specifications and a glossary. Preliminary work including literature and resource review, mapping of existing competencies, and review of other health professions' EPAs informed the development of our EPAs list. Workgroup members achieved consensus using a modified Delphi process. A Qualtrics survey using a validated rubric for the assessment of EPAs as described in peer-reviewed literature was used. Dental educators, including academic deans, were surveyed for feedback on the content and format of the EPAs. Based on findings in the literature analysis of existing EPAs and competencies in health professions, a list of EPAs was developed along with a description of specifications. The EPA workgroup (nine members from multiple institutions) used the Delphi process in receiving feedback from various experts. A list of 11 core EPAs was vetted by dental educators including academic deans (n=∼23), and the process of development was reviewed by EPAs experts outside dental education. A glossary was developed to align language. These EPAs define the scope of dental practice. This report represents Phase 1 of the EPA framework development and vetting process. Future directions will include a broader vetting of the EPA list, faculty development, and national standardized technology that support this work to optimize implementation.

Evaluating the Quality of Narrative Feedback for Entrustable Professional Activities in a Surgery Residency Program.

We assessed the quality of narrative feedback given to surgical residents during the first five years of Competency-Based Medical Education (CBME) implementation. CBME requires ongoing formative assessments and feedback on learners' performance. We conducted a retrospective cross-sectional study using assessments of Entrustable Professional Activities (EPAs) in the Surgical Foundations curriculum at Queen's University from 2017-2022. Two raters independently evaluated quality of narrative feedback using the Quality of Assessment of Learning (QuAL) Score (0-5). A total of 3,900 EPA assessments were completed over 5 years. Fifty-seven percent (2229/3900) of assessments had narrative feedback documented with a mean QuAL score of 2.16±1.49. Of these, 1614 (72.4%) provided evidence about the resident's performance, 951 (42.7%) provided suggestions for improvement, and 499/2229 (22.4%) connected suggestions to the evidence. There was no meaningful change in narrative feedback quality over time (r=0.067, P=0.002). Variables associated with lower quality of narrative feedback include: Attending role (2.04±1.48) compared to medical student (3.13±1.12, P<0.001) and clinical fellow (2.47±1.54, P<0.001), concordant specialties between the assessor and learner (2.06±1.50 vs. 2.21±1.49, P=0.025), completion of the assessment one month or more after the encounter versus one week (1.85±1.48 vs. 2.23±1.49, P<0.001), and resident entrustment versus not entrusted to perform the assessed EPA (2.13±1.45 vs. 2.35±1.66; P=0.008). The quality of narrative feedback was similar for assessments completed under direct and indirect observation (2.18±1.47 vs. 2.06±1.54; P=0.153). Just over half of the EPA assessments of surgery residents contained narrative feedback with overall fair quality. There was no meaningful change in the quality of feedback over 5 years. These findings prompt future research and faculty development.

Motivations for Entrustable Professional Activity Assessment: Gaps Between Curriculum Theory and Resident Reality

Background Previous research demonstrates mixed reactions from residents toward competency-based medical education (CBME), and entrustable professional activities (EPAs) specifically. However, understanding what motivates residents to obtain EPAs may be vital to the longevity of CBME, given the emphasis on assessment for learning under this paradigm. Objective This study explored resident perspectives across 3 domains: motivation for obtaining EPAs, perceived importance of EPAs, and overall thoughts on CBME curriculum. Methods This was a sequential exploratory mixed-methods study involving 2 phases of data collection. Phase 1 was semi-structured interviews with residents enrolled in CBME at one Canadian institution from November 2019 to July 2020. Analyses included thematic and manifest content analysis. Phase 2 was an electronic close-ended survey to capture residents’ primary motivation for requesting EPAs and importance of EPAs for learning. Survey data were analyzed descriptively. Results Of 120 eligible residents, 25 (21%) and 107 (89%) participated in the interview and survey, respectively. Program requirement was the dominant motivation for obtaining EPAs. There was variability in perceived importance of EPAs on learning. Increased resident workload, gaming the system to maximize EPA scores, and lack of shared ownership from preceptors were cited as critiques of the curriculum. Survey responses corroborated interview findings. Conclusions Although many residents recognize the value of EPAs, the majority are not intrinsically motivated to seek out assessment under the current CBME framework.

An Analysis of Workplace-Based Assessments for Core Entrustable Professional Activities for Entering Residency: Does Type of Clinical Assessor Influence Level of Supervision Ratings?

The authors describe use of the workplace-based assessment (WBA) coactivity scale according to entrustable professional activities (EPAs) and assessor type to examine how diverse assessors rate medical students using WBAs. A WBA data collection system was launched at Oregon Health and Science University to visualize learner competency in various clinical settings to foster EPA assessment. WBA data from January 14 to June 18, 2021, for medical students (all years) were analyzed. The outcome variable was level of supervisor involvement in each EPA, and the independent variable was assessor type. A total of 7,809 WBAs were included. Most fourth-, third-, and second-year students were assessed by residents or fellows (755 [49.5%], 1,686 [48.5%], and 918 [49.9%], respectively) and first-year students by attending physicians (803 [83.0%]; P < .001). Attendings were least likely to use the highest rating of 4 (1 was available just in case; 2,148 [56.7%] vs 2,368 [67.7%] for residents; P < .001). Learners more commonly sought WBAs from attendings for EPA 2 (prioritize differential diagnosis), EPA 5 (document clinical encounter), EPA 6 (provide oral presentation), EPA 7 (form clinical questions and retrieve evidence-based medicine), and EPA 12 (perform general procedures of a physician). Residents and fellows were more likely to assess students on EPA 3 (recommend and interpret diagnostic and screening tests), EPA 4 (enter and discuss orders and prescriptions), EPA 8 (give and receive patient handover for transitions in care), EPA 9 (collaborate as member of interprofessional team), EPA 10 (recognize and manage patient in need of urgent care), and EPA 11 (obtain informed consent). Learners preferentially sought resident vs attending supervisors for different EPA assessments. Future research should investigate why learners seek different assessors more frequently for various EPAs and if assessor type variability in WBA levels holds true across institutions.

Validity of Entrustable Professional Activities in a National Sample of General Surgery Residency Programs.

The American Board of Surgery has endorsed competency-based education as vital to the assessment of surgical training. From 2018 to 2020, a national pilot study was conducted at 28 general surgery programs to evaluate feasibility of implementing entrustable professional activities (EPAs) for 5 common general surgical conditions. ACGME core competency Milestones were also rated for each resident by program clinical competency committees. This study aimed to evaluate the validity of general surgery EPAs compared with Milestones. Prospectively collected, de-identified EPA Pilot Study data were analyzed. EPAs studied were EPA-1 (gallbladder), EPA-2 (inguinal hernia), EPA-3 (right lower quadrant pain), EPA-4 (trauma), and EPA-5 (consult). Variables abstracted included levels of EPA entrustment (1 to 5) and corresponding ACGME Milestone subcompetency ratings (1 to 5) for the same study cycle. Spearman's correlations were used to evaluate the relationship between summative EPA scores and corresponding Milestone ratings. A total of 493 unique residents received a summative entrustment decision. EPA summative entrustment scores had moderate-to-strong positive correlation with mapped Milestone subcompetencies, with median rho value of 0.703. Among operation-focused EPAs, median rho values were similar between EPA-1 (0.688) and EPA-2 (0.661), but higher for EPA-3 (0.833). EPA-4 showed a strong positive correlation with diagnosis and communication competencies (0.724), whereas EPA-5, mapped to the most Milestone subcompetencies, had the lowest median rho value (0.455). Moderate-to-strong positive correlation was noted between EPAs and patient care, medical knowledge, and communication Milestones. These findings support the validity of EPAs in general surgery and suggest that EPA assessments can be used to inform Milestone ratings by clinical competency committees.

“Doing well”: Intraoperative entrustable professional activity assessments provided limited technical feedback

BackgroundEntrustable Professional Activities (EPAs) allow for the assessment of specific, observable, essential tasks in medical education. Since being developed in non-surgical fields, EPA assessments have been implemented in surgery to explore intraoperative entrustment. However, assessment burden is a significant problem for faculty, and it is unknown whether EPA assessments enable formative technical feedback. EPAs' formative utility could inform how surgical programs facilitate technical feedback for trainees. We aimed to assess the extent to which narrative comments provided through the Fellowship Council (FC) EPA assessments contained technical feedback. MethodsThe FC previously collected EPA assessments for subspecialty surgical fellows from September 2020 to October 2022. Two raters reviewed assessments' narrative comments for inclusion of each skill area that makes up part of the Objective Structured Assessment of Technical Skills (OSATS). A third rater reconciled discrepant ratings. ResultsDuring the study period, there were 3302 completed EPA assessments, including 1191 fellow self-assessments, 1124 faculty assessments, and 987 assessments without an identified assessor role. We found that assessments' narrative comments related to a median of two of the seven OSATS areas (IQR:1–2). There were no comments relevant to any of the seven OSATS areas in 16.0 % of all assessments. ConclusionsIn this review of narrative comments for EPA assessments from the FC, we found that limited technical feedback of the kind included in the OSATS was provided in many assessments. These results suggest benefit to adjusting the EPA form, enhancing faculty development, or continuing additional types of targeted technical assessment intraoperatively. Key messageThis analysis of narrative comments from fellowship EPA assessments showed that many assessments included limited technical feedback. To allow for continued technical feedback for fellows, these results highlight the need for further refinements of the EPA assessment form, additional faculty development, or ongoing use of other types of technical assessment.

Current use of simulation for EPA assessment in emergency medicine.

Approximately five years ago, the Royal College emergency medicine programs in Canada implemented a competency-based paradigm and introduced the use of Entrustable Professional Activities (EPAs) for assessment of units of professional activity to assess trainees. Many competency-based medical education (CBME) based curricula, involve assessing for entrustment through observations of EPAs. While EPAs are frequently assessed in clinical settings, simulation is also used. This study aimed to characterize the use of simulation for EPA assessment. A study interview guide was jointly developed by all study authors and followed best practices for survey development. A national interview was conducted with program directors or assistant program directors across all the Royal College emergency medicine programs across Canada. Interviews were conducted over Microsoft Teams, interviews were recorded and transcribed, using Microsoft Teams transcribing service. Sample transcripts were analyzed for theme development. Themes were then reviewed by co-authors to ensure they were representative of the participants' views. A 64.7% response rate was achieved. Simulation has been widely adopted by EM training programs. All interviewees demonstrated support for the use of simulation for EPA assessment for many reasons, however, PDs acknowledged limitations and thematic analysis revealed certain themes and tensions for using simulation for EPA assessment. Thematic analysis revealed six major themes: widespread support for the use of simulation for EPA assessment, concerns regarding the potential for EPA assessment to become a "tick- box" exercise, logistical barriers limiting the use of simulation for EPA assessment, varied perceptions about the authenticity of using simulation for EPA assessment, the potential for simulation for EPA assessment to compromise learner psychological safety, and suggestions for the optimization of use of simulation for EPA assessment. Our findings offer insight for other programs and specialties on how simulation for EPA assessment can best be utilized. Programs should use these findings when considering using simulation for EPA assessment.

Design and Implementation of a National Program of Assessment Model - Integrating Entrustable Professional Activity Assessments in Canadian Specialist Postgraduate Medical Education.

Traditional approaches to assessment in health professions education systems, which have generally focused on the summative function of assessment through the development and episodic use of individual high-stakes examinations, may no longer be appropriate in an era of competency based medical education. Contemporary assessment programs should not only ensure collection of high-quality performance data to support robust decision-making on learners' achievement and competence development but also facilitate the provision of meaningful feedback to learners to support reflective practice and performance improvement. Programmatic assessment is a specific approach to designing assessment systems through the intentional selection and combination of a variety of assessment methods and activities embedded within an educational framework to simultaneously optimize the decision-making and learning function of assessment. It is a core component of competency based medical education and is aligned with the goals of promoting assessment for learning and coaching learners to achieve predefined levels of competence. In Canada, postgraduate specialist medical education has undergone a transformative change to a competency based model centred around entrustable professional activities (EPAs). In this paper, we describe and reflect on the large scale, national implementation of a program of assessment model designed to guide learning and ensure that robust data is collected to support defensible decisions about EPA achievement and progress through training. Reflecting on the design and implications of this assessment system may help others who want to incorporate a competency based approach in their own country.

Exploring the Quality of Feedback in Entrustable Professional Activity Narratives Across 24 Residency Training Programs.

Background Competency-based medical education (CBME) has been implemented in many residency training programs across Canada. A key component of CBME is documentation of frequent low-stakes workplace-based assessments to track trainee progression over time. Critically, the quality of narrative feedback is imperative for trainees to accumulate a body of evidence of their progress. Suboptimal narrative feedback will challenge accurate decision-making, such as promotion to the next stage of training. Objective To explore the quality of documented feedback provided on workplace-based assessments by examining and scoring narrative comments using a published quality scoring framework. Methods We employed a retrospective cohort secondary analysis of existing data using a sample of 25% of entrustable professional activity (EPA) observations from trainee portfolios from 24 programs in one institution in Canada from July 2019 to June 2020. Statistical analyses explore the variance of scores between programs (Kruskal-Wallis rank sum test) and potential associations between program size, CBME launch year, and medical versus surgical specialties (Spearman's rho). Results Mean quality scores of 5681 narrative comments ranged from 2.0±1.2 to 3.4±1.4 out of 5 across programs. A significant and moderate difference in the quality of feedback across programs was identified (χ2=321.38, P<.001, ε2=0.06). Smaller programs and those with an earlier launch year performed better (P<.001). No significant difference was found in quality score when comparing surgical/procedural and medical programs that transitioned to CBME in this institution (P=.65). Conclusions This study illustrates the complexity of examining the quality of narrative comments provided to trainees through EPA assessments.

Preceptors' perceptions of an entrustable professional activities-based community introductory pharmacy practice experience curriculum

IntroductionIn 2021, the Southeastern Pharmacy Experiential Education Consortium implemented Entrustable Professional Activities (EPAs) into the community introductory pharmacy practice experience (IPPE) curriculum at five colleges/schools of pharmacy. The objective of this study was to evaluate community IPPE preceptors' perceptions regarding the newly implemented EPA-based community IPPE curriculum and corresponding preceptor training. MethodsCommunity IPPE preceptors who precepted first-year student pharmacists in the EPA-based curriculum during the 2021 and 2022 community IPPE cycles were invited to complete a voluntary electronic survey. The survey collected preceptor feedback regarding the required EPA tasks, assessment tool, and preceptor development module. ResultsEighty-eight preceptors began the survey, and approximately half completed the entire survey. Greater than 92% of preceptors surveyed agreed or strongly agreed EPA domain tasks were developed at an appropriate level for a student to complete by the end of the community IPPE, and ≥ 94% agreed or strongly agreed tasks prepared a community IPPE student for the community advanced pharmacy practice experience (APPE). Overall, most preceptors agreed or strongly agreed that the assessment tool was easy to navigate and effective at evaluating students' performance. All preceptors who viewed the recorded preceptor development module found it helpful to their understanding of the new assessment tool. ConclusionsPreceptors' feedback supports the use of an EPA-based community IPPE curriculum to assess student performance and prepare students for community APPEs. Preceptor involvement is valuable in the evaluation of a revised experiential curricula to assure IPPE expectations are appropriate and align with contemporary pharmacy practice.

Using Natural Language Processing to Evaluate the Quality of Supervisor Narrative Comments in Competency-Based Medical Education.

Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments. Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores. All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively. This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels.

Only as Strong as the Weakest Link: Resident Perspectives on Entrustable Professional Activities and Their Impact on Learning.

Background Core to competency-based medical education (CBME) is the use of frequent low-stakes workplace-based assessments. In the Canadian context, these observations of performance are framed around entrustable professional activities (EPAs). Objective We aimed to explore residents' real-world perspectives of EPAs and their perceived impact on learning, because assessments perceived to be "inauthentic," or not truly reflective of their lived experiences, may interfere with learning. Methods Using constructivist grounded theory, we conducted 18 semistructured interviews in 2021 with residents from all programs that had implemented CBME at one tertiary care academic center in Canada. Participants were recruited via email through respective program administrators. Data collection and analysis occurred iteratively, and categories were identified using constant comparative analysis. Results Residents were strikingly polarized, perceiving EPAs as either a valuable opportunity for professional growth or as an onerous requirement that interfered with learning. Regardless of what view participants held, all perspectives were informed by: (1) the program administration and the perceived messaging from program to residents; (2) faculty assessors and their perceived degree of engagement, or "buy-in" with the EPA system; and ultimately (3) learner behavior. We theorized from these findings that all 3 aspects must be working in tandem for the assessment system to function as intended. Conclusions From the learners' perspective, there exists a dynamic, interdependent relationship between the 3 CBME stakeholders. As such, the perceived value of the EPA assessment system can only be as strong as the weakest link in the chain.

Feedback that Lands: Exploring How Residents Receive and Judge Feedback During Entrustable Professional Activities.

Receiving feedback from different types of assessors (e.g., senior residents, staff supervisors) may impact trainees' perceptions of the quantity and quality of data during entrustable professional activity (EPA) assessments. We evaluated the quality of EPA feedback provided by different assessors (senior residents, chief medical residents/subspecialty residents, and staff) and explored residents' judgements of the value of this feedback. From a database of 2228 EPAs, we calculated the frequency of contribution from three assessor groups. We appraised the quality of 60 procedure-related EPAs completed between July 2019 and March 2020 using a modified Completed Clinical Evaluation Report Rating (CCERR) tool. Next, we asked 15 internal medicine residents to sort randomly selected EPAs according to their judgements of value, as an elicitation exercise before a semi-structured interview. Interviews explored participants' perceptions of quality of written feedback and helpful assessors. Residents completed over 60% of EPA assessments. We found no difference in modified-CCERR scores between the three groups. When judging EPA feedback value, residents described a process of weighted deliberation, considering perceived assessor characteristics (e.g., credibility, experience with EPA system), actionable written comments, and their own self-assessment. Like other recent studies, we found that residents contributed most to procedure-related EPA assessments. To the established list of factors influencing residents' judgements of feedback value, we add assessors' adherence to, and their shared experiences of being assessed within, EPA assessment systems. We focus on the implications for how assessors and leaders can build credibility in themselves and in the practices of EPA assessments.

The Quality of Assessment for Learning score for evaluating written feedback in anesthesiology postgraduate medical education: a generalizability and decision study.

Competency based residency programs depend on high quality feedback from the assessment of entrustable professional activities (EPA). The Quality of Assessment for Learning (QuAL) score is a tool developed to rate the quality of narrative comments in workplace-based assessments; it has validity evidence for scoring the quality of narrative feedback provided to emergency medicine residents, but it is unknown whether the QuAL score is reliable in the assessment of narrative feedback in other postgraduate programs. Fifty sets of EPA narratives from a single academic year at our competency based medical education post-graduate anesthesia program were selected by stratified sampling within defined parameters [e.g. resident gender and stage of training, assessor gender, Competency By Design training level, and word count (≥17 or <17 words)]. Two competency committee members and two medical students rated the quality of narrative feedback using a utility score and QuAL score. We used Kendall's tau-b co-efficient to compare the perceived utility of the written feedback to the quality assessed with the QuAL score. The authors used generalizability and decision studies to estimate the reliability and generalizability coefficients. Both the faculty's utility scores and QuAL scores (r = 0.646, p < 0.001) and the trainees' utility scores and QuAL scores (r = 0.667, p < 0.001) were moderately correlated. Results from the generalizability studies showed that utility scores were reliable with two raters for both faculty (Epsilon=0.87, Phi=0.86) and trainees (Epsilon=0.88, Phi=0.88). The QuAL score is correlated with faculty- and trainee-rated utility of anesthesia EPA feedback. Both faculty and trainees can reliability apply the QuAL score to anesthesia EPA narrative feedback. This tool has the potential to be used for faculty development and program evaluation in Competency Based Medical Education. Other programs could consider replicating our study in their specialty.

Usability of ENTRUST as an Assessment Tool for Entrustable Professional Activities (EPAs): A Mixed Methods Analysis

As the American Board of Surgery transitions to a competency-based model of surgical education centered upon entrustable professional activities (EPAs), there is a growing need for objective tools to determine readiness for entrustment. This study evaluates the usability of ENTRUST, an innovative virtual patient simulation platform to assess surgical trainees' decision-making skills in preoperative, intra-operative, and post-operative settings. This is a mixed-methods analysis of the usability of the ENTRUST platform. Quantitative data was collected using the system usability scale (SUS) and Likert responses. Analysis was performed with descriptive statistics, bivariate analysis, and multivariable linear regression. Qualitative analysis of open-ended responses was performed using the Nielsen-Shneiderman Heuristics framework. This study was conducted at an academic institution in a proctored exam setting. The analysis includes n = 47 (PGY 1-5) surgical residents who completed an online usability survey following the ENTRUST Inguinal Hernia EPA Assessment. The ENTRUST platform had a median SUS score of 82.5. On bivariate and multivariate analyses, there were no significant differences between usability based on demographic characteristics (all p > 0.05), and SUS score was independent of ENTRUST performance (r = 0.198, p = 0.18). Most participants agreed that the clinical workup of the patient was engaging (91.5%) and felt realistic (85.1%). The most frequent heuristics represented in the qualitative analysis included feedback, visibility, match, and control. Additional themes of educational value, enjoyment, and ease-of-use highlighted participants' perspectives on the usability of ENTRUST. ENTRUST demonstrates high usability in this population. Usability was independent of ENTRUST score performance and there were no differences in usability identified in this analysis based on demographic subgroups. Qualitative analysis highlighted the acceptability of ENTRUST and will inform ongoing development of the platform. The ENTRUST platform holds potential as a tool for the assessment of EPAs in surgical residency programs.

Development and Pilot Testing of a Programmatic System for Competency Assessment in US Anesthesiology Residency Training.

In 2018, a set of entrustable professional activities (EPAs) and procedural skills assessments were developed for anesthesiology training, but they did not assess all the Accreditation Council for Graduate Medical Education (ACGME) milestones. The aims of this study were to (1) remap the 2018 EPA and procedural skills assessments to the revised ACGME Anesthesiology Milestones 2.0, (2) develop new assessments that combined with the original assessments to create a system of assessment that addresses all level 1 to 4 milestones, and (3) provide evidence for the validity of the assessments. Using a modified Delphi process, a panel of anesthesiology education experts remapped the original assessments developed in 2018 to the Anesthesiology Milestones 2.0 and developed new assessments to create a system that assessed all level 1 through 4 milestones. Following a 24-month pilot at 7 institutions, the number of EPA and procedural skill assessments and mean scores were computed at the end of the academic year. Milestone achievement and subcompetency data for assessments from a single institution were compared to scores assigned by the institution's clinical competency committee (CCC). New assessment development, 2 months of testing and feedback, and revisions resulted in 5 new EPAs, 11 nontechnical skills assessments (NTSAs), and 6 objective structured clinical examinations (OSCEs). Combined with the original 20 EPAs and procedural skills assessments, the new system of assessment addresses 99% of level 1 to 4 Anesthesiology Milestones 2.0. During the 24-month pilot, aggregate mean EPA and procedural skill scores significantly increased with year in training. System subcompetency scores correlated significantly with 15 of 23 (65.2%) corresponding CCC scores at a single institution, but 8 correlations (36.4%) were <30.0, illustrating poor correlation. A panel of experts developed a set of EPAs, procedural skill assessment, NTSAs, and OSCEs to form a programmatic system of assessment for anesthesiology residency training in the United States. The method used to develop and pilot test the assessments, the progression of assessment scores with time in training, and the correlation of assessment scores with CCC scoring of milestone achievement provide evidence for the validity of the assessments.

Developing entrustable professional activities for general dentistry at the University of North Carolina.

As part of curriculum innovation, the University of North Carolina (UNC) Adams School of Dentistry identified core entrustable professional activities (EPAs) that graduates must demonstrate for practice readiness. This paper describes the development of the UNC EPAs and the perceptions of the general dentistry faculty. Upon establishing a blueprint of knowledge, skills, and attitudes of UNC graduates, using a distributed leadership approach, faculty teams developed EPAs focused on the patient care process. The American Dental Education Association Compendium of Clinical Competency Assessments and Commission on Dental Accreditation Standards informed the team's work. Perceptions of the assessment framework were examined using a questionnaire completed by 13 general dentistry faculty considering the importance, accuracy, and agreement of each EPA, associated domains of competence, and encounter management on a 6-point rating scale. Distributed leadership was a useful strategy in EPA development to disperse decision-making and build ownership. Through multiple iterations, four EPAs (assessment, plan of care, collaborative care, and provision of care) with associated sub-EPAs emerged. EPAs included a description, required knowledge and skills, and rubrics for assessment. The general dentistry faculty reported a high level of importance, accuracy, and agreement with EPAs, domains of competence, and encounter management. EPAs provide a standardized manner to describe the comprehensive work dentists perform, shifting away from individual competencies. The UNC EPAs provide the foundation for longitudinal measures of competence preparing graduates for independent practice. With limited EPAs frameworks available in dentistry, we aim to inform the development and implementation of EPAs across dental education.

Preceptor perceptions of a redesigned entrustable professional activity (EPA) assessment tool in pharmacy practice experiences

IntroductionThe purpose of this study was to determine pharmacy preceptor perceptions of an entrustable professional activity (EPA) evaluation tool redesigned in 2019 and launched for use in May 2020. MethodsParticipants received an optional survey in June 2021 regarding their perceptions of the redesigned EPA assessment tool. The survey used a combination of open- and closed-ended questions regarding preceptor perceptions of the revised form, including perceptions of accuracy, usability, ease of use, grading, EPAs, and form layout. Preceptor perceptions were measured using seven-point Likert-type items and analyzed with descriptive statistics. Open-ended items were coded by at least two faculty members using qualitative content analysis. ResultsWe received 106 responses (19.5% response rate) and analyzed a total of 91 responses. Most questions related to perceived accuracy were rated positively, and preceptors indicated it was easier to assign grades and provide feedback on EPAs using the new form. Preceptors agreed that EPAs indicated when students on clinical rotations excel and when they need to improve. Preceptors reported they could effectively evaluate students and do so in a way that saves them time. Overall, preceptors agreed that the form instructions helped them accurately assess student performance, and the new narratives helped them remember the assessment standards. ConclusionsOverarchingly, preceptors perceived that the new EPA assessment tool allowed them to provide feedback and grades more accurately, efficiently, effectively, and easily. This information will assist our institution in determining next best steps for EPA assessment tool updates.

The influence of clinical coaching teams on quality of entrustable professional activity assessments.

Coaching is an important component of workplace-based assessment in competency-based medical education. Longitudinal coaching relationships have been proposed to enhance the trainee-supervisor relationship and promote high-quality assessment. The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments. EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score). All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality. The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.

The influence of resident and faculty gender on assessments in anesthesia competency-based medical education.

Competency-based medical education (CBME) relies on frequent workplace-based assessments of trainees, providing opportunities for conscious and implicit biases to reflect in these assessments. We aimed to examine the influence of resident and faculty gender on performance ratings of residents within a CBME system. This retrospective cohort study took place from August 2017 to January 2021 using resident assessment data from two workplace-based assessments: the Anesthesia Clinical Encounter Assessment (ACEA) and Entrustable Professional Activities (EPAs). Self-reported gender data were also extracted.The primary outcome-gender-based differences in entrustment ratings of residents on the ACEA and EPAs-was evaluated using mixed-effects logistic regression, with differences reported through odds ratios and confidence intervals (α = 0.01). Gender-based differences in the receipt of free-text comments on the ACEA and EPAs were also explored. In total, 14,376 ACEA and 4,467 EPA assessments were analyzed. There were no significant differences in entrustment ratings on either assessment tool between men and women residents. Regardless of whether assessments were completed by men or women faculty, entrustment rates between men and women residents were not significantly different for any postgraduate year level. Additionally, men and women residents received strengths-related and actions-related comments on both assessments at comparable frequencies, irrespective of faculty gender. We found no gender-based differences in entrustment ratings for both the ACEA and EPAs, which suggests an absence of resident gender bias within this CBME system. Given considerable heterogeneity in rater leniency, future work would be strengthened by using rater leniency-adjusted scores rather than raw scores.