Emergency Medicine Milestones Research Articles

Entrustable Professional Activities: Making Sense of the Emergency Medicine Milestones

BackgroundThe Next Accreditation System (NAS) is being implemented by the Accreditation Council for Graduate Medical Education with seven specialties, including Emergency Medicine (EM), which began in July 2013. The NAS represents a more structured method of accreditation, with dependence on outcomes and less emphasis on educational process. A key component of the NAS is the individual resident semiannual reporting of the Milestone proficiency levels for all sub-competencies, which are more specific areas of domain for the general competencies. All specialties are struggling to some extent with developing assessment mechanisms for the Milestones. At the heart of this struggle is the conceptualization of the Milestones themselves—descriptors of the individual. In practice, faculty assess clinical care provided to the patient by the resident. This creates difficulty for faculty to assign a resident to a specific sub-competency proficiency level when their focus has been on assessment of clinical care. ObjectivesThe objectives of this article include the discussion of whether Entrustable Professional Activities (EPAs) could be defined and linked to milestones in a way that, once implemented, could inform Clinical Competency Committees of the Milestone proficiency reporting. Discussion: EPAs are units of professional work, or clinical care that may help translate aspects of clinical care into Milestone proficiencies. This article explores EPAs in depth, and discusses how EPAs may be used within EM as one method of assigning proficiency levels to residents. ConclusionsEPAs may be a useful tool to inform Milestone proficiency placement of residents. Because EPAs are based on clinical descriptions rather than individual physician descriptions, there may be less faculty development needed for Milestone sub-competency assessment.

Practicing Emergency Physicians Report Performing Well on Most Emergency Medicine Milestones

BackgroundThe Accreditation Council for Graduate Medical Education's Next Accreditation System endorsed specialty-specific milestones as the foundation of an outcomes-based resident evaluation process. These milestones represent five competency levels (entry level to expert), and graduating residents will be expected to meet Level 4 on all 23 milestones. Limited validation data on these milestones exist. It is unclear if higher levels represent true competencies of practicing emergency medicine (EM) attendings. ObjectiveOur aim was to examine how practicing EM attendings in academic and community settings self-evaluate on the new EM milestones. MethodsAn electronic self-evaluation survey outlining 9 of the 23 EM milestones was sent to a sample of practicing EM attendings in academic and community settings. Attendings were asked to identify which level was appropriate for them. ResultsSeventy-nine attendings were surveyed, with an 89% response rate. Sixty-one percent were academic. Twenty-three percent (95% confidence interval [CI] 20%−27%) of all responses were Levels 1, 2, or 3; 38% (95% CI 34%−42%) were Level 4; and 39% (95% CI 35%−43%) were Level 5. Seventy-seven percent of attendings found themselves to be Level 4 or 5 in eight of nine milestones. Only 47% found themselves to be Level 4 or 5 in ultrasound skills (p = 0.0001). ConclusionsAlthough a majority of EM attendings reported meeting Level 4 milestones, many felt they did not meet Level 4 criteria. Attendings report less perceived competence in ultrasound skills than other milestones. It is unclear if self-assessments reflect the true competency of practicing attendings. The study design can be useful to define the accuracy, precision, and validity of milestones for any medical field.

Ultrasound competency assessment in emergency medicine residency programs.

In the Model of the Clinical Practice of Emergency Medicine (EM), bedside ultrasound (US) is listed as one of the essential procedural skills. EM milestones released by Accreditation Council for Graduate Medical Education and American Board of Emergency Medicine require residents to demonstrate competency in bedside US. The purpose of this study was to assess the current methods used by EM residency training programs to evaluate resident competency in bedside US. This was a cross-sectional survey study. A questionnaire on US education and competency assessment was electronically sent to all EM residency program directors and emergency US directors. The survey consisted of questions regarding the US rotation, structure of US curriculum, presence of US fellowship, image archiving, quality assurance methods, feedback, competency assessment tools, and frequency of assessment. The survey responses are reported as the percentages of total respondents along with 95% confidence intervals (CIs). A total of 124 of 161 EM residency programs participated in this study, representing a 77% response rate. Twenty-six percent (95% CI = 18% to 34%) of programs assess competency only at the end of the US rotation. Eight percent (95% CI = 3% to 13%) assess competency only every 6 months, and 13% (95% CI = 7% to 19%) assess competency only annually. Eight percent (95% CI = 3% to 13%) assess competency only during the final year of training. Thirty percent (95% CI = 22% to 38%) of programs assess competency with a combination of the above intervals, and 16% (95% CI = 10% to 22%) do not assess US competency. Fourteen percent (95% CI = 8% to 20%) use objective structured clinical examinations (OSCEs), and 21% (95% CI = 14% to 28%) use standardized direct observation tools (SDOTs) to assess resident competency in US. Approximately one-third (33%, 95% CI = 24% to 41%) of standardized testing for US competency is conducted with multiple-choice questions. Thirty percent (95% CI = 21% to 38%) administer practical examinations to assess US skills. Currently, a majority of EM residency programs assess resident competency in bedside US. However, there is significant variation in the methods of competency assessment.

Programmatic assessment of level 1 milestones in incoming interns.

With the Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System, emergency medicine (EM) residency programs will be required to report residents' progress through the EM milestones. The milestones include five progressively advancing skill levels, with Level 1 defining the skill set of a medical school graduate and Level5, that of an attending physician. The ACGME stresses that multiple forms of assessment should be used to ensure capture of the multifaceted competencies. The objective of this study was to determine the feasibility and results of programmatic assessment of Level 1 milestones using multisource assessments for incoming EM interns in July. The study population was interns starting in 2012 and 2013. Interns' Level1 milestone assessment was done with four distinct methods: 1) the postgraduate orientation assessment (POA) by the Graduate Medical Education Office for all incoming interns (this multistation examination covers nine of the EM milestones and includes standardized patient cases, task completion, and computer-based stations); 2) direct observation of patient encounters by core faculty using a milestones-based clinical skills competency checklist; 3) the global monthly assessment at the end of the intern orientation month that was updated to reflect the EM milestones; and 4) faculty assessment during procedural labs. These occurred during the July orientation month that included the POA, clinical shifts, didactic sessions, and procedure labs. In the POA, interns were competent in 48% to 93% of the milestones assessed. Overall, competency was 70% to 80%, with low scores noted in aseptic technique (patient care Milestone 13 [PC13]) and written and verbal hand-off (interpersonal communications skills [ICS]2). In overall communication, 70% of interns demonstrated competency. In excess of 80% demonstrated competency in critical values interpretation (PC3), informed consent (PC9), pain assessment (PC11), and geriatric functional assessment (PC3). On direct observation, almost all Level 1 milestones were achieved (93% to 100%); however, only 78% of interns achieved competency in pharmacotherapy (PC5). On global monthly evaluations, all interns met Level 1 milestones. A multisource assessment of EM milestones is feasible and useful to determine Level 1 milestones achievement for incoming interns. A structured assessment program, used in conjunction with more traditional forms of evaluation such as global monthly evaluations and direct observation, is useful for identifying deficits in new trainees and may be able inform the creation of early intervention programs.

Milestone myths and misperceptions.

In 1999 the Accreditation Council for Graduate Medical Education (ACGME) introduced 6 clinical competencies.1 The longitudinal integration of these competencies into educational curricula and assessment of residents and fellows was intended to also generate outcomes assessments–based information on the educational effectiveness of training programs. In 2008, to further this process, ACGME Chief Executive Officer Thomas J. Nasca, MD, announced his vision for “The Next Step in the Outcomes-Based Accreditation Project,”2 which became the Next Accreditation System (NAS). This vision was notable for 2 elements: development of specialty-specific, competency-based Milestones, along with design and implementation of Milestone assessment tools. The ultimate goal was to further transform graduate medical education from a process focus to an outcomes-based focus. In July 2013, 7 specialties that had volunteered to be early adopters of the NAS began their journey down this exciting and relatively uncharted path.3 As a program director and member of the committee who created the Emergency Medicine Milestones, I have been struck by the incredible diversity of approaches during the Milestone rollout process.4,5 Given this diversity, inevitably some applications of the Milestone framework may not result in the lofty aspirational goals that we hoped for. The Milestones, along with other elements of the NAS, are intended to promote program improvements via self-study driven by objective data. The Next Accreditation System emphasizes trainee assessment based on observable behaviors, using stable, reproducible methods that eliminate that old educational saw, “I know it when I see it.” The community designing the Milestones aimed to make the assessment processes transparent and to stimulate creativity in the graduate medical education community with this paradigm shift. As the Milestones are beginning to be used, there are some misperceptions about the Milestones and misinterpretations in how they should be used (box). This perspective seeks to offer added clarity and address these misperceptions. Box Common Misinterpretations and Misperceptions About Milestones Milestones are a global rating scale for all specialties Milestone levels accurately correspond to year of training The Clinical Competency Committee is a brand new process The assessment methodologies that are currently in place have no utility in Milestones The assessment methodologies for the Clinical Competency Committee are yet to be designed Milestones will eliminate grade inflation Milestone reporting will be so onerous that it will interfere with training Milestones should not be shared with residents

Teaching and evaluating multitasking ability in emergency medicine residents - what is the best practice?

Multitasking is an essential skill to develop during Emergency Medicine (EM) residency. Residents who struggle to cope in a multitasking environment risk fatigue, stress, and burnout. Improper management of interruption has been causally linked with medical errors. Formal teaching and evaluation of multitasking is often lacking in EM residency programs. This article reviewed the literature on multitasking in EM to identify best practices for teaching and evaluating multitasking amongst EM residents. With the advancement in understanding of what multitasking is, deliberate attempts should be made to teach residents pitfalls and coping strategies. This can be taught through a formal curriculum, role modeling by faculty, and simulation training. The best way to evaluate multitasking ability in residents is by direct observation. The EM Milestone Project provides a framework by which multitasking can be evaluated. EM residents should be deployed in work environments commiserate with their multitasking ability and their progress should be graduated after identified deficiencies are remediated.

An Alumni Survey Using Emergency Medicine Milestones as a Needs Assessment for Curriculum Improvement

An Alumni Survey Using Emergency Medicine Milestones as a Needs Assessment for Curriculum Improvement

How Competent Are Emergency Medicine Interns for Level 1 Milestones: Who Is Responsible?

The Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME) includes the implementation of developmental milestones for each specialty. The milestones include five progressively advancing skill levels, with Level 1 defining the skill level of a medical student graduate, and Level 5, that of an attending physician. The goal of this study was to query interns on how well they thought their medical school had prepared them to meet the proposed emergency medicine (EM) Level 1 milestones. In July 2012, an electronic survey was distributed to the interns of 13 EM residency programs, asking interns whether they were taught and assessed on the proposed Level 1 milestones. Of possible participants, 113 of 161 interns responded (70% response rate). The interns represented all four regions of the country. The interns responded that the rates of Level 1 milestones they had been taught ranged from 61% for ultrasound to 98% for performance of focused history and physical examination. A substantial number of interns (up to 39%) reported no instruction on milestones such as patient disposition, pain management, and vascular access. Graduating medical students were less commonly assessed than taught the milestones. Skills with technology, including "explain the role of the electronic health record and computerized physician order entry," were assessed for only 39% of interns, and knowledge (USMLE) and history and physical were assessed in nearly all interns. Disposition, ultrasound, multitasking, and wound management were assessed less than half of the time. Many entering EM interns may not have had either teaching or assessment on the knowledge, skills, and behaviors making up the Level 1 milestones expected for graduating medical students. Thus, there is a potential gap in the teaching and assessment of EM interns. Based on these findings, it is unclear who will be responsible (medical schools, EM clerkships, or residency programs) for ensuring that medical students entering residency have achieved Level 1 milestones.

The Emergency Medicine Milestones: A Validation Study

The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties sought to define milestones for skill and knowledge acquisition during residency training. Milestones are significant objective observable events. The milestones are listed within a structure that is derived from the ACGME general competencies. Major groups of milestones are called "subcompetencies." The original 24 subcompetencies containing 255 milestones for emergency medicine (EM) were developed through a multiorganizational group representing most EM stakeholder groups. To assure that the milestones reflected EM resident progress throughout training, the EM Milestones Working Group (EM MWG) sought to validate the individual milestones. A computer-based survey was sent to all EM residency programs. The survey period began on April 30, 2012, and concluded on May 15, 2012. Respondents were asked to assign each milestone to a specific level of skill or knowledge acquisition. These levels ranged from a beginning resident to an accomplished clinician. There were two different forms that divided the milestones into two groups of 12 subcompetencies each. Surveys were randomly assigned to programs. There were five respondents (the program director and four key faculty) requested from each of the 159 residences. There were responses from 96 programs (60.4%). Of the 795 survey recipients, 28 were excluded due to prior exposure to the EM milestones. Of the remaining 767 potential respondents, 281 completed the survey (36.6%) within a 16-day period. Based on the survey results, the working group adjusted the milestones in the following ways: one entire subcompetency (teaching) was eliminated, six new milestones were created, 34 milestones were eliminated, 26 milestones were reassigned to a lower level score, and 20 were reassigned to a higher level. Nineteen milestones were edited to provide greater clarity. The final result was 227 discrete milestones among 23 subcompetencies. The EM milestones were validated through a milestone assignment process using a computer-based survey completed by program directors and key faculty. Milestones were revised in accordance with the results to better align assignment within each performance level.

The Development of the Emergency Medicine Milestones

The Accreditation Council for Graduate Medical Education (ACGME) has outlined its "Next Accreditation System" (NAS) that will focus on resident and residency outcome measurements. Emergency medicine (EM) is one of seven specialties that will implement the NAS beginning July 2013. All other specialties will follow in July 2014. A key component of the NAS is the development of assessable milestones, which are explicit accomplishments or behaviors that occur during the process of residency education. Milestones describe competencies more specifically and identify specialty-specific knowledge, skills, attitudes, and behaviors (KSABs) that can be used as outcome measures within the general competencies. The ACGME and the American Board of Emergency Medicine (ABEM) convened an EM milestone working group to develop the EM milestones. This article describes the development, use within the NAS, and challenges of the EM milestones.