The Milestones 2.0 Development Process.

Abstract

The Accreditation Council for Graduate Medical Education (ACGME) Next Accreditation System ushered in the era of the Milestones Project, launching with 7 specialties in 2013. With the initial implementation of the Milestones, the ACGME promised to periodically review and revise the Milestones every 3 to 5 years to ensure that they remained relevant and useful.1 And with that, 2016 marked the beginning of the Milestones 2.0 process.When the Milestones were first launched in July 2013, it was certain that they would need to be reviewed and revised. Information was gathered both formally and informally through focus groups, surveys, and other communications such as during grand rounds or institutional talks regarding the Milestones. There were 5 key points that were continuously conveyed. The first being that for many specialties, there were too many subcompetencies, with 41 as the highest number.2 The second area of concern was that the Milestones were written using language that was too complex (labeled as “edu-speak”), which was not easily understood. The third point that frequently surfaced was that there was too much in each Milestone set, meaning that there were too many individual Milestones to evaluate within each level of a subcompetency. Fourth, many junior and midlevel faculty inquired about how they could participate in future iterations. And the last major issue stemmed from mostly designated institutional officials (DIOs), indicating that there was a lack of consistency across the non–patient care (PC) and medical knowledge (MK) subcompetencies, interpersonal and communication skills (ICS), practice-based learning and improvement (PBLI), professionalism (PROF), and systems-based practice (SBP). This was noted through observing a lack of inclusion of a specific topic (eg, accountability) for some specialties. There was also the issue of a topic being included throughout most specialties but demonstrated quite differently in each specialty. For example, self-directed learning was found to have been written 88 different ways when the non–PC and MK competencies for the 26 core specialties plus transitional year were reviewed.3To begin the Milestones 2.0 process, it was important to address the lessons that were learned. The first lesson—the number of subcompetencies—would still be left to the individual specialty work groups as each specialty determines the final number of subcompetencies. For the second and third lessons, decisions were made by ACGME Milestones staff that each subcompetency would be limited to 3 developmental trajectories, and wording would be monitored to address the language. Additionally, another document, called the “Supplemental Guide,” would serve as a way for the work group to demonstrate the intent through clinical examples. The fourth lesson led to a “Call for Volunteers” for each work group, which rendered applications from a wide variety of volunteers at all levels of experience. And for the final lesson, interdisciplinary and interprofessional groups were brought together to create Milestones for the subcompetencies under ICS, PBLI, PROF, and SBP that would be recommended to each specialty group.Once the decision was made on how to adjust the process, the initial steps of Milestones 2.0 revision could begin. The first activity was to address the variability found in the ICS, PBLI, PROF, and SBP subcompetencies. Four groups of interdisciplinary and interprofessional volunteers from a wide variety of specialties were brought together to create Milestones for the 4 competencies, labeled the “Harmonized Milestones.” Group members ranged from physicians to scholars to nurses and represented specialties from surgery to family medicine to pathology. The drafts were disseminated for public comment and received many positive comments. These cross-specialty Harmonized Milestones were then provided to each of the Milestones 2.0 work groups with the intent that each specialty would tailor the language to fit their needs. For example, the language for communicating with patients and families in ICS would look different in the pediatrics versus pathology Milestones 2.0.Each specialty undergoes a similar process for Milestones 2.0 development (Figure 1). After identifying a specialty that was prepared to begin the Milestones 2.0 process, a call for volunteers to participate in the work group was released along with a survey focused on identifying any problematic areas with the current Milestones. Typically, the work groups have 8 to 15 members.The work groups generally include 5 members selected from the call for volunteers, with the remaining spots filled with representatives from the ACGME specialty review committee, American Board of Medical Specialties specialty board, American Osteopathic Association, program director groups, a fellow/resident, and a non-physician member.4 When appropriate, members of the Association of American Medical Colleges and American Association of Colleges of Osteopathic Medicine are also included.4The work groups convene multiple times over 12 to 18 months and begin with a review of the results of the survey of the current trends in Milestones reporting data such as the rate of straight-lining, Box-and-whisker plots of the previous year's data, and percentage of graduates who obtained Level 4 for each subcompetency. The work group also brainstormed the knowledge, skills, and behaviors that the near-future graduate within their specialty should possess. This collective information serves as a foundation for the shared mental model necessary to complete the review and revision work of the current PC and MK Milestones. Harmonized Milestones are customized to meet the needs of the individual specialty while still maintaining the overall purpose. A Supplemental Guide is also produced during these meetings. There are 5 sections of the Supplemental Guide: overall intent of the subcompetency, examples for each level, suggested assessment tools, curriculum mapping (to be completed by the program), and notes and resources. Figure 2 shows an example of the Supplemental Guide, and the Box outlines its use.After a draft is completed, the Milestones and Supplemental Guide are both circulated for public comment for 3 to 4 weeks. An email is sent to the specialty program directors and coordinators, along with the DIOs through various channels, including the ACGME weekly e-Communication, program director listservs, and the ACGME website. The results of the public comments are integrated during the final edit of the Milestones and Supplemental Guide. Finalized specialty Milestones and Supplemental Guides are published on the ACGME website.In an effort to maintain a real-time review of the Milestones 2.0, a specialty-specific quality assurance program is implemented. Specialty programs have volunteered to submit annual feedback regarding the Milestones, identifying which subcompetencies are successful and which need to be reviewed. This continuous quality assessment will highlight any minor issues that may arise such as an outdated treatment modality in the example.As each specialty and program prepares to implement the new version of the Milestones, there are many steps to be completed and a variety of resources available.4,5 An early step for each program should be a comparison of their Milestones data against the nationally aggregated data to identify major discrepancies. If there are discrepancies, the program's program evaluation committee and the clinical competency committee (CCC) should investigate “why” it exists. These differences may or may not be a problem but understanding them can assist programs in preparing for further implementation. After understanding the program's data, each program's CCC would ideally work to create their own shared mental model of the new Milestones using the Supplemental Guide to identify their patient population and needs and their impact on the examples. Additionally, they should list out their own assessment tools and identify resources that can be used by future members of the CCC.A broader shared mental model, along with the new quality assurance process, might extend the amount of time between Milestones 2.0 and 3.0. And as we look to the future of the Milestones, the core competencies must first be reviewed. These core competencies have been in use for more than 20 years and the graduate medical education (GME) community has not performed an in-depth study to measure their influence or consider other competencies (eg, clinical reasoning). Additionally, there are many possible directions that the GME community could take, including a move away from time-based education.When the Milestones were implemented in 2013, the ACGME made a commitment to seek feedback and use it to review and revise future iterations of the Milestones. The Milestones 2.0 process has accomplished this through the work of expert volunteers. While Milestones 2.0 remain similar to the original version, adjustments have been made to make them more user-friendly. Additional educational assistance has been provided to programs. We hope these changes will provide better training for residents and fellows, a similar approach to assessing these attributes across specialties, and ultimately to better patient care.