Next Accreditation System Research Articles

Introducing the Neurology Milestones 2.0.

The turn of the 21st century heralded a new era in medical education. The Institute of Medicine reports To Err is Human and Crossing the Quality Chasm called for reform of the health care system to improve the quality and safety of patient care.1,2 Concurrently, interest in competency-based medical education increased, and the alignment of an outcomes-based approach to education with the Institute for Healthcare Improvement's “Triple Aim” of health care set the stage for the ongoing transformation in medical education.3,4 A major component of this transformation, specific to graduate medical education (GME), has been the Next Accreditation System (NAS) of The Accreditation Council for Graduate Medical Education (ACGME).

The Importance of Competency-Based Programmatic Assessment in Graduate Medical Education.

The Importance of Competency-Based Programmatic Assessment in Graduate Medical Education.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

Thriving During the 10-Year Accreditation Site Visit: Practical Tips From a Program's Perspective.

Thriving During the 10-Year Accreditation Site Visit: Practical Tips From a Program's Perspective.

Factors Leading to Trainee Entrustment in EGD and Colonoscopy: A Prospective Multicenter Study Evaluating Endoscopy Competence in the Era of the Next Accreditation System (NAS): Presidential Poster Award

Factors Leading to Trainee Entrustment in EGD and Colonoscopy: A Prospective Multicenter Study Evaluating Endoscopy Competence in the Era of the Next Accreditation System (NAS): Presidential Poster Award

A Prospective Multicenter Study Evaluating Endoscopy Competence Among Gastroenterology Trainees in the Era of the Next Accreditation System (NAS) - The EnCompAS Study

A Prospective Multicenter Study Evaluating Endoscopy Competence Among Gastroenterology Trainees in the Era of the Next Accreditation System (NAS) - The EnCompAS Study

Orthopaedic Resident Surgical Case Log Disparities Observed in the Next Accreditation System.

With the institution of the Next Accreditation System (NAS), case log procedures fundamentally changed. Unless multiple entries are made, only one procedure per case is credited for procedure counts. We hypothesized that the NAS caused notable changes in national procedure data. Accreditation Council for Graduate Medical Education Orthopaedic Surgery Case Logs National Data Reports from 2008 to 2016 were analyzed to calculate differences in case log data before and after NAS implementation. In the first academic year post-NAS, the average total procedures decreased by 36%. Total procedures increased the following 2 years but still represent a decrease of >30% from pre-NAS data. An average of 580 fewer total procedures per resident were reported in the 3 years post-NAS (P = 0.001). Regression analysis showed notable decreases in credited procedures in all but two categories. The decrease in logged procedures with the NAS may be related to new guidance, resident logging habits, an actual decrease in surgical experience, or unknown causes, or combinations of these factors. A new baseline of case data may be emerging post-NAS. To ensure the highest quality education, NAS case logs warrant continued study to determine how the data should be used in residency education and accreditation decisions.

The Perfect Storm Is on the Horizon!

Provide analysis of data identifying the impending mass turnover of Directors and Coordinators of General Surgery residency programs and the potential effect on successful continuing accreditation including the ACGME Next Accreditation System (NAS) and associated self study. The Association of Residency Coordinators in Surgery, Executive Committee (ARCS EC) anonymously surveyed 254 general surgery Program Coordinators in September/October 2016. This represents 60`% of all the members within the Association of Residency Coordinators in Surgery. Survey was accomplished using SurveyMonkey. Questions included demographics and experience of the Director and Coordinator, accreditation status, significant job stressors, and potential retirements or position changes planned or possible. 153 (60%) respondents completed the survey. Data from the survey indicates that 67% of Program Directors have been in their position less than six (6) years. 34% of Program Coordinators have been in their position less than five (5) years. 56% of coordinators have been on the job less than ten (10) years. Coordinators in 76% of programs reported significant levels of burnout. 59% have considered resigning from their position in the past year. Participants consistently reported increasing responsibilities and expectations combined with declining or inadequate levels of support as sources of job stress. Other contributors to Coordinator burnout were identified as related to the additional ACGME accreditation requirements and salaries not commensurate with workload. This survey represents a sentinel preliminary look at the possible impending manning crisis in general surgery residency program leadership. A Program Director is supported by a Program Coordinators who are burned out and considering a job change. The resultant potential turnover in personnel and loss of collective program knowledge may have devastating ramifications to program accreditation. Subsequent survey of the workforce will evaluate proposed solutions and interventions to prevent this outcome and secure the future success of general surgery programs.

Interpersonal and Communication Skills Competency Assessment in ID programs

BackgroundThere has been an increased emphasis over the past decade on quality metrics and competency assessment in healthcare. With the advent of the Next Accreditation System (NAS), assessment is imperative in order to meet ACGME requirements. As the ACGME defined the six physician in training competencies, it became incumbent upon training programs to investigate evaluative measures. Interpersonal and communication competency assessment methods vary among medical educators. Tools such as 360 evaluations, chart audits, patient satisfaction questionnaires, and OSCEs have proven helpful, yet difficult and time-consuming to administer. Ideally, direct observation of trainees would improve the instruction and documentation of communication skills. Clear understanding of the domains of GME Outcomes as well as creation of specific measures that are reliable and accurate while not creating an undue measurement burden are necessary.MethodsInfectious Diseases (ID) physicians rely heavily on communication and interpersonal skills to care for their patients. This multi institutional, qualitative study investigated the approaches used as well as the process employed by ID faculty regarding Interpersonal and Communication skill competency evaluation. Faculty interviews were recorded, transcribed and coded.ResultsBased on the interview transcripts, general themes discovered included a lack of standardization yet considerable value placed upon trainee competency in the areas of interpersonal and communication skills. In addition, ID faculty revealed current practices employed, challenges addressed as well as recommendations for assessment.ConclusionIncrease in accreditation requirements have contributed to additional evaluative responsibilities ID medical educators must adopt. Ascertaining competency assessment methods employed by other ID faculty can provide alternative means of assessment and increase understanding regarding evaluative tools.Disclosures All authors: No reported disclosures.

What Is a Rheumatologist and How Do We Make One?

Graduate medical education is a critical time in the training of a rheumatologist, and purposeful evaluation of abilities during this time is essential for long-term success as an independent practitioner. The internal medicine subspecialties collectively developed a uniform set of reporting milestones by which trainees can be assessed and receive formative feedback, providing clarity of accomplishment as well as areas for improvement in training. Furthermore, the reporting milestones provide a schema for assessment and evaluation of fellows by supervisors. The internal medicine subspecialties were also tasked with considering entrustable professional activities (EPAs), which define the abilities of a subspecialty physician who has attained sufficient mastery of the field to be accountable to stakeholders and participate in independent practice. Although EPAs have been established for a few specialties, they had not yet been described for rheumatology. EPAs have value as descriptors of the comprehensive abilities, knowledge, and skills of a practicing rheumatologist. The rheumatology EPAs have a role in defining a specialist in rheumatology upon completion of training, and also represent the ways our specialty defines our abilities that are enduring throughout practice. We describe the collaborative process of the development of both the subspecialty reporting milestones and the rheumatology EPAs. The reporting milestones evolved through discussions and collaborations among representatives from the Association of Specialty Professors, the Alliance for Academic Internal Medicine, the American Board of Internal Medicine, and the Accreditation Council for Graduate Medical Education. The EPAs were a product of deliberations by the Next Accreditation System (NAS) working group of the American College of Rheumatology (ACR) Committee on Rheumatology Training and Workforce Issues. Twenty-three subspecialty reporting milestones and 14 rheumatology EPAs were advanced and refined over the course of 3 subspecialty reporting milestone development summits and 3 ACR NAS working group meetings, respectively. The subspecialty reporting milestones and rheumatology EPAs presented here stipulate reasonable and measurable expectations for rheumatologists-in-training. Together, these tools aim to promote enrichment and greater accountability in the training of fellows. Additionally, the EPAs define, for all stakeholders, the expertise of a rheumatologist in practice.

A multidisciplinary performance improvement rotation in an internal medicine training program.

Performance Improvement (PI) in health care is defined as “combined and unceasing efforts of everyone - healthcare professionals, patients and their families, researchers, payers, planners and educators - to make the changes that will lead to better patient outcomes, better system performance and better professional development”.1 Emphasis on quality improvement with formal rotations in residency programmes is a recent phenomenon. Our Internal Medicine Training Program based at Sisters of Charity Hospital is a part of the State University of New York at Buffalo (SUNY Buffalo). We have a total of 37 medical residents each year. In 2006, we implemented a four-week rotation titled ‘Performance Improvement Rotation’ and focused on five specific areas: Core measures, Departmental peer review process, Clinical documentation, Patient safety goals and adverse event reporting. The four-week rotation, mandatory for the residents in second year of training, started with an orientation and concluded with a written examination with required performance at 75 percentile. While many programmes have their own curricula for patient safety rotations, in June 2012, the Accreditation Council for Graduate Medical Education (ACGME) established Clinical Learning Environment Review (CLER) as a component of the Next Accreditation System (NAS) in an attempt to focus on patient safety education and quality improvement in training programmes.2,3 ACGME is the body responsible for accrediting the majority of graduate medical training programmes for physicians in the United States. When ACGME established these CLER guidelines, we decided to evaluate and modify the existing curriculum. What was the curriculum? In the review of core measures, residents understood the Centers for Medicare & Medicaid Services (CMS) core measures.4 They performed chart review of patients admitted to the hospital. Core measures are specific clinical measures that, when viewed together, permit an assessment of the quality of care provided in a specific condition, for example, acute myocardial infarction. As a part of Peer Review Process, residents participated in a peer review module to identify and prevent patient safety deficiencies.5 To improve and standardize the documentation, the clinical documentation consultants discussed documentation requirements with chart review on the floors with the residents.6 Patient safety goals component of the rotation was diverse and included verbal order chart review, medication reconciliation, Do Not Use (DNU) abbreviations, patient identification guidelines as defined by Joint Commission on Accreditation of Healthcare Organizations (JCAHO)7 and look alike and sound alike medications. Adverse event reporting exercise had activities like participating in ‘Root Cause Analysis (RCA)’ of few patients.

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

ABSTRACTConstruct: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). Background: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. Approach: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013–June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. Results: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). Conclusions: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.

Implementing the Next Accreditation System: Results of the 2014-2015 Annual Data Review.

Implementing the Next Accreditation System: Results of the 2014-2015 Annual Data Review.

Reflections in a time of transition: orthopaedic faculty and resident understanding of accreditation schemes and opinions on surgical skills feedback

IntroductionOrthopaedic surgery is one of the first seven specialties that began collecting Milestone data as part of the Accreditation Council for Graduate Medical Education's Next Accreditation System (NAS) rollout. This transition from process-based advancement to outcome-based education is an opportunity to assess resident and faculty understanding of changing paradigms, and opinions about technical skill evaluation.MethodsIn a large academic orthopaedic surgery residency program, residents and faculty were anonymously surveyed. A total of 31/32 (97%) residents and 29/53 (55%) faculty responded to Likert scale assessments and provided open-ended responses. An internal end-of-rotation audit was conducted to assess timeliness of evaluations. A mixed-method analysis was utilized, with nonparametric statistical testing and a constant-comparative qualitative method.ResultsThere was greater familiarity with the six core competencies than with Milestones or the NAS (p<0.05). A majority of faculty and residents felt that end-of-rotation evaluations were not adequate for surgical skills feedback. Fifty-eight per cent of residents reported that end-of-rotation evaluations were rarely or never filled out in a timely fashion. An internal audit demonstrated that more than 30% of evaluations were completed over a month after rotation end. Qualitative analysis included themes of resident desire for more face-to-face feedback on technical skills after operative cases, and several barriers to more frequent feedback.DiscussionThe NAS and outcome-based education have arrived. Residents and faculty need to be educated on this changing paradigm. This transition period is also a window of opportunity to address methods of evaluation and feedback. In our orthopaedic residency, trainees were significantly less satisfied than faculty with the amount of technical and surgical skills feedback being provided to trainees. The quantitative and qualitative analyses converge on one theme: a desire for frequent, explicit, timely feedback after operative cases. To overcome the time-limited clinical environment, feedback tools need to be easily integrated and efficient. Creative solutions may be needed to truly achieve outcome-based graduate medical education.

Identifying the Challenges of “Small” Pathology Residency Programs and Creating Collaborative Solutions

The majority of pathology residency training programs in the United States are considered to be small training programs. Small training programs, regardless of specialty, encounter unique challenges that have been documented in the literature. With the implementation of the Next Accreditation System (NAS), and other Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements, adequate personnel and other resources are necessary. An online survey was conducted on the pathology program directors’ section listserv to help identify characteristics and challenges of small pathology residency training programs. A discussion group on small pathology residency programs was held at the 2015 Association of Pathology Chairs/Program Directors annual meeting, where the results of the survey were discussed and small breakout groups followed the discussion of the survey. The results of the online survey and discussion groups are discussed in this paper.

A single activity with a practice quality improvement project for faculty and a quality improvement project for residents

PurposeThe Next Accreditation System (NAS) requires radiation oncology residents to do a formal quality improvement project during their residency. The American Board of Radiology (ABR) Maintenance of Certification (MOC) program requires certified physicians to complete a Practice Quality Improvement (PQI) project approximately every 3 years. The purpose of our project was to develop a clinical transition of care policy via a process that resulted in quality improvement project credit for residents and PQI credit for participating faculty. Methods and materialsApproval for project implementation was obtained from the ABR MOC committee. The PQI project consisted of an initial survey to assess resident perception on resident transition of care in our department, formal sign-out training, and 2 postintervention surveys after 1 and 11 months. The primary endpoint was the percentage of questions with ≤1 unfavorable responses. Sign-test was used to determine response difference from neutral. ResultsOne hundred percent of surveyed residents completed the preintervention (n = 6), postintervention 1 (n = 7), and postintervention 2 (n = 8) surveys. In the preintervention, postintervention 1, and postintervention 2 surveys, 71.4%, 57.1%, and 57.1% of questions were answered with ≤1 unfavorable response, respectively. The number of questions with ≥75% favorable response was 7 (50%), 7 (50%), and 11 (78.5%) in the preintervention, postintervention 1, and postintervention 2 surveys, respectively (P = .13). A written sign-out template and monthly protected sign-out meetings were instituted. One resident and 3 attending physicians received credit for Accreditation Council of Graduate Medical Education NAS quality improvement and ABR MOC PQI projects, respectively. ConclusionsThis project shows the feasibility of a combined attending and resident physician effort to improve patient care and fulfill his or her respective ABR MOC PQI and Accreditation Council of Graduate Medical Education NAS requirements. Attending and resident physicians can tailor collaborative projects to fulfill MOC and NAS requirements unique to their subspecialty. Written sign-out templates and protected sign-out time may improve transition of care.

Improving the quality of patient care through a collaborative maintenance of certification and next accreditation system project.

e17728 Background: The recent implementation of the Accreditation Council of Graduate Medical Education (ACGME) next accreditation system (NAS) for resident physicians, and the mandatory American B...

Accreditation Standards for colon and rectal surgery residency: Overview of accreditation from the chair of the Residency Review Committee

The Accreditation Council for Graduate Medical Education (ACGME) oversees the function and accreditation of residency training programs in the United States and, increasingly, in the other parts of the world. This article briefly reviews the evolution of residency training and accreditation, and the structure and function of the ACGME. The ACGME is currently instituting a major redesign of the accreditation process, the Next Accreditation System (NAS), with the goals of increasing focus on measurable outcomes, improving the learning environment, decreasing administrative burdens, and encouraging educational innovation. The NAS and its components will be discussed.

Use of Milestones and Development of Entrustable Professional Activities in 2 Hematology/Oncology Training Programs

The Next Accreditation System (NAS) increases the focus on educational outcomes and meaningful evaluation of learners. This requires that key clinical faculty develop new assessment formats such as entrustable professional activities (EPAs). To build and develop milestone-based assessment tools supporting 5 EPAs for a hematology/oncology fellow continuity clinic, and to educate key clinical faculty regarding the Clinical Competency Committee (CCC) and the NAS. Program directors from 2 hematology/oncology fellowship programs developed 5 EPAs for continuity clinic evaluation supported by milestone-based assessment. The program directors met to create a unified CCC charter. Key clinical faculty helped to develop a milestone-based evaluation of fellow continuity clinic through creation of 5 hematology/oncology-specific EPAs. Formal entrustment regarding EPAs was deliberated by the CCC. A total of 18 fellows were evaluated. Clinical Competency Committee deliberation at each institution took approximately 10minutes per fellow for discussion and decision regarding entrustment for all 5 EPAs supporting continuity clinic. One-third of postgraduate year (PGY)-4s, 50% of PGY-5s, and 100% of PGY-6s were deemed competent in all 5 EPAs by the CCC. All hematology/oncology trainees in San Antonio were evaluated using milestone-based assessment for continuity clinic, and entrustment decisions regarding 5 EPAs were made by the CCC. This project may provide other programs with a sound basis for adoption and further development of the next generation of evaluation tools at their institutions. Entrustable professional activities that are rotation specific should be used as a starting point for linking to the competencies, subcompetencies, and the reporting milestones.

The Operative Entrustability Assessment

PURPOSE Documentation of resident operative performance remains challenging. Although numerous assessment tools exist, they tend to be lengthy and ill-suited for use after every case, leading to inconsistent or insufficient data informing mandatory semiannual resident reviews. For these reasons, we created a simple, online assessment of resident operative skill to facilitate documentation of training activities in compliance with the American Council of Graduate Medical Education’s (ACGME) Next Accreditation System (NAS). METHODS Working with the Johns Hopkins Technology Innovation Center, we developed an online assessment tool: the Operative Entrustability Assessment (OEA). The tool includes a 5-point scale with descriptive anchors, can be applied to any current procedural terminology code, can be mapped to the NAS Milestones, and mirrors the ACGME case log structure. Department-wide use of the OEA started in January 2014, and residents and faculty were instructed to complete OEAs for all plastic surgery residents performing operative procedures with Department of Plastic Surgery attending physicians. Residents initiate assessments, complete a self-assessment, and hand (or e-mail) the OEA to the attending for completion. Attendings assign cases to the most relevant NAS Milestone and grade the resident on his/her ability to perform the case autonomously. In-person evaluations provide opportunities for immediate feedback; scores are available for viewing in real time via the Data Dashboard. Data can be viewed by Milestone category, CPT code, or case. Data can be further sorted by postgraduate year level, specific dates, Milestone, and CPT code. RESULTS From January 21, 2014, to October 31, 2014, 1298 OEAs were completed by 21 residents and available for use by the Clinical Competency Committee. Six residents did not complete any OEAs due to being on off-service or elective rotations (n = 1); on required, nonclinical (eg, research) rotations (n = 3); or PGY-1 residents assigned to general surgery rotations (n = 2) during the time assessed. Clinical Competency Committee resident assessment and documentation took 275 minutes (4.58 hours), averaging 11.5 minutes (range, 2–28 minutes) per resident across all 36 Milestones, including discussion and recording of action items/assignment of learning activities. OEA data were most often used to inform Patient Care Milestones. Comparisons with the operating room case counts per attending physician estimate the OEA completion rate at approximately 60% of all resident-assisted cases; 38% of residents reported increased immediate feedback on operative performance. CONCLUSIONS Implementation of the OEA has greatly benefitted our semiannual and ongoing assessment of >20 residents, reducing evaluation time >5 times from pilot program evaluation times (1 hour per resident). The OEA also informs program mentor and director the meetings and assignment of training activities. Data transparency provides residents with their own progress compared to the pooled mean of their peers, empowering residents to proactively address areas for potential improvement. Completing OEAs immediately following cases provides valuable opportunities for immediate feedback on operative performance. We continue using the OEA and Data Dashboard for resident assessment and plan to make this tool, which has changed the ACGME case log from a passive to an active measure of operative skill, available to other Plastic Surgery Programs in the near future to validate these findings.