Competency-based Milestones Research Articles

Emergency Medicine Physician Residents' Perceptions of Emergency Medicine Clinical Pharmacists' Involvement in Their Training.

Although the clinical impact of emergency medicine clinical pharmacists (EMCPs) on patient care outcomes is well documented, their educational impact on resident physicians' training is not. To further highlight the utility of EMCPs, this study evaluated emergency medicine (EM) resident physicians' perceptions of EMCPs' involvement in their training. A voluntary, anonymous web-based survey was sent by email to all 44 EM resident physicians in July 2022. The survey included multiple choice, 5-point Likert scale, and free response questions derived from Accreditation Council for Graduate Medical Education pharmacotherapy competency-based milestones. Thirty-six of the 44 (82%) residents completed the survey and all 10 PGY-4 class residents completed the survey. Nearly half of the respondents (44.4%) reported they consulted/interacted with the EMCPs 3 to 5 times per week and this number increased with the level of training. Respondents most often consulted the EMCPs to obtain medication indications, antibiotic dosing, pediatric dosing, and contraindications. Overall, respondents primarily reported strongly agree to all survey questions. Nearly all respondents strongly agreed the EMCPs are an important part of the patient care team and provide education that is different from what a supervising physician provides. All respondents who completed the pharmacy elective strongly agreed the elective was valuable and strongly recommended other residents to complete it. Respondents reported EMCPs are an important part of the patient care team, play a significant role in their training, and provide education that is different from what a supervising physician provides. Our findings encourage other institutions to leverage physicians' views of EMCPs to expand their service line.

Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge

Objectives: This study aims to show whether correlation exists between pediatric residency applicants’ quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents’ ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones. Methods: In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance. Results: Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents). Conclusions: These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

Pediatric residency milestone performance is not predicted by the United States Medical Licensing Examination Step 2 Clinical Knowledge.

This study aims to show whether correlation exists between pediatric residency applicants' quantitative scores on the United States Medical Licensing Exam Step 2 Clinical Knowledge examination and their subsequent performance in residency training based on the Accreditation Council for Graduate Medical Education Milestones, which are competency-based assessments that aim to determine residents' ability to work unsupervised after postgraduate training. No previous literature has correlated Step 2 Clinical Knowledge scores with pediatric residency performance assessed by Milestones. In this retrospective cohort study, the United States Medical Licensing Exam Step 2 Clinical Knowledge Scores and Milestones data were collected from all 188 residents enrolled in a single categorical pediatric residency program from 2012 - 2017. Pearson correlation coefficients were calculated amongst available test and milestone data points to determine correlation between test scores and clinical performance. Using Pearson correlation coefficients, no significant correlation was found between quantitative scores on the Step 2 Clinical Knowledge exam and average Milestones ratings (r = -0.1 for post-graduate year 1 residents and r = 0.25 for post-graduate year 3 residents). These results demonstrate that Step 2 scores have no correlation to success in residency training as measured by progression along competency-based Milestones. This information should limit the importance residency programs place on quantitative Step 2 scores in their ranking of residency applicants. Future studies should include multiple residency programs across multiple specialties to help make these findings more generalizable.

To Boost or Not to Boost Residents and Fellows-That Is the Question.

To Boost or Not to Boost Residents and Fellows-That Is the Question.

Impact of COVID-19 on Radiology Trainee Safety, Education, and Wellness: Challenges Experienced and Proposed Solutions for the Future.

Impact of COVID-19 on Radiology Trainee Safety, Education, and Wellness: Challenges Experienced and Proposed Solutions for the Future.

A Day in the Life of an Internal Medicine Resident -A Time Study: What Is Changed from First to Third Year?

BackgroundIn the past decade, internal medicine residencies have undergone major changes in competency-based assessments, work-hour restrictions and the implementation of the electronic medical record. The aim of this study is to compare a typical day of a first year post-graduate (PGY1) to that of a third year post-graduate (PGY3) internal medicine resident and examine if the differences in their days demonstrate the American Board of Internal Medicine’s (ABIM) desired progression towards competency-based milestones and unsupervised practice.MethodsWe conducted an observational time study documenting 14,103 minutes, 9 major categories, and 17 subcategories while shadowing 10 internal medicine PGY1s and 10 PGY3s during inpatient, non-call days. The following day, house staff completed surveys of their perceived time allocation of the previous 24 hours.ResultsPGY1s spent an average of 12.5 hours managing an average of 6 patients. Thirty-eight percent of their time was spent on the computer, 21% discussing patients and 18% directly with patients. PGY3s, overseeing an average of 12 patients, worked 1.5 hours less per day (p<0.001), had 1.5 hours less computer time (p=0.001), 24 minutes less direct patient contact (p=0.045), and 36 minutes more patient care discussions (p=0.011).ConclusionThe difference between PGY1s’ and PGY3s’ daily time allocations is minimal. Whereas a PGY3 spends 1.5 hours less than a PGY1 on writing computer notes and discharges, they also work 1.5 hours less per day. The additional 36 minutes of patient care discussions was the only significant time quantity difference that would be considered a higher level of practice for the PGY3 compared to the PGY1. With residents now caring for fewer patients, there has been a marked increase in computer time per patient for both PGY1s and PGY3s.

74. LACK OF CORRELATION BETWEEN USMLE SCORES AND PERFORMANCE IN PEDIATRICS RESIDENCY TRAINING BASED ON ACGME MILESTONES RATINGS

Introduction The United States Medical Licensing Exam (USMLE) was designed to determine whether medical trainees are competent to become licensed physicians, not necessarily as a means of providing a comparative gradient on students’ knowledge. Despite this, many residency programs use USMLE scores in the residency selection process as indicators as to whether physicians will succeed in residency. Not only is this not the intended use of the USMLE, some previous studies have suggested that USMLE scores are not effective indicators of a physician's success during residency. One method of assessing residents’ progress through residency training are the Accreditation Council for Graduate Medical Education (ACGME) Milestones in the six clinical competency domains of Patient Care, Medical Knowledge, Systems Based Practice, Practice-Based Learning and Improvement, Professionalism, and Interpersonal Communications Skills. Ratings along these milestones are completed by supervising physicians who observe and assess residents in the clinical setting. No previous literature has tried to link USMLE Step scores to resident performance using Milestone ratings. Methods USMLE Step 1 and 2 Scores and Milestones Ratings in twenty-one different competencies were collected over five years (2012 - 2017) from 187 residents enrolled in a single large pediatrics residency program. Correlations were examined using Pearson's correlation coefficients. Results Among the residents studied, the mean score of Step 1 was 236, while the mean score of Step 2 was 250. While Step 1 and 2 had correlation (r2 = -0.628), no correlation was found between Step 1 and overall Milestones average (r2 = -0.130 and Conclusion These results suggest that USMLE scores have no correlation to success in residency training as measured by progression along competency-based milestones. These confirm previous claims about the limited utility of USMLE scores in predicting success in residency training.

Use of Emergency Department Pharmacists in Emergency Medicine Resident Milestone Assessment

IntroductionThe use of competency-based milestones for emergency medicine (EM) was mandated by the Accreditation Council for Graduate Medical Education in 2013. However, clinical competency committees (CCC) may lack diverse, objective data to assess these new competencies. To remedy the lack of objective data when assessing the pharmacotherapy sub-competency (PC5) we introduced a unique approach that actively involves departmental clinical pharmacists in determining the milestone level achieved by the resident.MethodsOur pharmacists assess the pharmacotherapy knowledge of the residents through multiple methods: direct observation of orders, communication with the residents while performing patient care within the emergency department (ED), and real-time chart review. This observation occurs informally on a daily basis in the ED and is incorporated into the routine work of the pharmacist. The pharmacists use the PC5 sub-competency as their standard evaluation tool in this setting to keep all assessments consistent.ResultsSince our residency program introduced pharmacist assessment of resident pharmacotherapy knowledge, the CCC has conducted seven biannual meetings. Of the 120 separate PC5 sub-competency assessments made during those meetings there was 100% agreement between the pharmacist’s assessment and the CCC’s final assessment of the trainee. A survey of the CCC members concluded that the pharmacists’ assessments were useful and aided in accurate resident evaluation.ConclusionThe use of ED pharmacists in assessing the pharmacotherapy sub-competency provides important information used in resident assessment of the PC5 milestone.

Integrating Foundational Sciences in a Clinical Context in the Post-clerkship Curriculum

To design, implement, and launch courses that integrate foundational science learning and clinical application in a post-clerkship undergraduate medical school curriculum. In Academic Year (AY) 15-16, as part of a comprehensive curricular revision, Vanderbilt University School of Medicine (VUSM) formally implemented "Integrated Science Courses" (ISCs) that combined rigorous training in the foundational sciences with meaningful clinical experiences. These courses integrated foundational sciences that could be leveraged in the clinical environment, utilized a variety of instructional modalities, and included quantitative and qualitative (competency-based milestones) student assessments. Each ISC underwent a rigorous quality improvement process that required input on foundational science content, student experience, and student performance assessment. Eleven ISCs were delivered to 173 students in AY15-16, with some students taking more than one ISC. Immediately after completing each course, 93% (n=222) of ISC enrollees completed a course evaluation. Students (91%; n=201) 'agreed' or 'strongly agreed' that foundational science learning informed and enriched the clinical experiences. Furthermore, 94% (n=209) of students thought that the clinical experiences informed and enriched the foundational science learning. Ninety-four percent of the students anticipated using the foundational science knowledge acquired in future clinical training and practice. The teaching of foundational sciences in the clinical workplace in the post-clerkship medical curriculum is challenging and resource-intensive, yet feasible. Additional experience with the model will inform the mix of courses as well as the breadth and depth of foundational science instruction that is necessary to foster scientifically-based clinical reasoning skills in each student.

Is feedback to medical learners associated with characteristics of improved patient care?

PurposeTo investigate the association of medical learner feedback with patient management and outcomes.MethodsThe authors investigated 27 articles that utilized patient data or chart reviews as a subset of a prior feedback scoping review. Data extraction was completed by two authors and all authors reviewed the descriptive data analysis.ResultsThe studies were predominantly short-term investigations conducted in the US at academic teaching hospitals (89%) with one medical discipline (78%), most commonly internal medicine (56%). Patient-related outcomes primarily involved improved documentation (26%) and adherence to practice guidelines (19%) and were mostly measured through chart reviews (56%) or direct observation (15%). The primary method of feedback delivery involved a written format (30%). The majority of the studies showed a positive effect of feedback on the patient-oriented study outcomes (82%), although most involved a non-rigorous study design.ConclusionsPublished studies focusing on the relationship between medical learner feedback and patient care are sparse. Most involve a single discipline at a single institution and are of a non-rigorous design. Measurements of improved patient outcomes are restricted to changes in management, procedures and documentation. Well-designed studies that directly link learner feedback to patient outcomes may help to support the use of feedback in teaching clinical outcomes improvement in alignment with competency-based milestones.Electronic supplementary materialThe online version of this article (10.1007/s40037-017-0375-8) contains supplementary material, which is available to authorized users.

The Medical-Legal Partnership Approach to Teaching Social Determinants of Health and Structural Competency in Residency Programs.

Medical-legal partnerships (MLPs) embed civil legal services lawyers into health care settings and interprofessional health care teams delivering care to low-income or otherwise vulnerable patients and communities. MLPs present the opportunity to instill in residents a practical understanding of the social determinants of health and provide them with concrete tools to address them. MLP training helps residents develop structural competency and build the skills necessary to address barriers to health at the patient, institutional, and population levels. Through a case study, this Perspective explores how residents can address health-harming legal needs working in partnership with interprofessional health care teams that include lawyers, and illustrates how such MLP experiences can relate to competency-based Milestones that are applicable to training residents in all specialties.

From Theory to Practice: Utilizing Competency-based Milestones to Assess Professional Growth and Development in the Foundational Science Blocks of a Pre-Clerkship Medical School Curriculum.

Physicians-in-training require skills and attitudes beyond medical knowledge in order to mature into successful clinicians. However, because assessments in pre-clerkship curricula historically have focused almost exclusively on medical knowledge, faculty contributions to early student development often have been limited. To address this challenge and enhance student progress, we re-designed our pre-clerkship curriculum to include settings in which diverse facets of student performance could be observed and fostered. Concurrently, we transitioned to an assessment strategy focused on competency-based milestones. The implementation of this strategy has allowed pre-clerkship science faculty to provide early-stage students with rich holistic feedback designed to stimulate their professional growth.

Competency-Based Resident Education, Milestones, and Simulation [19B

Competency-Based Resident Education, Milestones, and Simulation [19B

Systems that Teach: Medical Education and the Future Healthcare Workforce

Physician education has followed relatively rigid guidelines since the Flexner report of 1910. Medical education has been largely didactic with time-based progression and certifying exams, and with variable degrees of autonomy and supervision in graduate (post MD/DO degree) medical education programs. Innovative educational approaches now incorporate adult learning theories, encompass broader skill-sets, and utilize competency-based milestones rather than temporal based objectives to gauge progress. Human resource development (HRD) is more synergistic with evolving innovative medical education approaches than traditional approaches, and it can foster a healthy teaching system. In this article, we explore HRD strategies for medical education, including clinical simulation, workflow and educational redesign, and ongoing competency evaluation.

Validated Assessment Tools and Maintenance of Certification in Plastic Surgery: Current Status, Challenges, and Future Possibilities.

The transition to the Next Accreditation System is well underway, and a shift toward competency-based assessment in the form of milestones is now the standard. A significant effort has been completed by the Plastic Surgery Milestones Working Group to develop specific milestones and assessment tools for plastic surgery training. The history of the development toward competency-based assessment was reviewed. Data regarding the trends and regulations associated with board certification and the role of maintenance of certification were reviewed. The work of the Plastic Surgery Milestones Working Group has sparked interest in assessment and created an opportunity for further development. The efforts toward validating assessment tools by our colleagues working in other surgical specialties serve as a suitable roadmap for further progress. Board certification is an integral part of successful practice and should be regarded as an expectation. Despite the burdens associated with maintenance of certification, it serves a valuable function in ensuring optimal patient care and is often retrospectively seen as an important component of practice. The competency-based milestones are the new standard, and work on this new methodology of assessing plastic surgery trainees is expected to continue. Accurate assessment is critical to the pathways for board certification and maintenance of certification, which serve important roles for all parties involved in the delivery of medical care.

Mind the Gap: The Bumpy Transition From Medical School to Residency.

In 2008, the Accreditation Council for Graduate Medical Education began development of the Next Accreditation System, an assessment strategy based on residents’ achievement of specific milestones within defined competencies as they progress through training. 1 Medical specialties were tasked with creating milestones specific to their training programs. Although similar competency frameworks were used across specialties, these milestones were largely developed in isolation. Not surprisingly, the resulting milestones were ‘‘richly diverse.’’ 2 In several specialties, Level 1 is considered

An Extended Validity Argument for Assessing Feedback Culture

NEGEA 2015 CONFERENCE ABSTRACT (EDITED)Measuring an Organization's Culture of Feedback: Can It Be Done?Steven Rougas and Brian ClyneConstruct: This study sought to develop a construct for measuring formative feedback culture in an academic emergency medicine department. Four archetypes (Market, Adhocracy, Clan, Hierarchy) reflecting an organization's values with respect to focus (internal vs. external) and process (flexibility vs. stability and control) were used to characterize one department's receptiveness to formative feedback. The prevalence of residents' identification with certain archetypes served as an indicator of the department's organizational feedback culture. Background: New regulations have forced academic institutions to implement wide-ranging changes to accommodate competency-based milestones and their assessment. These changes challenge residencies that use formative feedback from faculty as a major source of data for determining training advancement. Though various approaches have been taken to improve formative feedback to residents, there currently exists no tool to objectively measure the organizational culture that surrounds this process. Assessing organizational culture, commonly used in the business sector to represent organizational health, may help residency directors gauge their program's success in fostering formative feedback. The Organizational Culture Assessment Instrument (OCAI) is widely used, extensively validated, applicable to survey research, and theoretically based and may be modifiable to assess formative feedback culture in the emergency department. Approach: Using a modified Delphi technique and several iterations of focus groups amongst educators at one institution, four of the original six OCAI domains (which each contain 4 possible responses) were modified to create a 16-item Formative Feedback Culture Tool (FFCT) that was administered to 26 residents (response rate = 55%) at a single academic emergency medicine department. The mean score of each item on the FFCT (range = 0–100) was analyzed. Convergent and divergent properties of the four archetypes were assessed using a multitrait–multimethod matrix of Pearson's coefficients. Expecting that items in one archetype would diverge from the others, whereas items within an archetype should have strong convergent properties, convergent validity was assessed by comparing items across domains that all related to the same archetype. Similarly, divergent validity was assessed by comparing the correlation of items within an archetype to the correlations of those items within a hetero-domain block (i.e., to other items within the same domain). Results: Three of the four domains of the FFCT (Overall Departmental Characteristics 35.4 ± 15.4, Departmental Foundation of Feedback 46.1 ± 16.7, and Departmental Emphasis of Feedback 30.3 ± 17.7) had the highest mean in the Market archetype (results/achievement oriented), whereas the final domain (Departmental Definition of Successful Feedback 34.8 ± 22.1) had the highest mean in the Clan archetype (personal growth/team achievement). Item responses in the Clan and Hierarchy archetypes had the strongest convergent and divergent validity, respectively. Item responses in the Adhocracy archetype had the weakest convergent and divergent validity. Conclusions: Although the sample size was small, this initial study demonstrates that a modified organizational culture assessment tool can feasibly be utilized to identify the primary formative feedback archetype of a cohort of residents. This may have future implications for measuring changes in culture after the implementation of strategic programs to address formative feedback. Future studies should examine the generalizability of the FFCT to other institutions, as well as address the weak validity evidence of the Adhocracy archetype in the FFCT.

Medical student milestones in emergency medicine.

Medical education is a continuum from medical school through residency to unsupervised clinical practice. There has been a movement toward competency-based medical education prompted by the Accreditation Council for Graduate Medical Education (ACGME) using milestones to assess competence. While implementation of milestones for residents sets specific standards for transition to internship, there exists a need for the development of competency-based instruments to assess medical students as they progress toward internship. The objective of this study was to develop competency-based milestones for fourth-year medical students completing their emergency medicine (EM) clerkships (regardless of whether the students were planning on entering EM) using a rigorous method to attain validity evidence. A literature review was performed to develop a list of potential milestones. An expert panel, which included a medical student and 23 faculty members (four program directors, 16 clerkship directors, and five assistant deans) from 19 different institutions, came to consensus on these milestones through two rounds of a modified Delphi protocol. The Delphi technique builds content validity and is an accepted method to develop consensus by eliciting expert opinions through multiple rounds of questionnaires. Of the initial 39 milestones, 12 were removed at the end of round 1 due to low agreement on importance of the milestone or because of redundancy with other milestones. An additional 12 milestones were revised to improve clarity or eliminate redundancy, and one was added based on expert panelists' suggestions. Of the 28 milestones moving to round 2, consensus with a high level of agreement was achieved for 24. These were mapped to the ACGME EM residency milestone competency domains, as well as the Association of American Medical Colleges (AAMC) core entrustable professional activities for entering residency to improve content validity. This study found consensus support by experts for a list of 24 milestones relevant to the assessment of fourth-year medical student performance by the completion of their EM clerkships. The findings are useful for development of a valid method for assessing medical student performance as students approach residency.

Milestone myths and misperceptions.

In 1999 the Accreditation Council for Graduate Medical Education (ACGME) introduced 6 clinical competencies.1 The longitudinal integration of these competencies into educational curricula and assessment of residents and fellows was intended to also generate outcomes assessments–based information on the educational effectiveness of training programs. In 2008, to further this process, ACGME Chief Executive Officer Thomas J. Nasca, MD, announced his vision for “The Next Step in the Outcomes-Based Accreditation Project,”2 which became the Next Accreditation System (NAS). This vision was notable for 2 elements: development of specialty-specific, competency-based Milestones, along with design and implementation of Milestone assessment tools. The ultimate goal was to further transform graduate medical education from a process focus to an outcomes-based focus. In July 2013, 7 specialties that had volunteered to be early adopters of the NAS began their journey down this exciting and relatively uncharted path.3 As a program director and member of the committee who created the Emergency Medicine Milestones, I have been struck by the incredible diversity of approaches during the Milestone rollout process.4,5 Given this diversity, inevitably some applications of the Milestone framework may not result in the lofty aspirational goals that we hoped for. The Milestones, along with other elements of the NAS, are intended to promote program improvements via self-study driven by objective data. The Next Accreditation System emphasizes trainee assessment based on observable behaviors, using stable, reproducible methods that eliminate that old educational saw, “I know it when I see it.” The community designing the Milestones aimed to make the assessment processes transparent and to stimulate creativity in the graduate medical education community with this paradigm shift. As the Milestones are beginning to be used, there are some misperceptions about the Milestones and misinterpretations in how they should be used (box). This perspective seeks to offer added clarity and address these misperceptions. Box Common Misinterpretations and Misperceptions About Milestones Milestones are a global rating scale for all specialties Milestone levels accurately correspond to year of training The Clinical Competency Committee is a brand new process The assessment methodologies that are currently in place have no utility in Milestones The assessment methodologies for the Clinical Competency Committee are yet to be designed Milestones will eliminate grade inflation Milestone reporting will be so onerous that it will interfere with training Milestones should not be shared with residents

Gastroenterology Educators Must Be Aware that the Perceived Difficulty of Core Competencies Change based Upon Fellowship Training Year

Purpose: The Accreditation Council for Graduate Medical Education (ACGME) is reassessing how it evaluates and accredits post-graduate training programs in the United States. It has established that training programs should structure their curriculum around six core competencies. These competencies categorize the skills to be mastered during training. Training programs will be assessed on trainees successfully reaching milestones within these competencies. As a trainee advances through the training program, it is reasonable to expect that there will be a shift in the perceived difficulty of mastering particular core competencies. Competency-based milestones and curricula will need to be designed to account for these changes in perception. This study sought to identify the shift in perceived difficulty in mastering the core competencies as they relate to training year. Methods: During monthly sessions between gastroenterology fellows and the program director at a university-based training program, issues related to core competencies are routinely discussed. The purpose of these meetings is to assess the effectiveness of the present curriculum and address concerns raised by the trainees. The ACGME core competencies include Patient Care, Medical Knowledge, Practice-Based Learning and Improvement, Interpersonal and Communication Skills, Professionalism and Systems-Based Practice. During the end of year review, fellows are asked to rank the core competencies as they perceive them from most challenging to least challenging at their level of training. Information collected during these sessions is used to enhance the curriculum of the training program. Results: Medical Knowledge and Systems-Based Practice ranked within the top three most challenging core competency amongst fellows in all three years. Medical Knowledge was considered the most challenging core competency in the first and second year of fellowship training. Systems-Based Practice ranked highest amongst fellows in the third year of training. Professionalism and Interpersonal Skills and Communication were considered the most facile competencies amongst trainees in all three years. Conclusion: Expected in 2014, trainees' progress and development will be measured by successful completion of competency milestones. As trainees advance through the fellowship program, there is a shift in the perceived difficulty of the core competencies. While Medical knowledge is the focus of the first year fellows, the senior fellows perceive Systems-based Practice as the greatest challenge. It will be imperative that competency-based milestones and curricula be designed to address the changing perceptions held by the trainees.