Milestone Ratings Research Articles

Longitudinal assessment of milestone development among internal medicine residents in Taiwan

The Accreditation Council for Graduate Medical Education (ACGME) milestones have been implemented in residency training worldwide. We investigated the development of individual competency in first-year residents (R1) and second-year postgraduate students (PGY2) who received internal medicine training in Taiwan. A multicenter observational cohort study was conducted to evaluate the competency-based milestone evaluation designed by the Taiwan Society of Internal Medicine in 2019. The evaluation was based on the ACGME-accredited milestone ratings. Periodic evaluation of milestone achievements of R1 and PGY2, who entered the internal medicine residency training at six medical centers, was performed. Each resident was evaluated every 3 months. Among the 98 R1 enrolled in 2019, substantial improvement in sub-competencies, including skill in performing procedures (Patient Care 4), clinical knowledge (Medical Knowledge 1), knowledge of diagnostic testing and procedures (Medical Knowledge 2), and identify impact the cost of health care and practices cost-effective care (Systems Based Practice 3) during the two years of training. Among the 107 R1 and 46 PGY2 enrolled in 2020, no significant difference in baseline milestone ratings was observed. However, the milestone assessments of R1 in 2020 showed improvement in nearly all sub-competencies compared with the stationary status of PGY2 in 2020. We demonstrate the application of ACGME-based accredited milestone ratings to target the educational goals of internal medicine residency training in Taiwan. Differences in milestone ratings between different PGY training systems exist. The long-term impact of performance among different PGY training systems requires further investigation.

Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents.

A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency. This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings. From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05). The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.

ACGME Milestones in the Real World: A Qualitative Study Exploring Response Process Evidence.

Since the Accreditation Council for Graduate Medical Education (ACGME) introduced the Milestones in 2013, the body of validity evidence supporting their use has grown, but there is a gap with regard to response process. The purpose of this study is to qualitatively explore validity evidence pertaining to the response process of individual Clinical Competency Committee (CCC) members when assigning Milestone ratings to a resident. Using a constructivist paradigm, we conducted a thematic analysis of semi-structured interviews with 8 Transitional Year (TY) CCC members from 4 programs immediately following a CCC meeting between November and December 2020. Participants were queried about their response process in their application of Milestone assessment. Analysis was iterative, including coding, constant comparison, and theming. Participant interviews identified an absence of formal training and a perception that Milestones are a tool for resident assessment without recognizing their role in program evaluation. In describing their thought process, participants reported comparing averaged assessment data to peers and time in training to generate Milestone ratings. Meaningful narrative comments, when available, differentiated resident performance from peers. When assessment data were absent, participants assumed an average performance. Our study found that the response process used by TY CCC members was not always consistent with the dual purpose of the Milestones to improve educational outcomes at the levels of residents and the program.

An Empirical Investigation Into Milestones Factor Structure Using National Data Derived From Clinical Competency Committees.

To investigate whether milestone data obtained from clinical competency committee (CCC) ratings in a single specialty reflected the 6 general competency domains framework. The authors examined milestone ratings from all 275 U.S. Accreditation Council for Graduate Medical Education-accredited categorical obstetrics and gynecology (OBGYN) programs from July 1, 2018, to June 30, 2019. The sample size ranged from 1,371 to 1,438 residents from 275 programs across 4 postgraduate years (PGYs), each with 2 assessment periods. The OBGYN milestones reporting form consisted of 28 subcompetencies under the 6 general competency domains. Milestone ratings were determined by each program's CCC. Intraclass correlations (ICCs) and design effects were calculated for each subcompetency by PGY and assessment period. A multilevel confirmatory factor analysis (CFA) perspective was used, and the pooled within-program covariance matrix was obtained to compare the fit of the 6-domain factor model against 3 other plausible models. Milestone ratings from 5,618 OBGYN residents were examined. Moderate to high ICCs and design effects greater than 2.0 were prevalent among all subcompetencies for both assessment periods, warranting the use of the multilevel approach in applying CFA to the milestone data. The theory-aided split-patient care (PC) factor model, which used the 6 general competency domains but also included 3 factors within the PC domain (obstetric technical skills, gynecology technical skills, and ambulatory care), was consistently shown as the best-fitting model across all PGYs by assessment period conditions, except for one. The findings indicate that in addition to using the 6 general competency domains framework in their rating process, CCCs may have further distinguished the PC competency domain into 3 meaningful factors. This study provides internal structure validity evidence for the milestones within a single specialty and may shed light on CCCs' understanding of the distinctive content embedded within the milestones.

The Accreditation Council for Graduate Medical Education Milestones in Integrated Plastic Surgery Programs: How Competency-Based Assessment Has Been Implemented.

The plastic surgery milestones are a central component of resident assessment. The authors performed a survey to evaluate how milestones have been implemented across integrated plastic surgery programs and how faculty perceive the Milestones Project has impacted their program. A 25-question survey was directed to the head of the clinical competency committee of all 82 integrated plastic surgery programs through the American Council of Academic Plastic Surgeons. The survey queried the composition of the committee, how ratings are generated, and the data used in generating these ratings. Committee leaders from 57 programs (69.5 percent) responded to the survey. For most programs (73 percent), one faculty member completed milestone ratings for each resident and reviewed them with the residents. To determine milestone ratings, 92 percent utilized resident assessment after every rotation, and 63 percent utilized in-service examination scores. For documented resident assessment overall, 96 percent of programs assess residents after every rotation; 37 percent asses after every procedure. Feedback is most frequently provided to residents after every rotation (52 percent), rather than only during committee reviews (32 percent) or after every documented assessment (16 percent). Sixty-four percent of respondents did not believe that milestones have helped in the mentorship role. Implementation of the milestones has varied among integrated plastic surgery programs. Structured training for core faculty directed to methods of assessment will make milestones a more effective a tool by which to improve resident evaluation and education. These findings provide a key data set by which to revise the milestones for their second iteration.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice. To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents. This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019. Longitudinal formative assessment using the Milestones assessment system reported to the ACGME. To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories. This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth. These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

The Use of Learning Analytics to Enable Detection of Underperforming Trainees: An Analysis of National Vascular Surgery Trainee ACGME Milestones Assessment Data.

This study aims to investigate at-risk scores of semiannual Accreditation Council for Graduate Medical Education (ACGME) Milestone ratings for vascular surgical trainees' final achievement of competency targets. ACGME Milestones assessments have been collected since 2015 for Vascular Surgery. It is unclear whether milestone ratings throughout training predict achievement of recommended performance targets upon graduation. National ACGME Milestones data were utilized for analyses. All trainees completing 2-year vascular surgery fellowships in June 2018 and 5-year integrated vascular surgery residencies in June 2019 were included. A generalized estimating equations model was used to obtain at-risk scores for each of the 31 subcompetencies by semiannual review periods, to estimate the probability of trainees achieving the recommended graduation target based on their previous ratings. A total of 122 vascular surgery fellows (VSFs) (95.3%) and 52 integrated vascular surgery residents (IVSRs) (100%) were included. VSFs and IVSRs did not achieve level 4.0 competency targets at a rate of 1.6% to 25.4% across subcompetencies, which was not significantly different between the 2 groups for any of the subcompetencies ( P = 0.161-0.999). Trainees were found to be at greater risk of not achieving competency targets when lower milestone ratings were assigned, and at later time-points in training. At a milestone rating of 2.5, with 1 year remaining before graduation, the at-risk score for not achieving the target level 4.0 milestone ranged from 2.9% to 77.9% for VSFs and 33.3% to 75.0% for IVSRs. The ACGME Milestones provide early diagnostic and predictive information for vascular surgery trainees' achievement of competence at completion of training.

IMPLEMENTING FAMILY THERAPY OBSERVED STRUCTURED CLINICAL EXAMS (OSCES) TO PROVIDE COMPETENCY-BASED FORMATIVE EVALUATION TO CHILD AND ADOLESCENT PSYCHIATRY TRAINEES

We will illustrate the use of Observed Structured Clinical Examinations (OSCEs) in child and adolescent psychiatry training to provide formative feedback on family therapy competencies. Specifically, we have 8 years of family therapy OSCE evaluative data from actors, faculty, and trainees targeting the following skills: 1) developing a systemic alliance; 2) reframing a presenting problem in systemic terms; 3) managing conflict; 4) developing a systemic formulation; and 5) exhibiting self-reflective practice. We will utilize a combination of brief didactic sessions interspersed with video examples, role-play, and hands-on interactive learning experiences to illustrate the use of OSCEs within a family therapy training program for child and adolescent psychiatry fellows. We will utilize smaller breakout groups for discussion and role-playing. Participants will be provided with handouts of the evaluation forms used in the family therapy OSCE. Workshop participants will learn how OSCEs can be utilized to provide formative feedback on family therapy competencies in the training of child and adolescent psychiatry fellows. They will understand the process for implementing family therapy OSCEs and have an opportunity to participate in assessing trainee strengths and weaknesses, providing feedback, and developing recommendations for ongoing training. They will also understand how the feedback from these OSCEs can be used to inform milestone ratings. Participants will have an opportunity to plan and problem solve as to how to implement a similar program with their own trainees. Family therapy OSCEs can be utilized to provide child and adolescent psychiatry trainees with structured, multi-informant feedback on family therapy competencies and to inform Accreditation Council for Graduate Medical Education (ACGME) milestone ratings.

Natural Language Processing to Estimate Clinical Competency Committee Ratings

Residency program faculty participate in clinical competency committee (CCC) meetings, which are designed to evaluate residents' performance and aid in the development of individualized learning plans. In preparation for the CCC meetings, faculty members synthesize performance information from a variety of sources. Natural language processing (NLP), a form of artificial intelligence, might facilitate these complex holistic reviews. However, there is little research involving the application of this technology to resident performance assessments. With this study, we examine whether NLP can be used to estimate CCC ratings. We analyzed end-of-rotation assessments and CCC assessments for all surgical residents who trained at one institution between 2014 and 2018. We created models of end-of-rotation assessment ratings and text to predict dichotomized CCC assessment ratings for 16 Accreditation Council for Graduate Medical Education (ACGME) Milestones. We compared the performance of models with and without predictors derived from NLP of end-of-rotation assessment text. We analyzed 594 end-of-rotation assessments and 97 CCC assessments for 24 general surgery residents. The mean (standard deviation) for area under the receiver operating characteristic curve (AUC) was 0.84 (0.05) for models with only non-NLP predictors, 0.83 (0.06) for models with only NLP predictors, and 0.87 (0.05) for models with both NLP and non-NLP predictors. NLP can identify language correlated with specific ACGME Milestone ratings. In preparation for CCC meetings, faculty could use information automatically extracted from text to focus attention on residents who might benefit from additional support and guide the development of educational interventions.

Assessing the Transition of Training in Health Systems Science From Undergraduate to Graduate Medical Education.

The American Medical Association Accelerating Change in Medical Education (AMA-ACE) consortium proposes that medical schools include a new 3-pillar model incorporating health systems science (HSS) and basic and clinical sciences. One of the goals of AMA-ACE was to support HSS curricular innovation to improve residency preparation. This study evaluates the effectiveness of HSS curricula by using a large dataset to link medical school graduates to internship Milestones through collaboration with the Accreditation Council for Graduate Medical Education (ACGME). ACGME subcompetencies related to the schools' HSS curricula were identified for internal medicine, emergency medicine, family medicine, obstetrics and gynecology (OB/GYN), pediatrics, and surgery. Analysis compared Milestone ratings of ACE school graduates to non-ACE graduates at 6 and 12 months using generalized estimating equation models. At 6 months both groups demonstrated similar HSS-related levels of Milestone performance on the selected ACGME competencies. At 1 year, ACE graduates in OB/GYN scored minimally higher on 2 systems-based practice (SBP) subcompetencies compared to non-ACE school graduates: SBP01 (1.96 vs 1.82, 95% CI 0.03-0.24) and SBP02 (1.87 vs 1.79, 95% CI 0.01-0.16). In internal medicine, ACE graduates scored minimally higher on 3 HSS-related subcompetencies: SBP01 (2.19 vs 2.05, 95% CI 0.04-0.26), PBLI01 (2.13 vs 2.01; 95% CI 0.01-0.24), and PBLI04 (2.05 vs 1.93; 95% CI 0.03-0.21). For the other specialties examined, there were no significant differences between groups. Graduates from schools with training in HSS had similar Milestone ratings for most subcompetencies and very small differences in Milestone ratings for only 5 subcompetencies across 6 specialties at 1 year, compared to graduates from non-ACE schools. These differences are likely not educationally meaningful.

Gender Differences in Milestone Ratings and Medical Knowledge Examination Scores Among Internal Medicine Residents.

To examine whether there are group differences in milestone ratings submitted by program directors working with clinical competency committees (CCCs) based on gender for internal medicine (IM) residents and whether women and men rated similarly on milestones perform comparably on subsequent in-training and certification examinations. This national retrospective study examined end-of-year medical knowledge (MK) and patient care (PC) milestone ratings and IM In-Training Examination (IM-ITE) and IM Certification Examination (IM-CE) scores for 2 cohorts (2014-2017, 2015-2018) of U.S. IM residents at ACGME-accredited programs. It included 20,098/21,440 (94%) residents, with 9,424 women (47%) and 10,674 men (53%). Descriptive statistics and differential prediction techniques using hierarchical linear models were performed. For MK milestone ratings in PGY-1, men and women showed no statistical difference at a significance level of .01 (P = .02). In PGY-2 and PGY-3, men received statistically higher average MK ratings than women (P = .002 and P < .001, respectively). In contrast, men and women received equivalent average PC ratings in each PGY (P = .47, P = .72, and P = .80, for PGY-1, PGY-2, and PGY-3, respectively). Men slightly outperformed women with similar MK or PC ratings in PGY-1 and PGY-2 on the IM-ITE by about 1.7 and 1.5 percentage points, respectively, after adjusting for covariates. For PGY-3 ratings, women and men with similar milestone ratings performed equivalently on the IM-CE. Milestone ratings were largely similar for women and men. Generally, women and men with similar MK or PC milestone ratings performed similarly on future examinations. Although there were small differences favoring men on earlier examinations, these differences disappeared by the final training year. It is questionable whether these small differences are educationally or clinically meaningful. The findings suggest fair, unbiased milestone ratings generated by program directors and CCCs assessing residents.

Using Learning Analytics to Examine Achievement of Graduation Targets for Systems-Based Practice and Practice-Based Learning and Improvement: A National Cohort of Vascular Surgery Fellows

Surgeons provide patient care in complex health care systems and must be able to participate in improving both personal performance and the performance of the system. The Accreditation Council for Graduate Medical Education (ACGME) Vascular Surgery Milestones are utilized to assess vascular surgery fellows' (VSF) achievement of graduation targets in the competencies of Systems Based Practice (SBP) and Practice Based Learning and Improvement (PBLI). We investigate the predictive value of semiannual milestones ratings for final achievement within these competencies at the time of graduation. National ACGME milestones data were utilized for analysis. All trainees entering the 2-year vascular surgery fellowship programs in July 2016 were included in the analysis (n = 122). Predictive probability values (PPVs) were obtained for each SBP and PBLI sub-competencies by biannual review periods, to estimate the probability of VSFs not reaching the recommended graduation target based on their previous milestones ratings. The rate of nonachievement of the graduation target level 4.0 on the SBP and PBLI sub-competencies at the time of graduation for VSFs was 13.1-25.4%. At the first time point of assessment, 6 months into the fellowship program, the PPV of the SBP and PBLI milestones for nonachievement of level 4.0 upon graduation ranged from 16.3-60.2%. Six months prior to graduation, the PPVs across the 6 sub-competencies ranged from 14.6-82.9%. A significant percentage of VSFs do not achieve the ACGME Vascular Surgery Milestone targets for graduation in the competencies of SBP and PBLI, suggesting a need to improve curricula and assessment strategies in these domains across vascular surgery fellowship programs. Reported milestones levels across all time point are predictive of ultimate achievement upon graduation and should be utilized to provide targeted feedback and individualized learning plans to ensure graduates are prepared to engage in personal and health care system improvement once in unsupervised practice.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

A Validity Framework for Effective Analysis and Interpretation of Milestones Data.

Assessment of Learning, for Learning: Operationalizing Milestones Data for Program-Level Improvement.

Assessment of Learning, for Learning: Operationalizing Milestones Data for Program-Level Improvement.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

The Science of Effective Group Process: Lessons for Clinical Competency Committees.

Correlations between Accreditation Council for Graduate Medical Education Obstetrics and Gynecology Milestones and American Board of Obstetrics and Gynecology qualifying examination scores: an initial validity study

Since the launch of the Outcome Project in 2001, the graduate medical education community has been working to implement the 6 general competencies. In 2014, all Obstetrics and Gynecology residency programs implemented specialty-specific milestones to advance competency-based assessment. Each clinical competency committee of the Obstetrics and Gynecology program assesses all residents twice a year on the milestones. These data are reported to the Accreditation Council for Graduate Medical Education as part of a continuous quality improvement effort in graduate medical education. This study aimed to evaluate the correlation between the Accreditation Council for Graduate Medical Education Obstetrics and Gynecology Milestones and residency program graduates' performance on the American Board of Obstetrics and Gynecology qualifying (written) examination. We conducted a validity study of all graduating (postgraduate year 4) Obstetrics and Gynecology residents in 2017 within Accreditation Council for Graduate Medical Education-accredited United States training programs (1260 residents from 242 programs). This cohort of residents began receiving milestone assessments during their postgraduate year 2 in 2014; the first-year milestones were implemented for all Accreditation Council for Graduate Medical Education-accredited Obstetrics and Gynecology programs. This cohort completed their sixth and final milestone assessment at graduation in June 2017 for a total of 6 periods of milestone assessments. Data regarding each resident's milestone ratings in each of the 28 Accreditation Council for Graduate Medical Education subcompetencies for Obstetrics and Gynecology were assessed for their association with candidates' American Board of Obstetrics and Gynecology qualifying examination scores using a generalized estimating equation regression model. Data were available and analyzed from 1184 residents from 240 programs, representing 94% of the total academic year 2017 graduates of Obstetrics and Gynecology residency training programs. There was a substantial association between most milestone ratings at the 6 assessment points and candidates' performance on the American Board of Obstetrics and Gynecology qualifying examination. The strongest associations with the American Board of Obstetrics and Gynecology were within all 7 of the subcompetencies of Medical Knowledge (range of slope correlation coefficients at final milestone ratings 3.84-5.17; slope coefficients can be interpreted as the gain in qualifying examination points per unit increase in milestone level). At the final milestone assessment, but more modest associations with the American Board of Obstetrics and Gynecology qualifying examination scores were also seen with 9 of the 11 Patient Care and Procedural Skills subcompetencies, the 2 of 2 Practice-Based Learning and Improvement subcompetencies, the 2 of 2 Systems-Based Practice subcompetencies, and 2 of the 3 Professionalism subcompetencies. Only 1 of the 3 Interpersonal and Communication Skills subcompetencies was associated with American Board of Obstetrics and Gynecology qualifying examination scores. The pattern of associations between the qualifying examination scores and milestone ratings for the 2017 graduating cohort of Obstetrics and Gynecology residents followed a logical pattern, with the strongest associations seen in Medical Knowledge, and lower to no associations in subcompetencies not as effectively assessed on multiple-choice examinations. Although some positive associations were noted for non-Medical Knowledge milestones, these associations could be caused by correlational rating errors with further study needed to better understand these patterns.

Do Milestone Ratings Predict Physical Medicine and Rehabilitation Board Certification Examination Scores?

The Accreditation Council of Graduate Medical Education developed the Milestones to assist training programs in assessing resident physicians in the context of their participation in Accreditation Council of Graduate Medical Education-accredited training programs. Biannual assessments are done over a resident's entire training period to define the trajectory in achieving specialty-specific competencies. As part of its process of initial certification, the American Board of Physical Medicine and Rehabilitation requires successful completion of two examinations administered approximately 9 mos apart. The Part I Examination measures a single dimensional construct, physical medicine and rehabilitation medical knowledge, whereas Part II assesses the application of medical and physiatric knowledge to multiple domains, including data acquisition, problem solving, patient management, systems-based practice, and interpersonal and communication skills through specific patient case scenarios. This study aimed to investigate the validity of the Milestones by demonstrating its association with performance in the American Board of Physical Medicine and Rehabilitation certifying examinations. A cohort of 233 physical medicine and rehabilitation trainees in 3-yr residency programs (postgraduate year 2 entry) in the United States from academic years 2014-2016, who also took the American Board of Physical Medicine and Rehabilitation Parts I and II certifying examinations between 2016 and 2018, were included in the study. Milestones ratings in four distinct observation periods were correlated with scores in the American Board of Physical Medicine and Rehabilitation Parts I and II Examinations. Milestones ratings of medical knowledge (but not patient care, professionalism, problem-based learning, interpersonal and communication skills, and systems-based practice) predicted performance in subsequent Part I American Board of Physical Medicine and Rehabilitation Examination, but none of the Milestone ratings correlated with Part II Examination scaled scores.

No Intern Left Behind: Using a Dedicated Transition to Discipline Block Improves Resident Outcomes.

This article was migrated. The article was marked as recommended. Introduction: A program was developed for incoming PGY-1 residents using the Accreditation Council for Graduate Medical Education milestones ratings. This program detects critical deficiencies and works to correct them as early as possible. Methods: A one month period was used for this transition to discipline block to identify at-risk learners. The block utilized cased-based discussions, interactive lectures, simulations, and clinical core rotations. All activities were tied to milestones measures to recognize deficiencies and provided a goal to correct the individual's progression. Results: Interns that completed the transition to discipline block were compared to the most recent previous class at the same institution. The same number of individuals with critical deficits were found in each class at first milestones rating (4 deficits per class, p value 1.0). The intervention classes had critical deficiencies recognized earlier and all identified deficiencies were extinguished earlier. Medical knowledge as compared by In-Training Examination percentile scores improved (Pre-Intervention Mean Percentile 28.9, Post-Intervention Mean Percentile 49.5, p value 0.005). Discussion: A milestones-based transition to residency block identified critical deficiencies earlier, which allowed for earlier intervention and improvement in resident performance. A similar process may benefit other residency programs.

Using Multisource Feedback to Assess Resident Communication Skills: Adding a New Dimension to Milestone Data.

Background: The Accreditation Council for Graduate Medical Education (ACGME) requires evaluation of residents’ communication skills. These evaluations should involve assessments from a variety of persons with different perspectives and opportunities to observe resident behavior. Our objectives with this study were to determine if parents, nurses, and physicians significantly differed in their ratings of residents’ communication skills; to ascertain the degree of association between these evaluations and ACGME milestone data; and to elicit feedback from residents about the specificity and usefulness of this type of evaluation compared to the evaluations they were typically provided.Methods: During the 2016-2017 academic year, parents of patients ready for discharge, nurses, and attending physicians completed evaluations of resident communication skills. A repeated measures multivariate analysis of variance compared communication skills scores across the 3 groups of raters. Resident ACGME milestone ratings for interpersonal and communication skills were correlated with the communication skills evaluations. Residents rated the specificity and usefulness of the 360-degree evaluations.Results: Parents rated residents’ communication skills significantly higher than nurses and physicians rated them. We found no significant difference between the nurse and physician ratings. A significant correlation was found between resident ratings by physicians and ACGME milestone data. Residents found the feedback from these evaluations to be more specific and useful in delineating their communication strengths and weaknesses than typical milestone feedback.Conclusion: Parents added a unique perspective about residents’ communication and should be included in resident evaluation when feasible. Residents appreciated the specificity and usefulness of the evaluation instrument.

Family Medicine Residency Graduates' Preparation for Quality Improvement Leadership.

Training in quality improvement (QI) is a standard component of family medicine residency education. Graduating family medicine residents' ability to lead QI initiatives is unknown. We assessed the preparedness of graduating family medicine residents to lead QI projects and to identify factors that may increase such readiness. Milestone data for all graduating family medicine residents were linked to a practice demographic questionnaire completed by the same residents who registered for the American Board of Family Medicine certification examination between 2014 and 2017. The change in self-assessed QI preparedness over time and its association with faculty-assigned milestone ratings were examined using descriptive and regression analyses. The questionnaire had a 100% response rate (12 208 responded). Between 2014 and 2017, the percentage of residents who self-reported being "extremely" or "moderately" prepared to lead QI projects increased from 72.7% (2208 of 3038) to 75.8% (2434 of 3210, P = .009). Self-reported QI team leadership was associated with 93% higher odds of feeling extremely prepared compared to moderately prepared (odds ratio 1.93, 95% CI 1.58-2.35). The average midyear faculty-assigned milestone rating for QI among residents who felt "extremely" prepared was 3.28 compared to 3.14 among those who felt "not at all" prepared. Over the past 4 years, family medicine residents' self-assessed preparedness to lead QI projects has barely increased. There was no correlation between self-assessed preparation and faculty-assigned milestone rating. However, we found a small association between self-reported QI leadership and self-assessed QI preparedness.