Milestone Ratings Research Articles

Mid-Year Medical Knowledge Milestones and ABSITE Scores in First-Year Surgery Residents

Accreditation Council for Graduate Medical Education (ACGME) Surgery milestone ratings in the "Knowledge of Diseases and Conditions" (MK1) sub competency have been shown to correlate with American Board of Surgery In Training Examination (ABSITE) scores, and hypothesized to predict them. To better assess the predictive value of the MK1 milestone and avoid the potential bias caused by previous years' ABSITE scores, we designed a study including only first-year (PGY-1) residents and analyzed the correlation between their mid-year MK1 ratings and their scores in the ABSITE they took approximately a month later. De-identified United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, mid-year MK1 milestone ratings and the subsequent ABSITE standard scores for the five academic years from 2014-2015 to 2018-2019 were collected and tabulated for 247 PGY-1 preliminary- and categorical-track residents from ten ACGME-accredited surgery residency programs. The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical residents' subset. Notably, controlling for all other independent predictors, each half-point increase in MK1 rating was associated with a 25-point increase in ABSITE score. Preliminary residents performed significantly worse on the ABSITE, and their scores did not correlate significantly with their MK1 ratings. The mid-year rating of PGY-1 residents' MK1 was predictive of their subsequent first ABSITE score for the entire cohort and for the categorical but not the preliminary residents. This finding suggests that evaluators correctly rated MK1 higher in the categorical residents who did perform better on the subsequent ABSITE.

74. LACK OF CORRELATION BETWEEN USMLE SCORES AND PERFORMANCE IN PEDIATRICS RESIDENCY TRAINING BASED ON ACGME MILESTONES RATINGS

Introduction The United States Medical Licensing Exam (USMLE) was designed to determine whether medical trainees are competent to become licensed physicians, not necessarily as a means of providing a comparative gradient on students’ knowledge. Despite this, many residency programs use USMLE scores in the residency selection process as indicators as to whether physicians will succeed in residency. Not only is this not the intended use of the USMLE, some previous studies have suggested that USMLE scores are not effective indicators of a physician's success during residency. One method of assessing residents’ progress through residency training are the Accreditation Council for Graduate Medical Education (ACGME) Milestones in the six clinical competency domains of Patient Care, Medical Knowledge, Systems Based Practice, Practice-Based Learning and Improvement, Professionalism, and Interpersonal Communications Skills. Ratings along these milestones are completed by supervising physicians who observe and assess residents in the clinical setting. No previous literature has tried to link USMLE Step scores to resident performance using Milestone ratings. Methods USMLE Step 1 and 2 Scores and Milestones Ratings in twenty-one different competencies were collected over five years (2012 - 2017) from 187 residents enrolled in a single large pediatrics residency program. Correlations were examined using Pearson's correlation coefficients. Results Among the residents studied, the mean score of Step 1 was 236, while the mean score of Step 2 was 250. While Step 1 and 2 had correlation (r2 = -0.628), no correlation was found between Step 1 and overall Milestones average (r2 = -0.130 and Conclusion These results suggest that USMLE scores have no correlation to success in residency training as measured by progression along competency-based milestones. These confirm previous claims about the limited utility of USMLE scores in predicting success in residency training.

A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents' Competence.

To investigate whether clinical competency committees (CCCs) were consistent in applying milestone ratings for first-year residents over time or whether ratings increased or decreased. Beginning in December 2013, the Accreditation Council for Graduate Medical Education (ACGME) initiated a phased-in requirement for reporting milestones; emergency medicine (EM), diagnostic radiology (DR), and urology (UR) were among the earliest reporting specialties. The authors analyzed CCC milestone ratings of first-year residents from 2013 to 2016 from all ACGME-accredited EM, DR, and UR programs for which they had data. The number of first-year residents in these programs ranged from 2,838 to 2,928 over this time period. The program-level average milestone rating for each subcompetency was regressed onto the time of observation using a random coefficient multilevel regression model. National average program-level milestone ratings of first-year residents decreased significantly over the observed time period for 32 of the 56 subcompetencies examined. None of the other subcompetencies showed a significant change. National average in-training examination scores for each of the specialties remained essentially unchanged over the time period, suggesting that differences between the cohorts were not likely an explanatory factor. The findings indicate that CCCs tend to become more stringent or maintain consistency in their ratings of beginning residents over time. One explanation for these results is that CCCs may become increasingly comfortable in assigning lower ratings when appropriate. This finding is consistent with an increase in confidence with the milestone rating process and the quality of feedback it provides.

Are Otolaryngology Milestones Predictive of Otolaryngology Training Examination Scores?

Otolaryngology residents take the otolaryngology training examination (OTE) yearly to assess their fund of knowledge. The Accreditation Council for Graduate Medical Education (ACGME) milestone evaluations are also conducted semiannually. Accurate prediction of training examination performance allows identification of residents who are performing well and those who need targeted remediation. Prior studies in other specialties have attempted to use milestone evaluations to help predict in-training examination scores. In this study, we aim to identify whether ACGME milestone evaluation scores predict OTE performance. Milestone ratings and OTE scores for residents at 2 US otolaryngology residency programs were collected. Multivariate analysis was achieved using linear mixed modeling. We considered a 2-tailed P value of ≤ .05 as statistically significant. Two US otolaryngology residency programs. Forty-eight otolaryngology residents postgraduate years 2 to 5. Otolaryngology training examination scores and ACGME milestone evaluations were collected from 48 residents from postgraduate year 2 to 5 between the years 2014 and 2017. One hundred eight OTE scores were available. Linear mixed-effect models were constructed, and after adjusting for level of training and OTE year, the total milestone rating made a negligible impact in estimating OTE percentage correct (β = -.01, P = .9). Similarly, total milestone rating demonstrated a minimal contribution in approximating OTE national stanine score after adjusting for the level of training (β = -.003, P = .9). In our study, ACGME milestone evaluations are not predictive of residents' OTE performance. What these milestone evaluation data mean and how they should be used continues to be an unanswered question. We should aim to identify the most effective applications of the milestone data collected yearly by otolaryngology programs.

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience.

The Accreditation Council for Graduate Medical Education Milestones were created as a criterion-based framework to promote competency-based education during graduate medical education. Despite widespread implementation across subspecialty programs, extensive validity evidence supporting the use of milestones within fellowship training is lacking. We assessed the construct and response process validity of milestones in subspecialty fellowship programs in an academic medical center. From 2014-2016, we performed a single center retrospective cohort analysis of milestone data from fellows across 5 programs. We analyzed summary statistics and performed multivariable linear regression to assess change in milestone ratings by training year and variability in ratings across fellowship programs. Finally, we examined a subset of Professionalism and Interpersonal and Communication Skills subcompetencies from the first 6 months of training to identify the proportion of fellows deemed "ready for independent practice" in these domains. Milestone data were available for 68 fellows, with 75 933 unique subcompetency ratings. Multivariable linear regression, adjusted for subcompetency and subspecialty, revealed an increase of 0.17 (0.16-0.19) in ratings with each postgraduate year level increase (P < .005), as well as significant variation in milestone ratings across subspecialties. For the Professionalism and Interpersonal and Communication Skills domains, mean ratings within the first 6 months of training were 3.78 and 3.95, respectively. We noted a minimal upward trend of milestone ratings in subspecialty training programs, and significant variability in implementing milestones across differing subspecialties. This may suggest possible difficulties with the construct validity and response process of the milestone system in certain medical subspecialties.

National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-up.

To evaluate validity evidence for internal medicine milestone ratings across programs for three resident cohorts by quantifying "not assessable" ratings; reporting mean longitudinal milestone ratings for individual residents; and correlating medical knowledge ratings across training years with certification examination scores to determine predictive validity of milestone ratings for certification outcomes. This retrospective study examined milestone ratings for postgraduate year (PGY) 1-3 residents in U.S. internal medicine residency programs. Data sources included milestone ratings, program characteristics, and certification examination scores. Among 35,217 participants, there was a decreased percentage with "not assessable" ratings across years: 1,566 (22.5%) PGY1s in 2013-2014 versus 1,219 (16.6%) in 2015-2016 (P = .01), and 342 (5.1%) PGY3s in 2013-2014 versus 177 (2.6%) in 2015-2016 (P = .04). For individual residents with three years of ratings, mean milestone ratings increased from around 3 (behaviors of an early learner or advancing resident) in PGY1 (ranging from a mean of 2.73 to 3.19 across subcompetencies) to around 4 (ready for unsupervised practice) in PGY3 (mean of 4.00 to 4.22 across subcompetencies, P < .001 for all subcompetencies). For each increase of 0.5 units in two medical knowledge (MK1, MK2) subcompetency ratings, the difference in examination scores for PGY3s was 19.5 points for MK1 (P < .001) and 19.0 for MK2 (P < .001). These findings provide evidence of validity of the milestones by showing how training programs have applied them over time and how milestones predict other training outcomes.

A Cross-Specialty Examination of Resident Error Disclosure and Communication Skills Using Simulation.

Disclosure of medical errors is important to patients and physicians, but formal disclosure training during the graduate medical education curriculum is limited. We examined resident competence related to error disclosure, using standardized patient (SP) ratings of resident communication skills. All first-year residents from medicine, radiology, emergency medicine, orthopedic surgery, and neurological surgery completed a 20-minute simulated session in which they were provided background information on a medical error they had made and were asked to disclose the error to an SP acting as a family member. Residents were then debriefed and completed a postscenario questionnaire. The SPs completed an 11-item communication assessment and 3 milestone rating tools on professionalism (PROF-1, PROF-3) and interpersonal and communication skills (ICS-1). Ninety-six residents from a single institution participated toward the end of the intern year. Communication assessment scores ranged from 23% to 100% (mean [SD], 80.6 [17.0]). Mean (SD) milestone ratings across specialties were 2.80 ± 0.92 for PROF-1, 2.48 ± 0.92 for PROF-3, and 2.45 ± 0.92 for ICS-1. One-way analysis of variance revealed no significant differences among specialties on milestone or communication ratings. Residents who accepted personal responsibility for the error (84.55 [14.06]) received significantly higher communication ratings from SPs compared with residents who did not (66.67 [19.52], P < .001). This SP assessment of error disclosure by first-year residents from multiple specialties was feasible and acceptable. It revealed areas of improvement as well as considerable variation in communication skills and professionalism among residents.

Milestone Ratings and Supervisory Role Categorizations Swim Together, but Is the Water Muddy?

ObjectiveThis single-specialty, multi-institutional study aimed to determine 1) the association between milestone ratings for individual competencies and average milestone ratings (AMRs) and 2) the association between AMRs and recommended supervisory role categorizations made by individual clinical competency committee (CCC) members. MethodsDuring the 2015–16 academic year, CCC members at 14 pediatric residencies reported milestone ratings for 21 competencies and recommended supervisory role categories (may not supervise, may supervise in some settings, may supervise in all settings) for residents they reviewed. An exploratory factor analysis of competencies was conducted. The associations among individual competencies, the AMR, and supervisory role categorizations were determined by computing bivariate correlations. The relationship between AMRs and recommended supervisory role categorizations was examined using an ordinal mixed logistic regression model. ResultsOf the 155 CCC members, 68 completed both milestone assignments and supervision categorizations for 451 residents. Factor analysis of individual competencies controlling for clustering of residents in raters and sites resulted in a single-factor solution (cumulative variance: 0.75). All individual competencies had large positive correlations with the AMR (correlation coefficient: 0.84–0.93), except for two professionalism competencies (Prof1: 0.63 and Prof4: 0.65). When combined across training year and time points, the AMR and supervisory role categorization had a moderately positive correlation (0.56). ConclusionsThis exploratory study identified a modest correlation between average milestone ratings and supervisory role categorization. Convergence of competencies on a single factor deserves further exploration, with possible rater effects warranting attention.

Straight Line Scoring by Clinical Competency Committees Using Emergency Medicine Milestones.

In 2013, milestone ratings became a reporting requirement for emergency medicine (EM) residency programs. Programs rate each resident in the fall and spring on 23 milestone subcompetencies. This study examined the incidence of straight line scoring (SLS) for EM Milestone ratings, defined as a resident being assessed the same score across the milestone subcompetencies. This descriptive analysis measured the frequencies of SLS for all Accreditation Council for Graduate Medical Education (ACGME)-accredited EM programs during the 2015-2016 academic year. Outcomes were the frequency of SLS in the fall and spring milestone assessments, changes in the number of SLS reports, and reporting trends. Chi-square analysis compared nominal variables. There were 6257 residents in the fall and 6588 in the spring. Milestone scores were reported for 6173 EM residents in the fall (99% of 6257) and spring (94% of 6588). In the fall, 93% (5753 residents) did not receive SLS ratings and 420 (7%) did, with no significant difference compared with the spring (5776 [94%] versus 397 [6%]). Subgroup analysis showed higher SLS results for residents' first ratings (183 of 2136 versus 237 of 4220, P < .0001) and for their final ratings (200 of 2019 versus 197 of 4354, P < .0001). Twenty percent of programs submitted 10% or more SLS ratings, and a small percentage submitted more than 50% of ratings as SLS. Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.

Competency Assessment in Family Medicine Residency: Observations, Knowledge-Based Examinations, and Advancement.

The Family Medicine (FM) Milestones are competency-based assessments of residents in key dimensions relevant to practice in the specialty. Residency programs use the milestones in semiannual reviews of resident performance from the time of entry into the program to graduation. Using a national sample, we investigated the relationship of FM competency-based assessments to resident progress and the complementarity of milestones with knowledge-based assessments in FM residencies. We used midyear and end-of-year milestone ratings for all FM residents in Accreditation Council for Graduate Medical Education-accredited programs during academic years 2014-2015 and 2015-2016. The milestones contain 22 items across 6 competencies. We created a summative index across the milestones. The American Board of Family Medicine database provided resident demographics and in-training examination (ITE) scores. We linked information to the milestone data. The sample encompassed 6630 FM residents. The summative milestone index increased, on average, for each cohort (postgraduate year 1 [PGY-1] to PGY-2 and PGY-2 to PGY-3) at each assessment. The correlation between the milestone index that excluded the medical knowledge milestone and ITE scores was r = .195 (P < .001) for PGY-1 to PGY-2 cohort and r = .254 (P < .001) for PGY-2 to PGY-3 cohort. For both cohorts, ITE scores and composite milestone assessments were higher for residents who advanced than for those who did not. Competency-based assessment using the milestones for FM residents seems to be a viable multidimensional tool to assess the successful progression of residents.

How do clinical competency committees use different sources of data to assess residents’ performance on the internal medicine milestones?A mixed methods pilot study

Purpose: This study examines how Clinical Competency Committees (CCCs) synthesize assessment data to make judgments about residents’ clinical performances.Methods: Between 2014 and 2015, after four six-month reporting periods to the Accreditation Council for Graduate Medical Education (ACGME), 7 of 16 CCC faculty at Rush University Medical Center completed questionnaires focused on their perspectives about rating residents on their achievement of the milestones and participated in a focus group. Qualitative data were analyzed using grounded theory. Milestones ratings for two six-month ACGME reporting cycles (n = 100 categorical residents) were also analyzed.Results: CCC members weighted resident rotation ratings highest (weight = 37%), followed by faculty rotation comments (weight = 27%) and personal experience with residents (weight = 14%) for making judgments about learner’s milestone levels. Three assessment issues were identified from qualitative analyses: (1) “design issues” (e.g. problems with available data or lack thereof); (2) “synthesis issues” (e.g. factors influencing ratings and decision-making processes) and (3) “impact issues” (e.g. how CCC generated milestones ratings are used).Conclusions: Identifying factors that affect assessment at all stages of the CCC process can contribute to improving assessment systems, including support for faculty development for CCCs. Recognizing challenges in synthesizing first and second-hand assessment data is an important step in understanding the CCC decision-making process.

Are Surgical Milestone Assessments Predictive of In-Training Examination Scores?

With the recent utilization of Accreditation Council for Graduate Medical Education developmental milestones as part of resident evaluation, we sought to explore whether milestone-based ratings were associated with American Board of Surgery In-Training Examination (ABSITE) scores. Mid-year milestone ratings were obtained from the Accreditation Council for Graduate Medical Education Accreditation Database System for years 2014, 2015, and 2016 for all postgraduate years 1-5 general surgery residents in our program and paired with ABSITE scores (n = 69) from January of the following year. Linear regression was used to assess predictor importance of milestones on both ABSITE percentage correct scores and ABSITE percentile scores. Minimal to small positive correlations were observed between milestones and ABSITE percentile scores (r = 0.09-0.25), while moderately large correlations were observed between milestones and percentage correct scores (r = 0.65-0.76). The Medical Knowledge 1 (MK1) milestone significantly predicted ABSITE percentage correct scores, and explained 60% of the variance (adjusted R2 = 0.603). MK1 also significantly predicted ABSITE percentile scores, although weaker in magnitude, with MK1 explaining 20% of the variance (adjusted R2 = 0.197). Postgraduate year level and other milestones were not influential predictors of ABSITE scores. The mid-year MK1 milestone rating was predictive of ABSITE scores and may serve as a useful marker for Clinical Competency Committees to identify residents who could benefit from additional support to prepare for the ABSITE, although given the small exploratory nature of this study, additional research is still needed.

Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study.

The Accreditation Council for Graduate Medical Education (ACGME) requires programs to report learner progress using specialty-specific milestones. It is unclear how milestones can best identify critical deficiencies (CDs) in trainee performance. Specialties developed milestones independently of one another; not every specialty included CDs within milestones ratings. This study examined the proportion of ACGME milestone sets that include CD ratings, and describes one residency program's experiences using CD ratings in assessment. The authors reviewed ACGME milestones for all 99 specialties in November 2015, determining which rating scales contained CDs. The authors also reviewed three years of data (July 2012-June 2015) from the University of Cincinnati Medical Center (UCMC) internal medicine residency assessment system based on observable practice activities mapped to ACGME milestones. Data were analyzed by postgraduate year, assessor type, rotation, academic year, and core competency. The Mantel-Haenszel chi-square test was used to test for changes over time. Specialties demonstrated heterogeneity in accounting for CDs in ACGME milestones, with 22% (22/99) of specialties having no language describing CDs in milestones assessment. Thirty-three percent (63/189) of UCMC internal medicine residents received at least one CD rating, with CDs accounting for 0.18% (668/364,728) of all assessment ratings. The authors identified CDs across multiple core competencies and rotations. Despite some specialties not accounting for CDs in milestone assessment, UCMC's experience demonstrates that a significant proportion of residents may be rated as having a CD during training. Identification of CDs may allow programs to develop remediation and improvement plans.

Validity Evidence From Ratings of Pediatric Interns and Subinterns on a Subset of Pediatric Milestones.

To investigate evidence for validity of faculty members' pediatric milestone (PM) ratings of interns (first-year residents) and subinterns (fourth-year medical students) on nine subcompetencies related to readiness to serve as a pediatric intern in the inpatient setting. The Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN) and the National Board of Medical Examiners collaborated to investigate the utility of assessments of the PMs for trainees' performance. Data from 32 subinterns and 179 interns at 17 programs were collected from July 2012 through April 2013. Observers used several tools to assess learners. At each site, a faculty member used these data to make judgments about the learner's current developmental milestone in each subcompetency. Linear mixed models were fitted to milestone judgments to examine their relationship with learner's rank and subcompetency. On a 5-point developmental scale, mean milestone levels for interns ranged from 3.20 (for the subcompetency Work effectively as a member of a team) to 3.72 (Humanism) and for subinterns from 2.89 (Organize and prioritize care) to 3.61 (Professionalization). Mean milestone ratings were significantly higher for the Professionalism competency (3.59-3.72) for all trainees compared with Patient Care (2.89-3.24) and Personal and Professional Development (3.33-3.51). Mean intern ratings were significantly higher than mean subintern ratings for all nine subcompetencies except Professionalization, Humanism, and Trustworthiness. The PMs had a coherent internal structure and could distinguish between differing levels of trainees, which supports their validation for documenting developmental progression of pediatric trainees.

Competent for Unsupervised Practice: Use of Pediatric Residency Training Milestones to Assess Readiness.

To describe clinical skills progression during pediatric residency using the distribution of pediatric milestone assessments by subcompetency and year of training and to determine reasonable milestone expectations at time of graduation. Multi-institutional cohort study of the milestones reported to the Accreditation Council for Graduate Medical Education for all 21 pediatric subcompetencies. Most subcompetencies were measured using five milestone levels (1 = novice, 2 = advanced beginner, 3 = competent, 4 = proficient, 5 = master); 3 subcompetencies had only four levels defined. Milestone assessments for 2,030 pediatric residents in 47 programs during academic year 2013-2014 were obtained. There was significant variation in end-of-year milestone ratings for residents within each level of training, which decreased as training level increased. Most (78.9%; 434/550) graduating third-year pediatric residents received a milestone rating of ≥ 3 in all 21 subcompetencies; fewer (21.1%; 116/550) received a rating of ≥ 4 in all subcompetencies. Across all training levels, professionalism and interpersonal communication skills were rated highest; quality improvement was rated lowest. Trainees entered residency with a wide range of skills. As they advanced, skill variability within a training level decreased. Most graduating pediatric residents were still advancing on the milestone continuum toward proficiency and mastery, and an expectation of milestone ratings ≥ 4 in all categories upon graduation is unrealistic; milestone ratings ≥ 3 upon graduation may be more realistic. Understanding current pediatric residents' and graduates' skills can help to identify key areas that should be specifically targeted during training.

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed. To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. Cross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3). Program director ratings on the RAES and milestone ratings. Correlations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. Corresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, -0.9; 95% CI, -1.0 to -0.8; P < .001; milestone medical knowledge 1 difference, -0.3; 95% CI, -0.3 to -0.3; P < .001; and medical knowledge 2 difference, -0.2; 95% CI, -0.3 to -0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001). Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

Use of a Case-based Conference to Assess Achievement of Milestones in Noncognitive Skills

The advent of competency-based education and training by the Accreditation Council for Graduate Medical Education heralds the shift of assessment of plastic surgery resident programs from structure and process to one of outcomes.1 In parallel, the appraisal of these outcomes is required by the recent implementation of milestones for periodic determination of the progress of the journey toward independent practice. The Plastic Surgery Milestones Project has defined 14 individual milestones in each of 2 of the 6 competencies specific to plastic surgery, patient care, and knowledge. The challenge is the assessment in the arenas of cognitive knowledge, noncognitive (intuitive) problem-solving skills, and technical proficiency. Recent or objective methods to milestone operating room performance have been presented2 and traditionally cognitive knowledge has been evaluated with written examinations (InService), yet little attention has been directed toward an assessment of noncognitive skills linked to the milestones. The opportunity to do so is the case-based conference. The following is our programmatic execution of that opportunity. The weekly conference is coupled to a specific module (not to institutional case flow) within the 3-year curriculum and consists of 3 cases, each distributed 48 to 72 hours in advance. Discussion of each case is between an assigned resident and a single faculty member and discussion proceeds through analysis and formulation of a treatment plan. The case provides the basis for an objective assessment within a narrowly defined clinical area, accomplished with 6 elements (slide analysis, H & P, tests, analysis of pathophysiology and anatomy, treatment objectives, and treatment plan). A 3-point scale is utilized (poor analysis, incomplete but re-routes with prodding, and good-excellent analysis). Rating is performed by both the assigned faculty member and 1 additional nonparticipating faculty member and it also provides the basis for resident counseling and remediation, when necessary. Interrater validity to date has been excellent. Milestone ratings are maintained electronically in the residents’ portfolio and are completely transparent. The adoption of the milestone approach to assessment has been enthusiastically received by both faculty and resident staff.

The Internal Medicine Reporting Milestones: Cross-sectional Description of Initial Implementation in U.S. Residency Programs.

High-quality assessment of resident performance is needed to guide individual residents' development and ensure their preparedness to provide patient care. To facilitate this aim, reporting milestones are now required across all internal medicine (IM) residency programs. To describe initial milestone ratings for the population of IM residents by IM residency programs. Cross-sectional study. IM residency programs. All IM residents whose residency program directors submitted milestone data at the end of the 2013-2014 academic year. Ratings addressed 6 competencies and 22 subcompetencies. A rating of "not assessable" indicated insufficient information to evaluate the given subcompetency. Descriptive statistics were calculated to describe ratings across competencies and training years. Data were available for all 21774 U.S. IM residents from all 383 programs. Overall, 2889 residents (1621 in postgraduate year 1 [PGY-1], 902 in PGY-2, and 366 in PGY-3) had at least 1 subcompetency rated as not assessable. Summaries of average ratings by competency and training year showed higher ratings for PGY-3 residents in all competencies. Overall ratings for each of the 6 individual competencies showed that fewer than 1% of third-year residents were rated as "unsatisfactory" or "conditional on improvement." However, when subcompetency milestone ratings were used, 861 residents (12.8%) who successfully completed training had at least 1 competency with all corresponding subcompetencies graded below the threshold of "readiness for unsupervised practice." Data were derived from a point in time in the first reporting period in which milestones were used. The initial milestone-based evaluations of IM residents nationally suggest that documenting developmental progression of competency is possible over training years. Subcompetencies may identify areas in which residents might benefit from additional feedback and experience. Future work is needed to explore how milestones are used to support residents' development and enhance residency curricula. None.

Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications

ABSTRACTConstruct: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME). Background: Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS. Approach: Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013–June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined. Results: Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001). Conclusions: Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.