Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents

Yoon Soo Park,Kenji Yamazaki,Stanley J Hamstra,Eric Holmboe

doi:10.1001/jamanetworkopen.2021.37179

Abstract

Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice. To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents. This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019. Longitudinal formative assessment using the Milestones assessment system reported to the ACGME. To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories. This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth. These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Highlights

Decisions for promotion and readiness for unsupervised practice in graduate medical education require ongoing monitoring of learner performance using robust and longitudinal assessment systems data.[1,2] These aspirations to use ongoing assessment data form an integral aspect of competency-based medical education from a developmental perspective, to identify areas for early remediation and to facilitate further growth for all learners, meeting goals at critical levels for transition to subsequent stages of training.[3,4]The Accreditation Council for Graduate Medical Education (ACGME) implemented Milestones through the Accreditation System (NAS) initiative in July 2013
For the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents)
Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills

Summary

Introduction

Decisions for promotion and readiness for unsupervised practice in graduate medical education require ongoing monitoring of learner performance using robust and longitudinal assessment systems data.[1,2] These aspirations to use ongoing assessment data form an integral aspect of competency-based medical education from a developmental perspective, to identify areas for early remediation and to facilitate further growth for all learners, meeting goals at critical levels for transition to subsequent stages of training.[3,4]The Accreditation Council for Graduate Medical Education (ACGME) implemented Milestones through the Accreditation System (NAS) initiative in July 2013. Decisions for promotion and readiness for unsupervised practice in graduate medical education require ongoing monitoring of learner performance using robust and longitudinal assessment systems data.[1,2]. These aspirations to use ongoing assessment data form an integral aspect of competency-based medical education from a developmental perspective, to identify areas for early remediation and to facilitate further growth for all learners, meeting goals at critical levels for transition to subsequent stages of training.[3,4]. The Accreditation Council for Graduate Medical Education (ACGME) implemented Milestones through the Accreditation System (NAS) initiative in July 2013. Every 6 months, the residency program’s Clinical Competency Committee (CCC) synthesizes assessment data into Milestone levels, which the program subsequently reports to the ACGME.[6-8]. Validity evidence supporting the use and interpretation of Milestones data warrants further investigation, from a longitudinal and developmental viewpoint of assessment data.[9-14]

Methods

Results

Discussion

Conclusion