749 Medical Student Milestones for Neurosurgery Sub-Interns: A Pilot Study of a Novel Competency-Based Evaluation Tool to Quantitatively Differentiate Residency Applicants

Abstract

INTRODUCTION: Resident selection is guided by few objective, quantitative datapoints. Utility of a standardized letter of recommendation (SLOR) has been limited by “grade inflation”. The Milestones program in neurosurgery residency standardizes “competence” definitions, recognizes domains of competence beyond patient care and medical knowledge, and highlights deficient areas. A similar approach for neurosurgery sub-interns could provide objective performance data to aid applicant evaluation. METHODS: Medical student milestones were adapted from the residency milestones or newly created according to group consensus of relevant competencies. Competency levels were graded ordinally 1 (level of 3rd-year medical student) to 4 (2nd-year resident). Faculty and self-evaluations were completed for 35 sub-interns across eight programs. Evaluators were surveyed on its usefulness. Average and cumulative milestone scores (CMS) within and across programs were evaluated using descriptive statistics. Kendall’s W tested inter-rater agreement. CMS were compared against percentile assignments on LOR using analysis of variance with post-hoc testing. RESULTS: The average faculty rating overall was 3.20, similar to that expected of an intern. Students scored highest in coachability and feedback (3.49; 3.67) and lowest in bedside procedural aptitude (2.90; 2.85) in faculty and self-evaluations. Median CMS was 26.5 (IQR 21.75-29.75; range 14-32). CMS in four programs evaluating at least five students ranged at least 11 points. A program with three faculty raters demonstrated scoring agreement across five students (p = 0.024). CMS differed significantly between SLOR percentile assignments, despite 25% of students assigned to top 5%ile while CMS-driven assignment of percentiles significantly differentiated quartiles of students. Faculty and students strongly endorsed the milestones form. CONCLUSIONS: The medical student milestones form was well-received and considered useful in objectively differentiating students within and across programs.