423 Trends in Utilization of Standardized Letters of Recommendation in the 2021-2022 Neurosurgery Application Process

Abstract

INTRODUCTION: Standardized LORs (SLORs) were introduced during the 2020-2021 neurosurgery residency application cycle, but suffered from poor inter-rater reliability and significant grade inflation. Changes were made to the SLOR template and utilization patterns in response to these shortcomings. METHODS: This cross-sectional study analyzed all SLORs submitted to a single neurosurgery residency program over the 2021-2022 cycle. Data from seven competency domains and the overall rating were recorded and stratified by academic category of letter writer. Inter-rater reliability was evaluated using Krippendorff’s Alpha. Frequency of letters written was evaluated using Kruskal-Wallis H Test. RESULTS: 90% of SLORs rated applicants among the Top 25%, but there was a significant decrease in the usage of the “Top 1%” and “Top 2-5%” ratings from the prior application cycle. Inter-rater reliability was poor across all competencies. Writers who completed one SLOR rated applicants higher and had a narrower range than those who completed multiple SLORs. CONCLUSIONS: Changes in the format and subsequent utilization patterns of SLORs have slightly decreased grade inflation; however, inter-rater reliability remains poor. The most wide-ranging evaluators submitted the highest number of SLORs, suggesting that future evaluation and usage of SLORs should emphasize letter-writer characteristics and numbers of SLORs written. Overall, SLORs have been well and broadly accepted with subtle improvements in the second year of utilization.