Standardized Letter Of Recommendation Research Articles

How Prominent Is Score Inflation, Gender, and Racial Bias in Orthopaedic Surgery Residency Recommendation Letters? A Systematic Review

Introduction Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant’s orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation. Question/purposes In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation? Methods PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race. Results Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race. Conclusion Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers’ abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident. Clinical Relevance We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant’s chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

Program Directors' Perspectives on Residency Applications in the Post‒United States Medical Licensing Examination Step 1 Era: A Case for Standardized Letters of Recommendation?

The United States Medical Licensing Examination (USMLE) Step 1 test evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements toward pass/fail systems for clerkship grading and trends toward abandonment of class ranking, assessing residency applications has become increasingly challenging. The Society of Academic Urologists convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology PDs in the US. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PDs. There was a 60.5% response rate (89/147). Letters of recommendation (LORs) were ranked as the most important predictor, with a mean rank of 2.39, median of 2 (IQR 1-3). Clerkship grades and USMLE Step 1 were comparable and ranked second. Medical school reputation ranked the lowest. There was significant subjective heterogeneity among categories; however, this was less so for LORs, which predominated as the most important factor among application elements (P < .001). To our knowledge, this is the largest sample size assessing PD perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving toward binary pass/fail systems create an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LORs to be the most important factor of residency applications, making a compelling argument for moving toward a standardized LOR to maximize this tool, mitigate bias, and improve interreviewer reliability.

Components of effective letters of recommendation: A cross-sectional survey of academic faculty.

Conventional merit-based criteria, including standardized test scores and grade point averages, have become less available to residency programs to help distinguish applicants, making other components of the application, including letters of recommendation (LORs), important surrogate markers for performance. Despite their impact on applications, there is limited published data on LORs in the international setting. A cross-sectional survey of academic faculty was conducted between 9 January 2023 and 12 March 2023 at two large academic medical centers in the United Arab Emirates. Descriptive statistics were used to tabulate variable frequencies. Of the 98 respondents, the majority were male (n = 67; 68.4%), Western-trained (n = 66; 67.3%), mid-career physicians (n = 46; 46.9%). Most respondents (n = 77; 78.6%) believed that the purpose of an LOR was to help an applicant match into their desired program. Letters rarely included important skills, such as leadership (n = 37; 37.8%), applicant involvement in research (n = 43; 43.9%), education (n = 38; 38.8%), or patient advocacy (n = 30; 30.6%). Most faculty (n = 81; 82.7%) were not familiar with standardized letters of recommendation. Only 7.3% (n = 7) of respondents previously received training in writing LORs, but 87.7% (n = 86) expressed an interest in this professional development opportunity. There is variability in perceptions and practices related to LOR writing in our international setting, with several areas for improvement. Given the increasing importance of LORs to a candidate's application, faculty development is necessary.

Milestones for neurosurgery sub-interns: a novel evaluation tool to quantitatively differentiate residency applicants.

The study objective was to create a novel milestones evaluation form for neurosurgery sub-interns and assess its potential as a quantitative and standardized performance assessment to compare potential residency applicants. In this pilot study, the authors aimed to determine the form's interrater reliability, relationship to percentile assignments in the neurosurgery standardized letter of recommendation (SLOR), ability to quantitatively differentiate tiers of students, and ease of use. Medical student milestones were either adapted from the resident Neurological Surgery Milestones or created de novo to evaluate a student's medical knowledge, procedural aptitude, professionalism, interpersonal and communication skills, and evidence-based practice and improvement. Four milestone levels were defined, corresponding to estimated 3rd-year medical student through 2nd-year resident levels. Faculty and resident evaluations as well as student self-evaluations were completed for 35 sub-interns across 8 programs. A cumulative milestone score (CMS) was computed for each student. Student CMSs were compared both within and between programs. Interrater reliability was determined with Kendall's coefficient of concordance (Kendall's W). Student CMSs were compared against their percentile assignments in the SLOR using analysis of variance with post hoc testing. CMS-derived percentile rankings were assigned to quantitatively distinguish tiers of students. Students and faculty were surveyed on the form's usefulness. The average faculty rating overall was 3.20, similar to the estimated competency level of an intern. Student and faculty ratings were similar, whereas resident ratings were lower (p < 0.001). Students were rated most highly in coachability and feedback (3.49 and 3.67, respectively) and lowest in bedside procedural aptitude (2.90 and 2.85, respectively) in both faculty and self-evaluations. The median CMS was 26.5 (IQR 21.75-29.75, range 14-32) with only 2 students (5.7%) achieving the highest rating of 32. Programs that evaluated the most students differentiated the highest-performing students from the lowest by at least 13 points. A program with 3 faculty raters demonstrated scoring agreement across 5 students (p = 0.024). The CMS differed significantly between SLOR percentile assignments, despite 25% of students being assigned to the top fifth percentile. CMS-driven percentile assignment significantly differentiated the bottom, middle, and top third of students (p < 0.001). Faculty and students strongly endorsed the milestones form. The medical student milestones form was well received and differentiated neurosurgery sub-interns both within and across programs. This form has potential as a replacement for numerical Step 1 scoring as a standardized, quantitative performance assessment for neurosurgery residency applicants.

Recent Trends and Future Directions for the Integrated Plastic Surgery Match.

The integrated plastic surgery residency match has risen to be the most competitive specialty in the 2022 match. This reality has prompted medical students to reach a high level of personal achievements, including pursuing research fellowships to boost research productivity. The competitive nature of this specialty has highlighted several barriers for applicants, such as those from groups underrepresented in surgery, of lower socioeconomic backgrounds, or without a home program. In recent years, there have been several changes to the match that stand to attenuate disparities among applicants, such as the transition to virtual interviews and the shift of the United States Medical Licensing Examination Step 1 score to pass-fail. The introduction of the Plastic Surgery Common Application and standardized letters of recommendation has altered the application process for the plastic surgery match. Given these recent trends, evaluating the current landscape and looking toward future directions for the integrated plastic surgery match becomes necessary. Understanding these changes will not only benefit medical students by giving them a transparent look into the match process but also provide a framework for other specialties to follow to increase accessibility to their specific specialty.

The Utility of the Plastic Surgery Standardized Letter of Recommendation Form in Predicting Residency Match Outcomes

Letters of recommendation play an important role in resident selection. While plastic surgery's Standardized Letter of Recommendation (SLOR) form most commonly serves as an adjunct to narrative letters, the SLOR provides objective data in the review process and could eventually replace narrative letters. The utility of the SLOR in predicting Match outcomes has not been studied. Applicant data from 225 first-time residency applicants in 2020-21 were collected. Logistic regression modeling was used to predict Match outcomes. This model was validated using 100 randomly selected applicants from 2019-20. Rank placement (SLOR Question 6) was the most important factor when predicting Match outcomes (p<0.0001). All other SLOR questions were not predictive and subject to notable score inflation. No SLOR score differences were noted based on race; female applicants were rated higher in two of ten domains (p<0.05). One question on the plastic surgery SLOR was highly predictive of an applicant matching. However, the remaining SLOR questions had little utility and were subject to gross score inflation. Further work should be done to optimize the utility of the SLOR in differentiating applicants. This has important implications in ensuring the selection of professional, competent residents according to the aims of the Accreditation Council of Graduate Medical Education.

23. The Utility of the Standardized Letter of Recommendation Form in Assessing Integrated Plastic Surgery Residency Applicants

PURPOSE: Identifying the right applicants to train as plastic surgeons is critical for both programs and our specialty. The ACAPS Standardized Letter of Recommendation (SLOR) form serves as an adjunct to narrative letters and is meant to provide objective, comparable data across applicants. The SLOR assesses applicants in 10 domains and asks the evaluator where they would place the applicant on their rank list (Question 6). The utility of the SLOR in predicting Match outcomes has not been studied. METHODS: From the 2020-21 cycle, 100 first-time applicants from US medical schools were randomly selected. Application characteristics were collected from applications submitted to one integrated program. Match outcomes were determined from publicly available data. Logistic regression modeling identified variables that were significant in predicting Match outcomes. RESULTS: The Match rate for this cohort was 69%, consistent with the specialty norm. None of the 10 SLOR domains contributed significantly (p>0.05) to predicting Match outcomes. There were no differences in SLOR scores based on gender or race. However, female gender (p=0.018), older age (p=0.025), higher Step 1 score (p=0.0005), more publications (p=0.0003), and Question 6 (p<0.0001) had a significant role in predicting Match outcomes. These variables alone successfully predicted matching in 93% of applicants using the logistic regression model. CONCLUSIONS: Based on these results, it is time to reconsider the questions and structure of the SLOR form to minimize data overload on evaluators. Further work should be done to identify questions that better stratify applicants in meaningful, unbiased ways.

749 Medical Student Milestones for Neurosurgery Sub-Interns: A Pilot Study of a Novel Competency-Based Evaluation Tool to Quantitatively Differentiate Residency Applicants

INTRODUCTION: Resident selection is guided by few objective, quantitative datapoints. Utility of a standardized letter of recommendation (SLOR) has been limited by “grade inflation”. The Milestones program in neurosurgery residency standardizes “competence” definitions, recognizes domains of competence beyond patient care and medical knowledge, and highlights deficient areas. A similar approach for neurosurgery sub-interns could provide objective performance data to aid applicant evaluation. METHODS: Medical student milestones were adapted from the residency milestones or newly created according to group consensus of relevant competencies. Competency levels were graded ordinally 1 (level of 3rd-year medical student) to 4 (2nd-year resident). Faculty and self-evaluations were completed for 35 sub-interns across eight programs. Evaluators were surveyed on its usefulness. Average and cumulative milestone scores (CMS) within and across programs were evaluated using descriptive statistics. Kendall’s W tested inter-rater agreement. CMS were compared against percentile assignments on LOR using analysis of variance with post-hoc testing. RESULTS: The average faculty rating overall was 3.20, similar to that expected of an intern. Students scored highest in coachability and feedback (3.49; 3.67) and lowest in bedside procedural aptitude (2.90; 2.85) in faculty and self-evaluations. Median CMS was 26.5 (IQR 21.75-29.75; range 14-32). CMS in four programs evaluating at least five students ranged at least 11 points. A program with three faculty raters demonstrated scoring agreement across five students (p = 0.024). CMS differed significantly between SLOR percentile assignments, despite 25% of students assigned to top 5%ile while CMS-driven assignment of percentiles significantly differentiated quartiles of students. Faculty and students strongly endorsed the milestones form. CONCLUSIONS: The medical student milestones form was well-received and considered useful in objectively differentiating students within and across programs.

MP25-05 PROGRAM DIRECTORS' PERSPECTIVES ON THE ASSESSMENT OF RESIDENCY APPLICANTS IN THE POST-USMLE STEP 1 ERA: A STRONG CASE FOR STANDARDIZED LETTERS OF RECOMMENDATION?

You have accessJournal of UrologyCME1 Apr 2023MP25-05 PROGRAM DIRECTORS' PERSPECTIVES ON THE ASSESSMENT OF RESIDENCY APPLICANTS IN THE POST-USMLE STEP 1 ERA: A STRONG CASE FOR STANDARDIZED LETTERS OF RECOMMENDATION? Angeline Johny, Patrick J. Shenot, Charles Green, Leah Chisholm, Steve Riggs, Stephen V. Jackman, Aqsa A. Khan, Peter N. Kolettis, Brian K. McNeil, and Wesley A. Mayer Angeline JohnyAngeline Johny More articles by this author , Patrick J. ShenotPatrick J. Shenot More articles by this author , Charles GreenCharles Green More articles by this author , Leah ChisholmLeah Chisholm More articles by this author , Steve RiggsSteve Riggs More articles by this author , Stephen V. JackmanStephen V. Jackman More articles by this author , Aqsa A. KhanAqsa A. Khan More articles by this author , Peter N. KolettisPeter N. Kolettis More articles by this author , Brian K. McNeilBrian K. McNeil More articles by this author , and Wesley A. MayerWesley A. Mayer More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000003253.05AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: The USMLE Step 1 exam evolved into a key metric utilized by program directors (PDs) in assessing candidates for residency. The transition to a USMLE Step 1 binary pass/fail scoring system has resulted in a loss of an important objective assessment. With national movements towards pass/fail systems for clerkship grading and trends towards abandonment of class ranking, assessing residency applications is becoming increasingly challenging. METHODS: The Society for Academic Urologists (SAU) convened a task force to, in part, assess the perspectives of urology PDs regarding the importance of various aspects of a residency application for predicting clinical performance. An anonymous survey was disseminated to all urology program directors in the United States. Perspectives on 11 potential application predictors of clinical performance and demographics were recorded. Descriptive statistics characterized PD responses. Friedman’s Test and pairwise Wilcoxon tests were used to evaluate the relative ranks assigned to application elements by PD’s. RESULTS: There was a 60.5% response rate (89/147). Letters of recommendation (LOR) was ranked as the most important predictor, with a mean of 2.39, median of 2 (IQR range 1- 3). Clerkship grades and USMLE Step 1 were comparable and ranked second with mean scores of 4.3, and 4.62 respectively. Medical school reputation ranked the lowest with a mean score of 8.47. There was significant heterogeneity among categories, however this was less so for LORs which predominated as the most important factor among application elements (p<0.001). CONCLUSIONS: To our knowledge, this is the largest sample size assessing program director perspectives on application factors that predict clinical performance. The second (clerkship grades) and third (USLME Step 1) most important factors moving to binary pass/fail systems creates an opportunity for actionable change to improve assessment objectivity. Our data demonstrate LOR to be the most important factor of residency applications. Thus, there may be a strong argument for moving towards a standardized LOR to maximize this tool, mitigate bias, and improve inter-reviewer reliability. Source of Funding: N/A © 2023 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 209Issue Supplement 4April 2023Page: e344 Advertisement Copyright & Permissions© 2023 by American Urological Association Education and Research, Inc.MetricsAuthor Information Angeline Johny More articles by this author Patrick J. Shenot More articles by this author Charles Green More articles by this author Leah Chisholm More articles by this author Steve Riggs More articles by this author Stephen V. Jackman More articles by this author Aqsa A. Khan More articles by this author Peter N. Kolettis More articles by this author Brian K. McNeil More articles by this author Wesley A. Mayer More articles by this author Expand All Advertisement PDF downloadLoading ...

Words matter: using natural language processing to predict neurosurgical residency match outcomes.

Narrative letters of recommendation (NLORs) are considered by neurosurgical program directors to be among the most important parts of the residency application. However, the utility of these NLORs in predicting match outcomes compared to objective measures has not been determined. In this study, the authors compare the performance of machine learning models trained on applicant NLORs and demographic data to predict match outcomes and investigate whether narrative language is predictive of standardized letter of recommendation (SLOR) rankings. This study analyzed 1498 NLORs from 391 applications submitted to a single neurosurgery residency program over the 2020-2021 cycle. Applicant demographics and match outcomes were extracted from Electronic Residency Application Service applications and training program websites. Logistic regression models using least absolute shrinkage and selection operator were trained to predict match outcomes using applicant NLOR text and demographics. Another model was trained on NLOR text to predict SLOR rankings. Model performance was estimated using area under the curve (AUC). Both the NLOR and demographics models were able to discriminate similarly between match outcomes (AUCs 0.75 and 0.80; p = 0.13). Words including "outstanding," "seamlessly," and "AOA" (Alpha Omega Alpha) were predictive of match success. This model was able to predict SLORs ranked in the top 5%. Words including "highest," "outstanding," and "best" were predictive of the top 5% SLORs. NLORs and demographic data similarly discriminate whether applicants will or will not match into a neurosurgical residency program. However, NLORs potentially provide further insight regarding applicant fit. Because words used in NLORs are predictive of both match outcomes and SLOR rankings, continuing to include narrative evaluations may be invaluable to the match process.

Use of standardized letters of recommendation for orthopaedic surgery residency: a national survey study

Background: Standardized letters of recommendation (SLORs) were introduced to facilitate the comparison of applicants for orthopaedic surgery residency positions, but concerns have arisen regarding the prevalence of their use and potential limitations. Methods: An 11-question electronic survey was sent to all letter of recommendation (LOR) authors and program coordinators who were identified as having completed or prepared a SLOR during the 2020 orthopaedic surgery residency match cycle. A total of 740 LOR authors and 218 program coordinators were invited via initial and reminder electronic mail messages. Results: The survey response rate was 18.1% for LOR authors and 25.2% for program coordinators. The proportion of LORs written that were SLORs significantly increased from prior to the 2020 match cycle to the 2020 match cycle (72.7% to 90.2%, ratio =1.240, P&lt;0.001). There was not a significant increase in the proportion of LORs that were SLORs prepared by program coordinators (83.7% to 77.6%, ratio =0.927, P=0.375). A majority of LOR authors and program coordinators were aware of electronic (82.1% and 76.5%, respectively) and paper SLORs (91.0% and 88.2%, respectively). Conclusions: This study found that SLOR usage increased among LOR authors. However, a parallel increase in SLOR preparation was not reported by program coordinators. Level of Evidence: Level III

A Call to Action for Standardizing Letters of Recommendation.

A Call to Action for Standardizing Letters of Recommendation.

Standardized Letter of Recommendation: Can Everyone Be Awesome?

Standardized letters of recommendation (SLOR) are hypothesized to decrease bias and provide consistent domains for evaluation. However, their ability to differentiate among applicants is unknown. The utilization and functionality of SLOR and the impact of SLOR domain rating on matching for colon and rectal surgery (CRS) residency applicants have yet to be assessed. Descriptive statistics and bivariate analysis were employed. Applicants were categorized into 3 groups; Top-tier(TT): applicants rated 100% Excellent/Very Good; Mid-Tier(MT) applicants rated 80-99% Excellent/Very Good; and Non-Top Tier(NTT) applicants rated <80% Excellent/Very Good. University of Hospitals Cleveland Medical Center. SLORs submitted to a single colorectal surgery residency in 2019 were analyzed RESULTS: A total of 101 applicants were included, 54 (53.5%) of the applicants were male. 75 (74.2%) applicants who applied to our residency matched into a CRS residency, compared to the national rate of 66%. Of the 101 applicants with SLOR, 54 (53.5%) were categorized as TT, 26 (25.7%) as MT, and 21(20.8%) as NTT. The univariable analysis demonstrated a statistically significant difference in research experience (p=0.029) and match status (p=0.01) between applicant tiers. There were no statistically significant differences between applicant-tier and demographics, foreign medical graduates (FMG), H-indices, ABSITE scores, type of residency, preliminary year, completing an unaccredited CRS, and applicants with an additional degree. On multivariable analysis age (OR=0.65; CI=0.48-0.87) and FMG applicants (OR=0.05; CI=0.01-0.44) were inversely associated with successfully matching. Compared to TT applicants, MT (OR=0.07; CI=0.01-0.57) and NTT (OR=0.04; CI=0.01-0.34) applicants were inversely associated with a successful match. Individuals who completed research prior to residency but after medical school was associated with successfully matching (p=0.009). The presence of MT and NTT ratings is associated with failure to match and may represent an area of concern for CRS programs rather than a tool to discern differences between candidates.

Development and Assessment of Early Utilization of the Standardized Letter of Recommendation for Use in the Radiology Residency Match

Letters of recommendation are essential for residency applications. Traditionally, narrative letters have been used. Standardized letters of recommendation (SLOR) have been developed and have benefits to the writer and reader. The goal was to develop an informative, meaningful, and efficient SLOR for the radiology residency Match and to assess its early use. An Association of University Radiologists (AUR) Research and Education Venture Fund Grant was awarded for the development of a SLOR for use in the radiology residency Match. Grant recipients developed the letter and modifications were based on feedback from the Alliance of Medical Student Educators in Radiology (AMSER) SLOR task force and a larger task force including additional AUR and Association of Program Directors in Radiology (APDR) members. AUR and APDR members were surveyed in 2020 to assess the radiology SLOR. The radiology SLOR became available for use in September 2018. It highlights the top six traits or abilities deemed most valued in a resident candidate and guides the writer to develop a concrete narrative. Top perceived benefits of the Radiology SLOR are ease of reading and interpretation. Top perceived drawbacks are an impersonal format and concerns regarding program directors' perceptions of the new SLOR. SLOR utilization increased in the second year of availability. The radiology SLOR was developed and first used in 2018. Its use increased over two years and will likely continue to increase given the perceived benefits and increasing awareness of the SLOR.

Reshaping Residency Recruitment: Achieving Alignment Between Applicants and Programs in Surgery

The residency recruitment process has become increasingly challenging for both applicants and program directors, in part, due to the inflation in the number of applications per student. As a result, it has become more daunting for programs to design processes that evaluate applicants holistically. Furthermore, the existing methods used to evaluate and select applicants do not necessarily predict success in residency and may inadvertently lend to gender, racial, and ethnic bias. This narrative review aims to identify innovative tools used in residency recruitment that will allow programs and applicants to better determine concordance of interests and achieve value alignment while supporting improved, objective evaluation of an applicant's unique attributes and experiences. PubMed was used to conduct a narrative review of recruitment strategies in admission processes of undergraduate and graduate medical education between 1975 and June 2021, using the designated Medical Subject Heading (MeSH0 terms. Inclusion criteria were established surrounding innovative tools to better objectively screen, evaluate, or select applicants. Strategies relying primarily on traditional metrics (United States Medical Licensing Examination (USMLE) scores, Alpha Omega Alpha status, and clerkship grades) were excluded. Forty-two articles met specific inclusion criteria. Using these articles, a framework was created with two specific aims: (1) to allow applicants and programs to express or assess interest and (2) to foster objective review of unique applicant attributes, skills, experiences, and competencies that align with program mission and values. The following five innovative tools for recruitment were identified: preference signaling, secondary applications, standardized letters of recommendation, situational judgment testing, and surgical simulation. As the number of applications continues to rise, strategies must be implemented to allow applicants and institutions to achieve better alignment or "fit," while also giving balanced consideration to all of an applicant's unique characteristics. A more holistic approach to applicant selection is a necessary tool in order to increase diversity and inclusion within the field of surgery.

Standardized Letters of Recommendation and Success in the Urology Match

Distinguishing between top residency applicants is challenging, but crucially important to ensure the continued success of individual programs and the field as a whole. Letters of recommendation (LOR) have long served as an important tool for selection of residents. In a survey by Weissbart et al, it was among the most important factors for selection of urology residents.1 In order to overcome the shortcomings of narrative LORs (such as subjectiveness, ambiguity, low inter-rater reliability, gender bias, etc.), standardized LORs (SLORs) were introduced in several specialties.

Characterizing Standardized Letters of Recommendation in Urology Residency Applications

ObjectiveTo identify the current formats of standardized letters of recommendation (SLORs) and evaluate their characteristics, the distribution of applicants’ ratings, correlation between SLOR domain ratings and conventional application metrics, and potential biases. MethodsWe evaluated all applications submitted to our residency program for the 2020-2021 urology match. Two main formats of SLOR were identified. We extracted application characteristics and SLOR domain ratings. ResultsNinety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 Clinical Knowledge scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in terms of domain ratings. Alpha Omega Alpha members received higher ratings in “urology resident potential,” “academic urologist potential,” and “performance as a sub-intern” domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs virtual interactions received similar ratings except for “communication”. ConclusionWhile it is promising to observe such number of SLORs submitted for the first time in urology, the current formats could benefit from further refinement in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical.

MP12-08 STANDARDIZED LETTERS OF RECOMMENDATION IN UROLOGY RESIDENCY APPLICATIONS

You have accessJournal of UrologyEducation Research II (MP12)1 Sep 2021MP12-08 STANDARDIZED LETTERS OF RECOMMENDATION IN UROLOGY RESIDENCY APPLICATIONS Behnam Nabavizadeh, Nizar Hakam, Michael Sadighian, Jordan Holler, Gregory Amend, Lindsay Hampson, and Benjamin Breyer Behnam NabavizadehBehnam Nabavizadeh More articles by this author , Nizar HakamNizar Hakam More articles by this author , Michael SadighianMichael Sadighian More articles by this author , Jordan HollerJordan Holler More articles by this author , Gregory AmendGregory Amend More articles by this author , Lindsay HampsonLindsay Hampson More articles by this author , and Benjamin BreyerBenjamin Breyer More articles by this author View All Author Informationhttps://doi.org/10.1097/JU.0000000000001985.08AboutPDF ToolsAdd to favoritesDownload CitationsTrack CitationsPermissionsReprints ShareFacebookLinked InTwitterEmail Abstract INTRODUCTION AND OBJECTIVE: Standardized letters of recommendation (SLORs) have been introduced for residency applications in several specialties to overcome the obstacles in traditional narrative letters of recommendations (e.g., ambiguity, gender bias, low interobserver reliability, etc.). Although the American Urological Association has not officially endorsed a uniform SLOR for candidates seeking urology residency, we observed a substantial increase in the usage of SLORs submitted to our residency program in the 2020–2021 match cycle. In this study, we aimed to identify the formats of SLOR used in urology, characterize their domain ratings, evaluate the correlation between SLOR domain ratings with several components of applicants’ background, and the presence of biases in the current formats of SLOR. METHODS: We evaluated all applications submitted to our residency program for the 2020–2021 urology match. Two main formats of SLOR were identified (Figure 1). We extracted application characteristics and SLOR domain ratings. RESULTS: Ninety SLORs from 82 applicants were reviewed. Applicants were highly rated among top tiers in both formats. Some correlations were observed between domain ratings and application metrics such as Step 1 and Step 2 CK scores, and percentage of Honors in core clinical clerkships. No statistically significant differences were found between female and male applicants in domain ratings. Alpha Omega Alpha members received higher ratings in “urology resident potential”, “academic urologist potential”, and “performance as a sub-intern” domains. Applicants from top 40 US medical schools performed better as sub-interns, and were more likely to be ranked higher. Letters from home institutions were associated with higher ratings in several domains. In-person vs. virtual interactions received similar ratings except for “communication”. CONCLUSIONS: While it is promising to observe such number of SLORs submitted for the first time in urology, the current formats need further revisions in their structures and domains to distinguish between highly qualified urology applicants more efficiently. Given the transition in Step 1 score reporting to pass/fail outcome, the need for a reliable urology-specific SLOR will be critical for the field to select the most qualified applicants to enter this competitive surgical subspecialty. Source of Funding: None © 2021 by American Urological Association Education and Research, Inc.FiguresReferencesRelatedDetails Volume 206Issue Supplement 3September 2021Page: e191-e191 Advertisement Copyright & Permissions© 2021 by American Urological Association Education and Research, Inc.MetricsAuthor Information Behnam Nabavizadeh More articles by this author Nizar Hakam More articles by this author Michael Sadighian More articles by this author Jordan Holler More articles by this author Gregory Amend More articles by this author Lindsay Hampson More articles by this author Benjamin Breyer More articles by this author Expand All Advertisement Loading ...

The Orthopaedic Resident Selection Process: Proposed Reforms and Lessons From Other Specialties.

Proposals for substantive reforms to the orthopaedic resident selection process are growing, given increasing applicant competitiveness, burgeoning inefficiencies and inequities of the current system, and impending transition of Step 1 to pass/fail. The COVID-19 pandemic has further catalyzed the need for reforms, offering unprecedented opportunities to pilot novel changes. However, a comprehensive collation of all proposed and implemented orthopaedic reforms is currently lacking. Thus, we aimed to characterize proposed orthopaedic-specific resident selection reforms in the context of reforms implemented by other specialties. EMBASE, MEDLINE, Scopus, and Web of Science databases were searched for references proposing reforms to the orthopaedic resident selection process published from 2005 to 2020. An inductive approach to qualitative content analysis was used to categorize reforms. Twenty-six articles proposing 13 unique reforms to the orthopaedic resident selection process were identified. The most commonly proposed reforms included noncognitive assessments (n = 8), application caps (n = 7), standardized letters of recommendation (n = 5), program-specific supplemental applications (n = 5), creation of a centralized database of standardized program information (n = 4), use of a standardized applicant composite score (n = 4), and a moratorium on postinterview communication (n = 4). Importantly, nearly all of these reforms have also been proposed or implemented by other specialties. Numerous reforms to the orthopaedic resident selection process have been suggested over the past 15 years, several of which have been implemented on a program-specific basis, including noncognitive assessments, supplemental applications, and standardized letters of recommendation. Careful examination of applicant and program experiences and Match outcomes after these reforms is imperative to inform future directions.

CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.

CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.