Enhancing Transparency in Urology Residency Match Process: The Importance of Standardized Letters of Recommendation.

The Council of Emergency Medicine Residency Directors (CORD) introduced the standardized letter of recommendation (SLOR) in 1997, and it has become a critical tool for assessing candidates for emergency medicine (EM) training. It has not itself been evaluated since the initial studies associated with its introduction. This study characterizes current SLOR use to evaluate whether it serves its intended purpose of being standardized, concise, and discriminating. This retrospective, multi-institutional study evaluated letters of recommendation from U.S. allopathic applicants to three EM training programs during the 2011-2012 Electronic Residency Application Service (ERAS) application cycle. Distributions of responses to each question on the SLOR were calculated, and the free-text responses were analyzed. Two pilots, performed on five applicants each, assisted in developing a strategy for limiting interrater reliability. Each of the three geographically diverse programs provided a complete list of U.S. allopathic applicants to their program. Upon randomization, each program received a list of coded applicants unique to their program randomly selected for data collection. The number of applicants was selected to reach a goal of approximately 200 SLORs per site (n = 602). Among this group, comprising 278 of 1,498 applicants (18.6%) from U.S. allopathic schools, a total of 1,037 letters of recommendation were written, with 724 (69.8%) written by emergency physicians. SLORs represented 57.9% (602/1037) of all LORs (by any kind of author) and 83.1% (602/724) of letters written by emergency physicians. Three hundred ninety-two of 602 SLORs had a single author (65.1%). For the question on "global assessment," students were scored in the top 10% in 234 of 583 of applications (40.1%; question not answered by some), and 485 of 583 (83.2%) of the applicants were ranked above the level of their peers. Similarly, >95% of all applicants were ranked in the top third compared to peers, for all but one section under "qualifications for emergency medicine." For 405 of 602 of all SLORs (67.2%), one or more questions were left unanswered, while 76 of all SLORs (12.6%) were "customized" or changed from the standard template. Finally, in 291 of 599 of SLORs (48.6%), the word count was greater than the recommended maximum of 200 words. Grade inflation is marked throughout the SLOR, limiting its ability to be discriminating. Furthermore, template customization and skipped questions work against the intention to standardize the SLOR. Finally, it is not uncommon for comments to be longer than guideline recommendations. As an assessment tool, the SLOR could be more discerning, concise, and standardized to serve its intended purpose.

Letters of recommendation are considered an important factor in the holistic ranking of orthopaedic residency applications. The standardized letter of recommendation (SLOR) was introduced in 2017 in response to the inherent bias and limited comparative value of traditional LORs. It includes standardized questions about an applicant's orthopaedic qualifications, designed to enable fair comparisons among candidates. However, disparate and inconsistent findings have made it difficult to draw meaningful conclusions from individual studies on the SLOR and narrative letter of recommendation. In this systematic review we asked: (1) What is the distribution of applicant SLOR rating among nine domains and summative scores? (2) Are there applicant characteristics associated with SLOR ratings? (3) Is there gender and racial bias in the SLOR and narrative letter of recommendation? PubMed, EBSCO, and Google Scholar electronic databases were queried on March 20, 2023, to identify all studies evaluating SLOR and narrative letter of recommendations between January 1, 2017, and March 20, 2023. Articles that evaluated orthopaedic SLOR or narrative LORs were included. Systematic reviews, case reports, duplicate studies between databases, and grey literature such as abstracts and articles on preprint servers were excluded. Eight studies including 6179 applicants and 18,987 letters of recommendation were included in the final analysis. The applicant classes ranged from years 2014 to 2020. Two reviewers independently evaluated the quality of the included studies using the Joanna Briggs Institute (JBI) tool for cross-sectional studies. The mean JBI score of included studies was 7.4 out of a maximum of 8, with higher scores indicating better quality. The primary outcome was to determine the distribution of applicant SLOR rankings in their summative score. Summative scores were separated into ranked to match, top one-third, middle one-third, lower one-third, and not a fit. Four studies reported the summative SLOR scores of applicants. Our secondary study goal was to assess applicant characteristics associated with SLOR summative scores and assess any bias present in the SLOR or narrative recommendation. Five studies compared SLOR ratings across applicant characteristics including gender, race, USMLE Step 1 score, USMLE Step 2 score, Alpha Omega Alpha (AΩA) membership, clerkship grades, and publications. Gender and racial bias were also assessed across five studies. Four studies utilized a linguistic analysis software to search letters of recommendation for differences in word category use by gender and race. Studies consistently found that a higher percentage of candidates were identified in the top percentile blocks than is mathematically possible. For example, the two studies that tallied the proportion of candidates ranking in the top 10% of applicants found that 36% and 42% were rated as being in the top 10%. Similarly, articles found more than 87% of applicants scored in the top one-third. Studies had mixed results, but in general, they suggested that AΩA status, higher Step 1 scores, and more research publications have a slightly positive association with increased SLOR scores. We found no evidence of gender bias against women, and in fact, most studies evaluating word choices found word choices for women candidates generally were stronger. Similarly, no consistent disparities were identified in word categories utilized in SLORs based on applicant race. Most notably, a mathematically impossible proportion of applicants were counted in the top percentiles in letters of recommendation. This compromises readers' abilities to differentiate candidates. Factors like AOA status and research publications displayed a modest positive correlation with SLOR scores. Gender bias against women or candidates from racial minority groups was not evident. We suggest that group SLOR authorship, with a consensus assessment by clerkship or residency directors, should be considered. We also recommend the incorporation of mean and SD scores for each letter writer (or group) alongside their letters. Furthermore, studies indicate that submitting all four SLOR letters can substantially improve an applicant's chances of success. We advise students to choose strategically and submit the maximum allowable number of SLORs.

CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.

In the early 1990s, an emergency medicine (EM) program director remediated a resident for over a year, to no avail. The resident's contract was not renewed, and a recommendation made that the resident consider another specialty. When this decision was discussed with the department chair and clerkship director, who had written a very positive and “flowery” narrative letter of recommendation (NLOR), the chair said, “We knew this resident would struggle.” The “fluffed up” letter was a disservice to colleagues and to the resident, who spent a difficult year in over her head. At the time, the general discussion among EM program directors indicated that accurate information transfer was a common limitation of the NLOR. Often, NLORs included no objective data (not even the EM clerkship grade) and provided no global comparison to other students. It was perceived that, often, one could not get to a “bottom line” view of the candidate despite a lengthy letter that was time consuming to prepare. In an attempt to address the problems with NLORs, a council of EM residency directors (CORD) subcommittee developed a standardized letter of recommendation (SLOR) in 1995, which was initiated in 1997.1 The SLOR offered more objective data than the NLOR, including an evaluation of EM clerkship performance and a prediction by the writer of how their program might rank the student. The original SLOR included the following 4 sections: (A) background information (clerkship performance); (B) qualifications for EM (personal characteristics relating to the choice of EM); (C) global assessment (comparisons to other students); and (D) written comments. As discussed in a paper by Girzadas et al,2 it quickly became clear that the SLOR was easier to prepare and read with the new format. Whether it made information transfer more reliable was a separate question, and that led to research of potential SLOR limitations. That research, similar to what Girzadas et al2 had found, suggested some problems.3,4 Potential biases resulting in grade inflation of the SLOR were uncovered. Areas of concern included gender bias, inexperience by letter writers, and the duration of time the letter writer knew the applicant.3,4 Another paper by a CORD task force demonstrated evidence of SLOR “grade inflation,” as 40% of reviewed SLORs rated their applicants in the “top 10%,” and over 95% of these SLORs rated applicants in the top third.5 Finally, when rank lists were compared with the global assessment question regarding estimated rank list position, overestimation on the SLOR occurred 66% of the time.6 The SLOR is central to 2 papers in this issue of the Journal of Graduate Medical Education. The study by Diab et al7 shows that the SLOR, with its measurable categories, allows research into the application process. Diab et al7 demonstrated a significant increase in the global assessment ranking of “outstanding” in letters where applicants did not waive their Family Educational Rights and Privacy Act (FERPA) rights, suggesting that if a faculty member is aware that an applicant may read their SLOR, the grade may be inflated.7 Thankfully, 93% of applicants waived their FERPA rights. The study is limited in that we do not know whether applicants who did not waive their rights were representative of the whole population of applicants, but it does suggest one should consider the possibility of bias if no waiver is present. The paper by Hegarty et al8 describes the work of a CORD SLOR task force that was convened in 2011 to review the SLOR and determine whether improvements could be recommended. Although only 37% of the group surveyed had read the CORD guidelines in the previous year, these guidelines were very general and did not include specific recommendations for each question or even for each of the 4 sections. The consequence was great variability in how the question regarding “One Key Comment from ED Faculty Evaluations” was addressed in section A. The manner in which answers were interpreted was similarly variable in section B. Perhaps the most interesting finding from the 2011 CORD SLOR task force work was the way question 12, regarding score inflation, was answered by those surveyed. The results indicated that 36.2% reported they “rarely” inflated, 21.4% reported that they “sometimes” inflated, and 2.6% frequently inflated. My math (36.2 + 21.4 + 2.6 = 60.2) tells me that 60.2% of those surveyed admitted to inflating scores. This clearly agrees with earlier studies that showed SLOR inflation with a number of potential contributors. It seems clear that the SLOR is neither completely objective nor highly accurate in terms of applicant ranking. Readers of the SLOR ideally would need to determine where the writer falls on the inflation scale, but that is not easily done. We are left to wonder what factors, such as FERPA or experience or time knowing the applicant or never really learning one's fractions, are at play when the SLOR is written. The main question underlying the inflation issue is how best to honestly advocate for our students without doing them or one's colleagues a disservice. The recommendations of the CORD task force resulted in a new 2013–2014 edition of the SLOR with a mindful name change: the standardized letter of evaluation (SLOE). The SLOE applied task force recommendations in an attempt to standardize the writer response process. In section B, the “Qualifications” questions compare students to “peers” rather than use a number of adjectives that were hard to define. Other changes included eliminating confusing questions or parts of questions, such as the “Key Comments” requested in section A and the multiplication numbers in section C that described rank (2×, 4×, and 6×). However, the most significant change may be the theoretical shift that the new name represents. This is not a form to be used to blindly and subjectively recommend our students but rather to objectively evaluate their performance and attributes and accurately identify how they compare to a competitive pool of applicants. Despite this aim, there are persistent limitations to the SLOE. For example, all program directors want to meet the students who score highest in every SLOE category, and we avoid those with the lowest SLOE scores. However, most applicants do not fit this profile. Thus, we must seek to understand the applicant's true ranking from how much positive or negative concordance exists among SLOE sections, from the ever-shrinking narratives provided or by prioritizing the items or sections that we feel are the best predictors of performance in our specific programs. The greatest positive result from the CORD SLOR task force study is the effort by CORD to continue to evaluate and refine this important tool. Future efforts are needed to improve the accuracy and reliability of the SLOE, especially with a consistently increasing applicant pool. Further studies to better elucidate causes of rank inflation could include the use of more detailed instructions that are section-based or question-based. Additional outcome data for the applicants who are scored in the middle or lower thirds may help program directors have more confidence in considering these applicants. We all recognize that despite the limitations discussed, the SLOE is not only the best tool for EM educators, it may be the best tool among any specialty. At the end of the day, even though we know the SLOE is not perfect, EM faculty spend less time creating and interpreting these imperfect recommendations than we did when the NLOR was the only option.

Women are underrepresented in neurosurgical training, with progressively fewer women in successive ranks. Letters of recommendation (LORs) are crucial for granting residency interviews, but data suggest they are biased by the applicant's gender, with women applicants receiving lower-quality letters. This study examines whether the standardized LOR (SLOR) format, introduced in the 2020 neurosurgery match, reduces gender bias. The authors reviewed Electronic Residency Application Service applications submitted to a single neurosurgery training program in 2020. LORs were separated by applicant gender and subjected to quantitative linguistic analysis using a custom MATLAB script. LORs were assessed for length, classification as letters of minimal assurance, and gendered stereotypic adjectives. An LOR provides an overall assessment of a candidate's potential and typically includes a description of the writer's relationship to the applicant, a description of intent as demonstrated by the record of the applicant, and a review of the applicant's accomplishments. Omission of any of these three components gives the reader only a partial grasp of a candidate's qualifications, resulting in a letter of minimal assurance. Gendered stereotypic adjectives are the result of a significant body of research devoted to analysis of gender differences in language. These adjectives fall within five word groups: standout, ability, research, grindstone, and teaching, with the latter two categories more often being used to describe females. SLORs were analyzed by noting the percentile ranking in each category. United States Medical Licensing Examination Step 1 scores were compared. Univariate analysis was performed for each LOR variable, using two-tailed t-tests. The distribution of SLOR percentile rankings was evaluated using a Kolmogorov-Smirnov test. A p value < 0.05 was considered significant. Lastly, match outcome data were analyzed to assess impact on outcome. The authors reviewed 910 narrative LORs (NLORs) and 566 SLORs. Of the applicants, 71% were male and 29% were female. There were no significant differences in LOR length, total letters per applicant, or Step 1 scores by gender. Female applicants were more likely to have received letters of minimal assurance (5% vs 1%) and be described by teaching words (p = 0.0038). This difference persisted when evaluating narrative portions from SLORs alone. There were no differences in percentile rankings or match rates. Women receive more letters of minimal assurance and are more likely to be described with teaching words in NLORs, indicating a gender bias. Whether this influences match outcome is difficult to determine based on poor match data, but the present research suggests it does not. Nevertheless, SLORs may help to reduce the identified narrative bias.

Develop a standardized letter of recommendation (SLOR) for otolaryngology residency application that investigates the qualities desired in residents and the letter writer's experience. Compare this SLOR to narrative letters of recommendation (NLORs). Prospective SLOR/NLOR comparison. The SLOR was sent to an NLOR writer for each applicant. The applicant's NLOR/SLOR pair was blinded and ranked in seven categories by three reviewers. Inter-rater reliability and NLOR/SLOR rankings were compared. Means of cumulative NLOR and SLOR scores were compared to our departmental rank list. Thirty-one SLORs (66%) were collected. The SLORs had higher inter-rater reliability for applicant's qualifications for otolaryngology, global assessment, summary statement, and overall letter ranking. Writer's background, comparison to contemporaries/predecessors, and letter review ease had higher inter-rater reliability on the NLORs. Mean SLOR rankings were higher for writer's background (P = .0007), comparison of applicant to contemporaries/predecessors (P = .0031), and letter review ease (P < .0001). Mean SLOR writing time was 4.17 ± 2.18 minutes. Mean ranking time was significantly lower (P < .0001) for the SLORs (39.24 ± 23.45 seconds) compared to the NLORs (70.95 ± 40.14 seconds). Means of cumulative SLOR scores correlated with our rank list (P = .004), whereas means of cumulative NLOR scores did not (P = .18). Means of cumulative NLOR and SLOR scores did not correlate (P = .26). SLORs require little writing time, save reviewing time, and are easier to review compared to NLORs. Our SLOR had higher inter-rater reliability in four of seven categories and was correlated with our rank list. This tool conveys standardized information in an efficient manner.

The American Orthopaedic Association (AOA) released the standardized letter of recommendation (SLOR) form to provide standardized information to evaluators of orthopaedic residency applicants. The SLOR associates numerical data to an applicant's letter of recommendation. However, it remains unclear whether the new letter form effectively distinguishes among orthopaedic applicants, for whom letters are perceived to suffer from "grade inflation." In addition, it is unknown whether letters from more experienced faculty members differ in important ways from those written by less experienced faculty. (1) What proportion of SLOR recipients were rated in the top 10th percentile and top one-third of the applicant pool? (2) Did letters from program leaders (program directors and department chairs) demonstrate lower aggregate SLOR scores compared with letters written by other faculty members? (3) Did letters from away rotation program leaders demonstrate lower aggregate SLOR scores compared with letters written by faculty at the applicant's home institution? This was a retrospective, single institution study examining 559 applications from the 2018 orthopaedic match. Inclusion criteria were all applications submitted to this residency. Exclusion criteria included all letters without an associated SLOR. In all, 1852 letters were received; of these, 26% (476) were excluded, and 74% (1376) were analyzed for SLOR data. We excluded 12% (169 of 1376) of letters that did not include a final summative score. Program leaders were defined as orthopaedic chairs and program directors. Away rotation letters were defined as letters written by faculty during an applicant's away rotation. Our study questions were answered accounting for each subcategory on the SLOR (scale 1-10) and the final ranking (scale 1-5) to form an aggregated score from the SLOR form for each letter. All SLOR questions were included in the creation of these scores. Correlations between program leaders and other faculty letter writers were assessed using a chi-square test. We considered a 1-point difference on 5-point scales to be a clinically important difference and a 2-point difference on 10-point scales to be clinically important. We found that 36% (437 of 1207) of the letters we reviewed indicated the candidate was in the top 10th percentile of all applicants evaluated, and 51% (619 of 1207) of the letters we reviewed indicated the candidate was in the top one-third of all applicants evaluated. We found no clinically important difference between program leaders and other faculty members in terms of summative scores on the SLOR (1.9 ± 0.7 versus 1.7 ± 0.7, mean difference -0.2 [95% CI -0.3 to 0.1]; p < 0.001). We also found no clinically important difference between home program letter writers and away program letter writers in terms of the mean summative scores (1.9 ± 0.7 versus 1.7 ± 0.7, mean difference 0.2; p < 0.001). In light of these discoveries, programs should examine the data obtained from SLOR forms carefully. SLOR scores skew very positively, which may benefit weaker applicants and harm stronger applicants. Program leaders give summative scores that do not differ substantially from junior faculty, suggesting there is no important difference in grade inflation between these faculty types, and as such, there is no strong need to adjust scores by faculty level. Likewise, away rotation letter writers' summative scores were not substantially different from those of home institution letters writers, indicating that there is no need to adjust scores between these groups either. Based on these findings, we should interpret letters with the understanding that overall there is substantial grade inflation. However, while weight used to be given to letters written by senior faculty members and those obtained on away rotations, we should now examine them equally, rather than trying to adjust them for overly high or low scores. Level III, therapeutic study.

The standardized letter of evaluation (SLOE) was created in 1997 to provide residency program directors (PDs) with a summative evaluation that incorporates normative grading (i.e., comparisons to peers applying to emergency medicine [EM] training). Although the standard letter of recommendation (SLOR) has become increasingly popular and important in decision-making, it has not been studied in the past 12 years. To assess the SLOR's effectiveness and limitations, the perspective of EM PDs was surveyed in this study. After validation of the questionnaire by 10 retired PDs, the survey was sent to the PD of each of the 159 EM residencies that existed at that time. The survey was circulated via the Council of Emergency Medicine Residency Directors' (CORD) listserv from January 24, 2013, to February 13, 2013. Weekly e-mail reminders to all PDs served to increase participation. A total of 150 of 159 PDs (94.3%) completed the questionnaire. Nearly all respondents (149 of 150; 99.3%) agreed that the SLOR is an important evaluative tool and should continue to be used. In the application process, 91 of 150 (60.7%) programs require one or more SLORs, and an additional 55 (36.7%) recommend but do not require a SLOR to be considered for interview. When asked to identify the top three factors in deciding who should be interviewed, the SLOR was ranked first (139 of 150; 92.7%), with EM rotation grades ranked second (73 of 150; 48.7%). The factors that were most often identified as the top three that diminish the value of the SLOR in order were 1) "inflated evaluations" (121 of 146; 82.9%), 2) "inconsistency between comments and grades" (106 of 146; 72.6%), and 3) "inadequate perspective on candidate attributes in the written comments" and "inexperienced authors" (60 of 146; 41.1% each). The SLOR appears to be the most important tool in the EM PD's armamentarium for determining which candidates should be interviewed for residency training. Although valuable, the SLOR's potential utility is hampered by a number of factors, the most important of which is inflated evaluations. Focused changes in the SLOR template should be mindful that it appears, in general, to be successful in its intended purpose.

To evaluate gender-based differences in narrative letters of recommendation (NLORs) and standardized letters of recommendation (SLORs) submitted for applicants to an otolaryngology head and neck surgery (OHNS) residency program. Retrospective review. Nine hundred fifty-eight letters of recommendation (LORs) submitted to one OHNS program in 2013 to 2014 were analyzed. NLORs (n = 590) and SLORs (n = 368) were reviewed. Male writers composed over 85% of all LORs. Compared with male writers, female writers of LORs were more likely to compose a letter of minimal assurance (P < 0.025). Female writers of SLORs were more likely to rank applicants higher in communication skills (P < 0.035) and match potential (P < 0.045). Analysis of LORs by applicant gender revealed that female applicants were more likely than male applicants to be described as team players (P < 0.025) and less likely to receive a letter of minimal assurance (P < 0.001). Compared with SLORs, NLORs written for male applicants were more likely to reference their leadership potential (P < 0.001). Female applicants were less likely to be described as "bright" (P < 0.001) and more likely to have their appearance mentioned (P < 0.03) in NLORs when compared to SLORs. Although we found that SLORs reduce gender biases compared to NLORs, there remain significant differences in how female and male applicants are described. Greater awareness of historical biases likely has contributed to more equitable letter writing, although impactful gender biases remain when reviewing applicants. NA. Laryngoscope, 127:2738-2745, 2017.

Development and Assessment of Early Utilization of the Standardized Letter of Recommendation for Use in the Radiology Residency Match

Objective Letters of recommendation (LORs) are an inescapable aspect of the application process. Standardized LORs (SLORs) have been developed and compared with traditional narrative LORs (NLORs). This study investigated whether there was a difference in degree of association between LOR types and face-to-face interviews. Interviewer preference for LOR was anonymously surveyed. Study Design Survey. Setting Single-institution otolaryngology residency program, 1 year before and 4 years after introduction of SLORs. Subjects and Methods Residency interviewers indicated on a visual analog scale how well their impression of an applicant compared between the LOR and the face-to-face interview. Interviewers assessed each applicant, each year, based on LOR type. Results Of 2573 assessments, 964 were collected (37.5% response rate), including 927 responses related to NLORs, 561 to SLORs, and 316 to medical student performance evaluations (ie, dean's letters). The average association of VAS scores between LORs and interviews ranged from 72 to 81 across years and LOR types. Sixty-one percent of interviewers preferred NLORs, and 13% preferred SLORs. Reasons for these preferences included more information provided in NLORs versus faster read time with SLORs. Conclusions Interviewers' perceptions of applicants based on LORs and face-to-face interviews were comparable across LOR types over the last 5 years. For many reasons, the general utility of the LOR remains questionable, and the continued need for it should be critically assessed.

Background: Standardized letters of recommendation (SLORs) were introduced to facilitate the comparison of applicants for orthopaedic surgery residency positions, but concerns have arisen regarding the prevalence of their use and potential limitations. Methods: An 11-question electronic survey was sent to all letter of recommendation (LOR) authors and program coordinators who were identified as having completed or prepared a SLOR during the 2020 orthopaedic surgery residency match cycle. A total of 740 LOR authors and 218 program coordinators were invited via initial and reminder electronic mail messages. Results: The survey response rate was 18.1% for LOR authors and 25.2% for program coordinators. The proportion of LORs written that were SLORs significantly increased from prior to the 2020 match cycle to the 2020 match cycle (72.7% to 90.2%, ratio =1.240, P&lt;0.001). There was not a significant increase in the proportion of LORs that were SLORs prepared by program coordinators (83.7% to 77.6%, ratio =0.927, P=0.375). A majority of LOR authors and program coordinators were aware of electronic (82.1% and 76.5%, respectively) and paper SLORs (91.0% and 88.2%, respectively). Conclusions: This study found that SLOR usage increased among LOR authors. However, a parallel increase in SLOR preparation was not reported by program coordinators. Level of Evidence: Level III

The Utility of the Plastic Surgery Standardized Letter of Recommendation Form in Predicting Residency Match Outcomes

Historically, narrative letters of recommendation have been utilized in the selection of applicants for otolaryngology residency programs. In the last two application cycles, our specialty adopted a standardized letter of recommendation (SLOR). The intent was to decrease time burden for letter writers and to provide readers with an objective evaluation of applicants. The objective of this study was to determine attributes in the SLOR that correlate with matching into a residency program. We performed a retrospective study using SLOR, United States Medical Licensing Examination (USMLE) step 1 scores, and matched outcomes of applicants who applied to our institution for the 2013 and 2014 match cycle. We included the following variables from the SLOR in the statistical analysis to determine which ones were associated with matching: patient care, medical knowledge, communication skills, procedural skills, research, initiative and drive, commitment to otolaryngology, commitment to academic medicine, match potential, and USMLE1 scores. We identified 532 applicants and 963 SLOR. In successful applicants, scores for patient care, medical knowledge, communication skills, initiative and drive, and match potential were statistically higher (P < 0.05). Scores for professionalism, procedural skills, research, commitment to otolaryngology, commitment to academic medicine, and USMLE step 1 scores were not higher among successfully matched applicants. Although SLOR can save time for letter writers and provide an objective description of applicants, the utility of individual domains within the SLOR is questionable. Additionally, it is concerning that applicants' professionalism and procedural skills are not correlated with matching in our specialty. NA. Laryngoscope, 126:1071-1076, 2016.

Background:Letters of recommendation (LOR) are an important component of trainee advancement and assessment. Examination of word use in LOR has demonstrated significant differences in how letter writers describe female and male applicants. Given the emphasis on increasing both gender and racial diversity among orthopaedic surgeons, we aimed to study gender and racial differences in LOR for applicants to orthopaedic surgery residencies.Methods:All applications submitted to a single, academic orthopaedic residency program in 2018 were included. Self-identified gender and race were recorded. The LOR were analyzed via a text analysis software program using previously described categories of communal, agentic, grindstone, ability, and standout words. We examined the relative frequency of word use in letters for (1) male and female applicants and (2) white and underrepresented in orthopaedics (UiO) applicants, with the subgroup analysis based on whether standardized (using the American Orthopaedic Association template) or traditional (narrative) LOR were used.Results:Two thousand six hundred twenty-five LOR were submitted for 730 applicants (79% men). Fifty-nine percent of applicants were self-identified as white, and 34% were self-identified as UiO. In traditional LOR, standout words (odds ratio [OR] 1.07; p = 0.01) were more likely to be used in letters for women compared with men, with no difference in any other word-use category. In standardized LOR, there were no gender-based differences in any word category. In traditional LOR, grindstone words (OR = 0.96; p = 0.02) were more likely to be used in letters for UiO than white applicants, whereas standout words (OR = 1.05; p = 0.04) were more likely to be used in letters for white candidates. In standardized LOR, there were no race-based differences in any word category use.Conclusions:Small differences were found in the categories of words used to describe male and female candidates and white and UiO candidates. These differences were not present in the standardized LOR compared with traditional LOR. It is possible that the use of standardized LOR may reduce gender- and race-based bias in the narrative assessment of applicants.