Resident Selection Committees Research Articles

Bias Reduction Practices in Underrepresented Groups in Ophthalmology Resident Recruitment

Best recruitment practices for increasing diversity are well established, but the adoption and impact of these practices in ophthalmology residency recruitment are unknown. To describe the adoption of bias reduction practices in groups underrepresented in ophthalmology (URiO) residency recruitment and determine which practices are effective for increasing URiO residents. This cross-sectional survey study used an 18-item questionnaire included in the online survey of the Association of University Professors in Ophthalmology (AUPO) Residency Program Directors. Data collection occurred from July 2022 to December 2022. The data were initially analyzed on January 16, 2023. Participants included residency program directors (PDs) in the AUPO PD listserv database. Descriptive analysis of resident selection committee approaches, evaluation of applicant traits, and use of bias reduction tools. Primary outcome was diversity assessed by presence of at least 1 resident in the last 5 classes who identified as URiO, including those underrepresented in medicine (URiM), lesbian, gay, bisexual, transgender, queer, intersex, and asexual plus, or another disadvantaged background (eg, low socioeconomic status). Multivariate analyses of recruitment practices were conducted to determine which practices were associated with increased URiO and URiM. Among 106 PDs, 65 completed the survey (61.3%). Thirty-nine PDs used an interview rubric (60.0%), 28 used interview standardization (43.0%), 56 provided at least 1 bias reduction tool to their selection committee (86.2%), and 44 used postinterview metrics to assess diversity, equity, and inclusion efforts (67.7%). Application filters, interview standardization, and postinterview metrics were not associated with increased URiO. Multivariate logistic regression analysis showed larger residency class (odds ratio [OR], 1.34; 95% CI, 1.09-1.65; P = .01) and use of multiple selection committee bias reduction tools (OR, 1.47; 95% CI, 1.13-1.92; P = .01) were positively associated with increased URiO, whereas use of interview rubrics (OR, 0.72; 95% CI, 0.59-0.87; P = .001) and placing higher importance of applicant interest in a program (OR, 0.83; 95% CI, 0.75-0.92; P = .02) were negatively associated. URiM analyses showed similar associations. Ophthalmology residency interviews are variably standardized. In this study, providing multiple bias reduction tools to selection committees was associated with increased URiO and URiM residents. Prioritizing applicant interest in a program may reduce resident diversity. Interview rubrics, while intended to reduce bias, may inadvertently increase inequity.

Does Masked Interviewing Encourage Holistic Review in Residency Selection? A Mixed-Methods Study

Problem: Medical educators increasingly champion holistic review. However, in U.S. residency selection, holistic review has been difficult to implement, hindered by a reliance on standardized academic criteria such as board scores. Masking faculty interviewers to applicants’ academic files is a potential means of promoting holistic residency selection by increasing the interview’s ability to make a discrete contribution to evaluation. However, little research has directly analyzed the effects of masking on how residency selection committees evaluate applicants. This mixed-methods study examined how masking interviews altered residency selection in an anesthesiology program at a large U.S. academic medical center. Intervention: During the 2019–2020 residency selection season in the University of Pennsylvania’s Department of Anesthesiology & Critical Care, we masked interviewers to the major academic components of candidates’ application files (board scores, transcripts, letters) on approximately half of interview days. The intent of the masking intervention was to mitigate the tendency of interviewers to form predispositions about candidates based on standardized academic criteria and thereby allow the interview to make a more independent contribution to candidate evaluation. Context: Our examination of the masking intervention used a concurrent, partially mixed, equal-status mixed-methods design guided by a pragmatist approach. We audio-recorded selection committee meetings and qualitatively analyzed them to explore how masking affected the process of candidate evaluation. We also collected independent candidate ratings from interviewers and consensus committee ratings and statistically compared ratings of candidates interviewed on masked days to ratings from conventional days. Impact: In conventional committee meetings, interviewers focused on how to reconcile academic metrics and interviews, and their evaluations of interviews were framed according to predispositions about candidates formed through perusal of application files. In masked meetings, members instead spent considerable effort evaluating candidates’ “fit” and whether they came off as tactful. Masked interviewers gave halting opinions of candidates and sometimes pushed for committee leaders to reveal academic information, leading to masking breaches. Higher USMLE Step 1 score and higher medical school ranking were statistically associated with more favorable consensus rating. We found no significant differences in rating outcomes between masked and conventional interview days. Lessons learned: Elimination of academic metrics during the residency interview phase does not straightforwardly promote holistic review. While critical reflection among medical educators about the fairness and utility of such metrics has been productive, research and intervention should focus on the more proximate topic of how programs apply academic and other criteria to evaluate applicants.

The Impact of the COVID-19 Pandemic on the Experiences of Female, Underrepresented-in-Medicine Surgical Applicants During the 2020-2021 Residency Match

INTRODUCTION: Persons from backgrounds underrepresented-in-medicine (UiM) face disparities in the residency application process, manifested as leading to a lower likelihood of invitations to interview for and therefore match into surgical residency programs. Given the unique challenges imposed by the global pandemic during the 2020−2021 application cycle, we aimed to describe the impact of COVID-19, transition to virtual interviews, and simultaneous social unrest surrounding UiM identities among UiM applicants. METHODS: An anonymous electronic survey was distributed to applicants who responded to a surgical society’s emails/social-media posts between October and December 2020. Differences between self-identified female UiM and non-UiM respondents were compared using descriptive-statistics (quantitative-questions) and grounded-theory determination of emergent-themes (qualitative-questions). RESULTS:N = 238 applicants responded; 45.8% (n = 109) identified as UiM. Compared to non-UiM applicants, UiM applicants reported greater concern about: delayed board examinations (44.0% vs 29.5%, p = 0.042), missed mentorship (38.5% vs 19.4%, p = 0.001), and lack of research experiences (27.5% vs 14.0%, p = 0.020). They focused on the importance of a “holistic” review of applications, concern for interview monopolization, detrimental effects on mental health due to social and political events, and prioritization of diversity/equity/inclusion initiatives when selecting residency programs. CONCLUSION: UiM applicants faced important challenges in their surgical residency applications during 2020-2021. Medical education governing bodies, program directors, and resident selection committees should be vigilant about mitigating these challenges in post-pandemic years. Strategies to increase recruitment of UiM applicants include: transparency/adaptation of holistic application review practices, strategic distribution of interview invitations with discouragement of interview hoarding, and formalization of program’s diversity, equity, and inclusion initiatives.

Real-Time Dual Video Conferencing of Simulated Tracheostoma Maturation During Residency Interviews.

There are few reports of dexterity tests being done in a distance telecommunication setting for residency applicant evaluation. To report the feasibility and suitability of a virtual suturing skills assessment during residency interviews when added to the standard assessment process. A suturing simulation was developed and implemented during otolaryngology-head and neck surgery (OHNS) residency interviews for the 2020-2021 cycle at one program. On the day of the interview, the activity was completed in real time using 2-camera video conferencing with the 2 resident assessors providing a numerical assessment based on an adapted scoring rubric from prior suturing activities at the institution. The exercise involved suturing a 3/4-inch Penrose drain circumferentially with half-vertical mattress stitches to simulate the maturation of a tracheostoma. The residency selection committee then completed a 7-item Likert-type survey, developed by the authors, to evaluate the simulation exercise. Fifty-one applicants representing all interviewees in the cycle successfully completed this assessment without technologic disruptions. The total cost associated with obtaining and providing the necessary supplies to applicants was $34.78 per interviewee. Time required to complete the suturing task was estimated to range from 10 to 20 minutes. The residency selection committee viewed this exercise as a success (14 of 16, 87.5%) and viewed the results as a valuable adjunct in the overall assessment of candidates (15 of 16, 93.8%). A simple motor exercise completed over real-time telecommunication was feasible and perceived as helpful to the residency selection committee when assessing OHNS residency candidates.

Exploring Characteristics of Academic General Surgery Residency Applicants: A Group Concept-Mapping Approach

Holistic review, which emphasizes qualitative attributes over objective measures, has been proposed as a method for selecting candidates for surgical residency in order to improve diversity in graduate medical education, and, ultimately, the field of surgery. This study seeks to articulate desirable traits of applicants as a first-step in standardizing the holistic review process. Using Group Concept Mapping, a web-based mixed-methods participatory research methodology, residency selection committee members were asked to 1) list desirable characteristics of applicants, 2) group these into categories, 3) rate their importance to academic/clinical success on a 5-point Likert scale (1 = not at all important, 5 = extremely important), and 4) rate the degree to which each characteristic is feasible to assess on a 3-point Likert scale (1 = not at all feasible, 3 = very feasible). Grouped characteristics submitted to hierarchical cluster analysis depicted committee's consensus about desirable qualities/criteria for applicants. Bivariate scatter-plots and pattern-matching graphics demonstrated which of these criteria were most important and reliably assessed. A single academic general surgery residency training program in Western Pennsylvania. Members of the selection committee for the UPMC General Surgery Residency program who had participated in at least 1 prior cycle of applicant selection. Desirable characteristics of highly qualified applicants into an academic general surgery residency were clustered into domains of 1) scholarly work and research, 2) grades/formal assessments, 3) program fit, 4) behavioral assets, and 5) aspiration. Behavioral assets, which was felt to be the most important to clinical and academic success were considered to be the least feasible to reliably assess. Within this domain, initiative, being self-motivated, intellectual curiosity, work ethic, communication skills, maturity and self-awareness, and thoughtfulness were viewed as most frequently reliably assessed from the application and interview process. High quality applicants possess several behavioral assets that faculty deem are important to academic and clinical success. Adapting validated metrics for assessing these assets, may provide a solution for addressing subjectivity and other challenges scrutinized by critics of holistic review.

AADPRT Recommends Virtual Interviewing For 2022-2023 Recruitment Year

AADPRT Recommends Virtual Interviewing For 2022-2023 Recruitment Year

Innovation in Resident Selection: Life Without Step 1.

INTRODUCTIONThe announcement of Step 1 shifting to a Pass/Fail metric has prompted resident selection committees (RSCs) to pursue objective methods of evaluating prospective residents. Regardless of the program's specialty or affiliated hospital/school, RSCs universally aim to recognize and choose applicants who are an “optimal fit” to their programs.1 An optimal fit can be defined as a candidate who thrives in the clinical and academic setting, both contributing to and benefiting from their respective training environments.OBJECTIVEThe objective of this scoping review is to evaluate alternative, innovative methods by which RSCs can evaluate applicants and predict success during residency. Objective methods include: Step 2 scores, Traditionally Used Metrics (core clerkship scores), interview performance, musical talent, sports involvement, AOA membership, research publications, unprofessional behavior, Dean's letters, Rank list, judgement testing, and specialty-specific shelf exams.13–15METHODSA scoping review was performed in compliance with the guidelines indicated by the PRISMA Protocol for scoping review.18 9308 results were identified in the original PubMed search for articles with the key words “Resident Success”. screening and application of inclusion and exclusion criteria yielded 97 articles that were critically appraised via review of full manuscript.RESULTSOf the articles that focused on personality traits, situational judgement testing, and specialty specific pre-assessment, all of them demonstrated some level of predictability for resident success. Standardized Letter of Recommendations, Traditionally Used Metrics, and STEP 2 did not show a unanimous consensus in demonstrating predictability of a resident's success, this is because some articles suggested predictability and some articles disputed predictability.CONCLUSIONThe authors found personality traits, situational judgement testing, and specialty specific assessments to be predictive in selecting successful residents. Further research should aim to analyze exactly how RSCs utilize these assessment tools to aid in screening their large and competitive applicant pools to find residents that will be successful in their program.

Rationales for a Lottery Among the Qualified to Select Medical Trainees: Decades of Dutch Experience

Rationales for a Lottery Among the Qualified to Select Medical Trainees: Decades of Dutch Experience

CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.

CORR Insights®: More Than One-third of Orthopaedic Applicants Are in the Top 10%: The Standardized Letter of Recommendation and Evaluation of Orthopaedic Resident Applicants.

Toward Creating Equity in Awards Received During Medical School: Strategic Changes at One Institution.

Membership in the Alpha Omega Alpha Honor Medical Society (AΩA) is a widely recognized achievement valued by residency selection committees and employers. Yet research has shown selection favors students from racial/ethnic groups not underrepresented in medicine (not-UIM). The authors describe efforts to create equity in AΩA selection at the University of California, San Francisco, School of Medicine, through implementation of a holistic selection process, starting with the class of 2017, and present outcomes.Informed by the definition of holistic review, medical school leaders applied strategic changes grounded in evidence on inclusion, mitigating bias, and increasing opportunity throughout the AΩA selection process. These addressed increasing selection committee diversity, revising selection criteria and training committee members to review applications using a new instrument, broadening student eligibility and inviting applications, reviewing blinded applications, and making final selection decisions based on review and discussion of a rank-ordered list of students that equally weighted academic achievement and professional contributions.The authors compared AΩA eligibility and selection outcomes for 3 classes (2014-2016) during clerkship metric-driven selection, which prioritized academic achievement, and 3 classes (2017-2019) during holistic selection. During clerkship metric-driven selection, not-UIM students were 4 times more likely than UIM students to be eligible for AΩA (P = .001) and 3 times more likely to be selected (P = .001). During holistic selection, not-UIM students were 2 times more likely than UIM students to be eligible for AΩA (P = .001); not-UIM and UIM students were similarly likely to be selected (odds ratio = .7, P = .12)This new holistic selection process created equity in representation of UIM students among students selected for AΩA. Centered on equity pedagogy, which advocates dismantling structures that create inequity, this holistic selection process has implications for creating equity in awards selection during medical education.

The system of values and styles of success in the medical career: A longitudinal study.

The purpose of the study was to investigate the relation between system values during medical education and styles of success in the medical career. The participants were first examined when they applied to the medical school. Questionnaires were given to these students each academic year. Medical doctors who had participated in the first phase of the study completed a questionnaire 4 years after their graduation, too. The baseline questionnaire measured the system values. The follow-up questionnaire included measures of quality of life, work stress and burnout, satisfaction with medicine as a career and professional competency. The identified 3 groups of students representing 3 types of careers had been different regarding their preferred terminal values and instrumental values. Out of 3 groups, 2 presented a high risk of burnout. What is more the life goals (terminal values) are relatively stable, but preferable modes of behavior (instrumental values) are likely to change. The most important differences between students who may suffer from burnout later as doctors and those who are at a lower risk are e.g., family security, freedom, happiness, mature love, self-respect, social recognition and wisdom. The Rokeach Value Survey may be applied to identify specific tendencies in the development of medical career. The obtained results may be used by the medical school admission officers as well as resident selection committees in order to identify candidates who may be at risk of professional difficulties. Authors can identify during medical education which student will be at risk of professional burnout after studies. Thus there could be a selection of interventions directed toward differentiated groups of students, e.g.: supplying them with proper coping vs. guiding them, to increase life satisfaction and productivity. Int J Occup Med Environ Health 2018;31(6):823-835.

Current Interview Trail Metrics in the Otolaryngology Match.

Objectives To identify how applicants to otolaryngology residency determine how to apply to, interview with, and rank programs on the interview trail and to determine the extent of the financial burden of the otolaryngology interview trail. Study Design Web-based survey distributed in March and April 2016. Setting Otolaryngology residency applicants throughout the United States. Subjects and Methods Applicants to otolaryngology residency during the 2016 match cycle and current otolaryngology residents were surveyed. Results Median number of applications, interview offers, interviews attended, and programs ranked was not different during the 2016 match and the previous 5 match years. The most important factor affecting the number of applications was the need to apply widely to ensure sufficient interview offers. The most common reason for declining an interview offer was scheduling conflict. Applicants during the 2016 match spent a median of $5400 applying and interviewing for otolaryngology residency. Conclusions Median number of applications, interview offers, interviews attended, and programs ranked has not changed. The most cited reason for applying to many programs was to increase the chances of matching, but this is not statistically likely to increase match success. We advocate for continued attempts to make the otolaryngology match process more transparent for both applicants and resident selection committees, but recognize that applicants are likely to continue to overapply for otolaryngology residency positions.

Preresidency publication record and its association with publishing during paediatric residency

To determine whether an association exists between the publication of journal articles before and during paediatrics residency. A retrospective search of PubMed was conducted for publications by all 567 Canadian paediatricians certified between 2009 and 2012, inclusive. Paediatricians were separated into groups based on the number of articles published preresidency (0 or ≥1) and during residency (0 or ≥1). The methodology was validated using a group of local paediatricians who were contacted to verify whether their publications were identified accurately. A total of 160 of 567 (28%) certified paediatricians had preresidency publications; of these, 93 (58%) subsequently published during their residency period. Among the remaining 407 (72%) paediatricians without preresidency publications, 129 (32%) published during residency. The association between publication before and during paediatric residency was statistically significant (OR 2.98 [95% CI 2.04 to 4.36]; P<0.001). Results from the validation analysis suggested the methodology correctly identified pre- and during residency publication status with 87% and 90% accuracy, respectively. Individuals with previous publications were more likely to publish as residents; however, 42% of individuals with pre-residency publications did not publish as residents. Residency selection committees may find these data helpful in assessing the publication potential of their applicants. In addition, this information may assist in building more targeted and individualized research curricula within residency programs.

Increasing Competitiveness for an Otolaryngology Residency: Where We Are and Concerns about the Future.

For graduating medical students, securing a residency in otolaryngology-head and neck surgery has become exceedingly difficult. This commentary explores the ways that applicants and residency programs are reacting to the increasing competitiveness in applying to, interviewing for, and matching to an otolaryngology residency. The commonly held perceptions of applicants are compared to perspectives held by residency program directors and resident selection committees. Unintended consequences of the growing trend for medical students to add a research year to their curriculum are presented. Some cautions and suggestions about how to improve the application and selection process are offered.

The sense of coherence and styles of success in the medical career: a longitudinal study.

BackgroundSense of coherence is related to well-being, stress and life satisfaction among medical students and physicians. The purpose of the study was to investigate relation between sense of coherence during medical education and styles of success in the medical career.MethodsThe participants were first examined when they applied to medical school in 1999. Questionnaires were given to these students each academic year from 2000 to 2005. Also, 54 medical doctors who had participated in the first phase of the study completed a questionnaire in 2009, four years after graduation. The baseline questionnaire measured the sense of coherence. The follow-up questionnaire included measures of quality of life, work stress and burnout, satisfaction with medicine as a career, and professional competency.ResultsMedical students with the highest sense of coherence later have the highest quality of life and income, and are the least overwhelmed by work stress, but they also show the least satisfaction with medicine as a career and a low level of professional competence.ConclusionsAntonovsky’s SOC-29 questionnaire can be used to identify specific tendencies in the development of the medical career. Our results may be useful to medical school admissions officers and resident selection committees, in order to identify candidates at risk for failure.

Predictors of an Academic Career on Radiology Residency Applications

To evaluate radiology residency applications to determine if any variables are predictive of a future academic radiology career. Application materials from 336 radiology residency graduates between 1993 and 2010 from the Department of Radiology, Duke University and between 1990 and 2010 from the Department of Radiology, Stanford University were retrospectively reviewed. The institutional review boards approved this Health Insurance Portability and Accountability Act-compliant study with a waiver of informed consent. Biographical (gender, age at application, advanced degrees, prior career), undergraduate school (school, degree, research experience, publications), and medical school (school, research experience, manuscript publications, Alpha Omega Alpha membership, clerkship grades, United States Medical Licensing Examination Step 1 and 2 scores, personal statement and letter of recommendation reference to academics, couples match status) data were recorded. Listing in the Association of American Medical Colleges Faculty Online Directory and postgraduation publications were used to determine academic status. There were 72 (21%) radiologists in an academic career and 264 (79%) in a nonacademic career. Variables associated with an academic career were elite undergraduate school (P = .003), undergraduate school publications (P = .018), additional advanced degrees (P = .027), elite medical school (P = .006), a research year in medical school (P < .001), and medical school publications (P < .001). A multivariate cross-validation analysis showed that these variables are jointly predictive of an academic career (P < .001). Undergraduate and medical school rankings and publications, as well as a medical school research year and an additional advanced degree, are associated with an academic career. Radiology residency selection committees should consider these factors in the context of the residency application if they wish to recruit future academic radiologists.

Ultimate Publication Rate of Unpublished Manuscripts Listed on Radiology Residency Applications at One Institution

This study was designed to assess the ultimate publication rate of unpublished manuscripts listed on radiology residency applications. A retrospective review was performed on all 628 Electronic Residency Application Service applications submitted to a single-institution radiology residency program in 2010. Publication status of unpublished manuscripts listed as "accepted," "in press," "provisional accepted," or "submitted" was assessed two years later by searching PubMed, Google Scholar, and journal- or conference-specific Web sites. Manuscripts were considered "published" if they appeared in a peer-reviewed journal with the applicant as an author. Impact factors of proposed and publishing journals were compared. Five hundred fourteen (81.8%) of the 628 applicants listed a total of 1,431 manuscripts: 952 (66.5%) "published," 64 (4.5%) "in press," 95 (6.6%) "accepted," 27 (1.9%) "provisional accepted," and 293 (20.5%) "submitted." After two years, 81.3% (52/64) of "in press," 58.9% (56/95) of "accepted," 70.4% (19/27) of "provisional accepted," and 43.7% (128/293) of "submitted" manuscripts were published (P < .01). The proposed and publishing journal matched in 50.8% (65/128) of "submitted" manuscripts (P < .01). The impact factor of the proposed versus publishing journal was 4.189 versus 2.898 (P < .01) for "submitted" manuscripts. One-third of manuscripts listed on radiology residency applications were unpublished at the time of application. More than half of those listed as "submitted," and approximately one-third of those listed as "accepted," "in press," or "provisional accepted," remained unpublished after two years. Residency selection committees should consider these publication rates when assessing applicants.

The Tip of the Iceberg

Many factors limit the ability of resident selection committees to reliably determine which medical student applicants will be top performers in their internship and residency programs. Resident selection committees expend significant time and effort to read applications and then interview and score applicants, all with the goal of creating a rank order list (ROL) for the National Residency Matching Program. Although much of the information used in this process is outside the control of the selection committee, one factor they can control is how they use committee member scores to create the ROL. ROLs are typically generated using the average score assigned by committee members. With this approach, a single harsh faculty member can strongly influence the rank order of applicants if he or she scores only some of the applicants.The author of this commentary discusses the challenges inherent in the creation of ROLs and examines a new approach, described by Ross and Moore in this issue, that can eliminate the problem of biased scoring. ROLs created using this new ROSS-MOORE (Recruitment Outcomes Simulation System-Moore Optimized Ordinal Rank Estimator) approach will better represent the rank ordering of each faculty member of the selection committee. However, ROLs will remain poorly predictive of future performance in internship and residency programs until evaluative data supplied by medical schools are more accurate in predicting which students will become excellent physicians.

Programs Selection Criteria for Neurological Surgery Applicants in the United States: A National Survey for Neurological Surgery Program Directors

To determine the criteria used by neurosurgery resident selection committees in different programs and to assess the process of selection. A questionnaire based on the Electronic Residency Application Service (ERAS) guidelines was mailed to all 100 neurosurgery residency program directors in August 2011. Items included related to selection process, factors affecting decision, and degree of satisfaction by the process. Responses were recorded by the use of 4- and 3-point Likert scales of importance. Mean values were used to create rank orders of selection criteria. Multivariate analysis was conducted to determine predictors of long-term satisfaction with resident selection. The overall response rate was 46%. In the selection process the most important factors (>3.0) were the interview process (mean ± SD= 3.80 ± 0.65), United States Medical Licensing Examination (USMLE I) (3.58 ± 0.54), and letters of recommendations (3.56 ± 0.54). High satisfaction with selected residents was recorded in 60.86% of program directors. In multivariate analysis, predictors of long-term satisfaction with resident selection included less emphasis on letters of recommendation (P= 0.037) and greater weight on applicant extramural activity (P= 0.038). Interview process, USMLE I, and letters of recommendations were the most important factors affecting residents selection. Most of the program directors are satisfied by this process and applicants pool as it reflects somehow their performance during residency.

Associations between residency selection strategies and doctor performance: a meta-analysis

The purpose of this study was to use meta-analysis to establish which of the information available to the resident selection committee is associated with resident or doctor performance. Multiple electronic databases were searched to 4 September 2012. Two reviewers independently selected studies that met the present inclusion criteria and extracted data in duplicate; disagreement was resolved by consensus. Risk for bias was assessed using a customised bias assessment tool. Measures of association were converted to a common effect size (Hedges' g). Meta-analysis was performed using the random-effects model for each selection strategy and all outcomes without pooling. Sensitivity analysis for each selection strategy-outcome pair was performed with pooling of effect size. Eighty studies involving a total of 41704 participants were included in the meta-analysis. Seventeen different selection strategies and 17 outcomes were assessed across these studies. The strongest positive associations referred to examination-based selection strategies, such as the US Medical Licensing Examination (USMLE) Step 1, and examination-based outcomes, such as scores on in-training examinations. Moderate positive associations were present for medical school marks and both examination-based and subjective outcomes. Minimal or no associations were seen for the selection tools represented by interviews, reference letters and deans' letters. Standardised examination performance and medical school grades show the strongest associations with current measures of doctor performance. Deans' letters, reference letters and interviews all show a lower than expected strength of association given the relative value often assigned to them during resident doctor selection. Objective selection strategies are potentially the most useful to residency selection committees based on current evaluative methods. However, reports in the literature of validated long-term doctor performance outcomes are scant.