The Jackson 5 System for Creating Rank Order Lists: Easy as A-B-C, 1-2-3

Abstract

Every year, residency and fellowship programs create rank order lists (ROLs) that organize applicants by preference, with the most desirable candidates at the top. This ranking serves as a proxy of predicted performance within the environment of that residency program.1,2 The Henry Ford Hospital Emergency Medicine and Transitional Year programs traditionally created ROLs using quantified data from a candidate's application (file score) added to an average interview score to calculate a composite score. One problem with this approach was that a candidate with a high file score and low interview score would rank similarly to a candidate with modest file and interview scores, even though the latter candidate likely had less risk and was therefore more desirable. In addition, this method was inefficient, as lengthy discussions often compared adjacent candidates who were likely to perform equally well. Several years ago, we modified our ROL creation process to address these issues and improve efficiency. We designed a risk-based, criterion-referenced approach called the “Jackson 5” (J5) method for creating ROLs.J5 involves a fundamental change in the approach to ROL creation. Rather than compare candidates to each other and rank them, each candidate is independently assessed on their probability of success in the residency based on their application and interview. Candidates with the most desirable applications are assigned the letter code “A,” indicating that these candidates will likely require minimal academic guidance during residency (Table). Similarly, candidates with poor communication skills or values that do not align with the program's mission are assigned the numeric code “4.” Each candidate is therefore assigned to 1 of 16 J5 cohorts ranging from “A1” to “D4.”On the initial ROL, the cohorts are ranked in the order predetermined by the program faculty (A1, B1, A2, B2, C1, C2, A3, B3, C3, D1, A4, D2, B4, C4, D3, D4). The faculty then review the ROL with the understanding that candidates within a cohort will likely perform similarly as residents, and the individual rank of each candidate within a cohort is less important. The final ROL discussion focuses around whether a candidate was assigned to the correct cohort instead of how they compare to adjacent candidates.We have used J5 since 2017. To measure the effort required to finalize an ROL, we calculated the Pearson correlation coefficient between initial and final ROLs for 4 sequential recruitment cycles. Initial ROLs were created using the traditional method (TRAD, comparing candidates to each other and ranking using a composite scoring system) and using the J5 system (J5ROL, described above). After creation of each initial ROL, faculty had a detailed discussion to create a final ROL (FIN). We hypothesized that if the J5ROL-FIN correlation was better than the TRAD-FIN correlation, it implied that less effort was required to finalize and submit a ROL to the Match. Our results confirmed this, as J5ROL-FIN correlation was 0.89 (95% CI 0.87-0.90) compared to 0.81 for TRAD-FIN (95% CI 0.78-0.84; P<.001).A faculty survey revealed broad acceptance of J5 and preference over the traditional method. For example, 75% (9 of 12) agreed that J5 was more efficient compared to the traditional method, and 71% (10 of 14) felt that J5 allowed them to adequately express their opinion on candidates.In conclusion, J5 provides an efficient, feasible, and acceptable framework to create a ROL. Any site or specialty may implement this because it only requires a rethinking of the approach to comparing candidates and developing program-specific application and interview cohort criteria. We hope that it helps residency programs better recruit candidates aligned with their program mission.