Do Chief Resident Scores on the In-Service Examination Predict Their Performance on the American Board of Urology Qualifying Examination?

Abstract

The American Urological Association In-Service Examination and the American Board of Urology Qualifying Examination are written multiple choice tests that cover all domains in urology. We investigated whether In-Service Examination performance could identify chief residents who scored in the lowest quartile on the Qualifying Examination. All urology chief residents in the United States and Canada in 2008 and 2009 were eligible to participate in this study. In-Service Examination 2008 and Qualifying Examination 2009 performance data were obtained from the American Urological Association and American Board of Urology, respectively. Data were analyzed with the Pearson correlation and descriptive statistics. Of the 257 American and Canadian chief residents who completed the Qualifying Examination 2009, 194 (75%) enrolled in this study and were included in analysis. Overall In-Service Examination 2008 scores correlated significantly with Qualifying Examination 2009 scores (r=0.55, p<0.001), accounting for 30% of score variance. Substantial variability in In-Service Examination-Qualifying Examination rankings was notable among individual residents. An In-Service Examination 2008 cutoff percentile rank of 40% identified chief residents in the lowest quartile on the Qualifying Examination 2009 with 71% sensitivity, 77% specificity, and a likelihood ratio of 3.1 and 0.4 (positive and negative likelihood ratios, respectively). The substantial variability of In-Service Examination-Qualifying Examination performance among individual chief residents limits In-Service Examination predictive utility. A single In-Service Examination score should not be used to make a high stakes judgment about an individual resident. In-Service Examination scores should be used as 1 part of an overall evaluation program to prospectively identify residents who could benefit from additional educational support.