Who r u?: On the (in)accuracy of incumbent-based estimates of range restriction in criterion-related and differential validity research.

Abstract

Correcting validity estimates for selection procedures for range restriction typically involves comparing variance in predictor scores between all job applicants and applicants who were selected. However, some research on criterion-related and differential validity of cognitive ability tests has relied on range restriction corrections based on data from job incumbents. Unfortunately, there remains ambiguity concerning the accuracy of this incumbent-based approach vis-à-vis the applicant-based approach. To address this issue, we conducted several Monte Carlo simulations, as well as an analysis of college admissions data. Our first simulation study showed that incumbent-based range restriction corrections result in downwardly biased estimates of criterion-related validity, whereas applicant-based corrections were quite accurate. Our second set of simulations showed that incumbent-based range restriction corrections can produce evidence of differential validity when there is no differential validity in the population. In contrast, applicant-based corrections tended to accurately estimate population parameters and showed little, if any, evidence of differential validity when there is no differential validity in the population. Analysis of data for the ACT as a predictor of academic performance revealed similar patterns of bias for incumbent-based corrections in an academic setting. Overall, the present findings raise serious concerns regarding the use of incumbent-based range restriction corrections in lieu of applicant-based corrections. They also cast doubt on recent evidence for differential validity of predictors of job performance. (PsycINFO Database Record