Prediction Bias and Selection Bias: An Empirical Analysis

Abstract

Empirical examination of the relation between prediction bias and selection bias has been hampered by the lack of a valid definition of selection bias. In this article, we show that such a definition is possible in the special case of fixed-n selection, where the only reason for rejecting otherwise acceptable applicants is that their number exceeds the number of available places. Using the proposed definition, the empirical relation between prediction bias and selection bias with respect to prominent social groups is examined. The results indicate that although the 2 biases are related, the relation is not isomorphic: First, it is mediated by the selection ratio; second, for most selection ratios, its strength is only moderate. Thus, both the direction and the magnitude of prediction bias may be misleading as a basis for determining the corresponding direction and magnitude of selection bias. In particular, lack of prediction bias does not entail lack of selection bias. Hence, examination of selection bias with respect to social groups cannot rely only on empirical investigation of prediction bias. Rather, it calls for direct investigation of selection bias per se.