The Multidimensionality of Measurement Bias in High‐Stakes Testing: Using Machine Learning to Evaluate Complex Sources of Differential Item Functioning

Abstract

AbstractTest developers and psychometricians have historically examined measurement bias and differential item functioning (DIF) across a single categorical variable (e.g., gender), independently of other variables (e.g., race, age, etc.). This is problematic when more complex forms of measurement bias may adversely affect test responses and, ultimately, bias test scores. Complex forms of measurement bias include conditional effects, interactions, and mediation of background information on test responses. I propose a multidimensional, person‐specific perspective of measurement bias to explain how complex sources of bias can manifest in the assessment of human knowledge, skills, and abilities. I also describe a data‐driven approach for identifying key sources of bias among many possibilities—namely, a machine learning method commonly known as regularization.