Multidimensional DIF Analyses: The Effects of Matching on Unidimensional Subtest Scores

Abstract

Popular techniques for assessing differential item functioning (DIF) assume that the test under study is unidimensional. When this assumption is tenable, number-correct score is a reasonable matching criterion. When a test is intentionally multidimensional, matching on a single test score does not ensure comparability and may result in inflated error rates. An alternate approach is to match on all relevant traits simultaneously, using a procedure such as logistic regression. In this study, data were generated to simulate two-dimensional tests. The dimensional structure of the tests, the discrimination levels of the items, and the correlation between the traits measured by the test were varied. Standard DIF analyses were conducted using total test score as the matching variable. High false-positive error rates were found. Items were divided into subtests using nonlinear factor analysis and DIF analyses were repeated with subtest scores as the matching criteria. False-positive error rates were reduced for most datasets. The dimensional structure of the test and the discrimination level of the items influenced false-positive rates for both sets of DIEF analyses. The findings suggest that assessing the dimensional structure of a test can be an important first step in DIF analysis. If a dataset is intentionally multidimensional, conditioning on scores reflecting each dimension can enhance the validity of the analyses.