Comparing Methods of Assessing Differential Item Functioning in a Computerized Adaptive Testing Environment

Abstract

Mantel‐Haenszel and SIBTEST, which have known difficulty in detecting non‐unidirectional differential item functioning (DIF), have been adapted with some success for computerized adaptive testing (CAT). This study adapts logistic regression (LR) and the item‐response‐theory‐likelihood‐ratio test (IRT‐LRT), capable of detecting both unidirectional and non‐unidirectional DIF, to the CAT environment in which pretest items are assumed to be seeded in CATs but not used for trait estimation. The proposed adaptation methods were evaluated with simulated data under different sample size ratios and impact conditions in terms of Type I error, power, and specificity in identifying the form of DIF. The adapted LR and IRT‐LRT procedures are more powerful than the CAT version of SIBTEST for non‐unidirectional DIF detection. The good Type I error control provided by IRT‐LRT under extremely unequal sample sizes and large impact is encouraging. Implications of these and other findings are discussed.