Assessing Differential Item Functioning Among Multiple Groups: A Comparison of Three Mantel-Haenszel Procedures

Abstract

It is often the case in performing a differential item functioning (DIF) analysis that comparisons are made between a single reference group and multiple focal groups. Conducting a separate test of DIF for each focal group has several undesirable qualities: (a) the Type I error rate will exceed the intended nominal level if the level of significance for each individual test is not appropriately adjusted, (b) the power may not be as high as a single test that assesses DIF among all groups simultaneously, and (c) substantial time and computing resources are required. These drawbacks are potentially avoided by using a procedure that has the capacity to assess DIF across all groups simultaneously. In this study I compare the performance of three methods of assessing DIF across multiple demographic groups; the Mantel-Haenszel chi-square statistic with no adjustment to the alpha level, the Mantel-Haenszel chi-square statistic with a Bonferroni adjusted alpha level, and the Generalized Mantel-Haenszel statistic (GMH) that offers a single test of significance across all groups. Simulations were conducted in which there was a single reference group and 1, 2, 3, and 4 focal groups, having from 1 to all of the focal groups in a given condition experiencing DIF. Additional conditions that were varied included group size, focal group ability distribution, and magnitude of matching criterion contamination. The results suggest that GMH is in general the most appropriate procedure because its Type I error rate remained at the nominal level of 0.05, and its power was consistently among the highest.