A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models.

Yanlou Liu,Hao Yin,Tao Xin,Laicheng Shao,Lu Yuan

doi:10.3389/fpsyg.2019.01137

Abstract

As a class of discrete latent variable models, cognitive diagnostic models have been widely researched in education, psychology, and many other disciplines. Detecting and eliminating differential item functioning (DIF) items from cognitive diagnostic tests is of great importance for test fairness and validity. A Monte Carlo study with varying manipulated factors was carried out to investigate the performance of the Mantel-Haenszel (MH), logistic regression (LR), and Wald tests based on item-wise information, cross-product information, observed information, and sandwich-type covariance matrices (denoted by Wd, WXPD, WObs, and WSw, respectively) for DIF detection. The results showed that (1) the WXPD and LR methods had the best performance in controlling Type I error rates among the six methods investigated in this study and (2) under the uniform DIF condition, when the item quality was high or medium, the power of WXPD, WObs, and WSw was comparable with or superior to that of MH and LR, but when the item quality was low, WXPD, WObs, and WSw were less powerful than MH and LR. Under the non-uniform DIF condition, the power of WXPD, WObs, and WSw was comparable with or higher than that of LR.

Highlights

Cognitive diagnostic models (CDMs) as a class of discrete latent variable models have been developed to provide finer-grained and multidimensional diagnostic feedback information about examinees’ strengths and weaknesses on a set of attributes
The results showed that (1) the WXPD and logistic regression (LR) methods had the best performance in controlling Type I error rates among the six methods investigated in this study and (2) under the uniform differential item functioning (DIF) condition, when the item quality was high or medium, the power of WXPD, WObs, and WSw was comparable with or superior to that of MH and LR, but when the item quality was low, WXPD, WObs, and WSw were less powerful than MH and LR
The Type I error rates for WXPD and LR were in the range of [0.025, 0.075] under most of the simulation conditions, which suggested that WXPD and LR had the best performance in controlling Type I error rates among the six methods investigated in this study

Summary

INTRODUCTION

Cognitive diagnostic models (CDMs) as a class of discrete latent variable models have been developed to provide finer-grained and multidimensional diagnostic feedback information about examinees’ strengths and weaknesses on a set of attributes. Hou et al (2014) and Wang et al (2014) found that the Wald statistic (Wd) based on the information matrix estimation method developed by de la Torre (2011) yielded inflated Type I error rates. Svetina et al (2018) evaluated the impact of Q-matrix misspecification on the performance of LR, MH, and Wd for detecting DIF in CDMs. Hou et al (2014) and Wang et al (2014) found that the Wald statistic (Wd) based on the information matrix estimation method developed by de la Torre (2011) yielded inflated Type I error rates.

BACKGROUND

RESULTS

SUMMARY AND DISCUSSION