Testing Person Fit in Cognitive Diagnosis

Abstract

In cognitive diagnosis, the test-taking behavior of some examinees may be idiosyncratic so that their test scores may not reflect their true cognitive abilities as much as that of more typical examinees. Statistical tests are developed to recognize the following: (a) nonmasters of the required attributes who correctly answer the item (spuriously high scores) and (b) masters of the attributes who fail to correctly answer the item (spuriously low scores). For a person, nonzero probability of aberrant behavior is tested as the alternative hypothesis, against normal behavior as the null hypothesis. The two generalized likelihood ratio test statistics used, with the null hypothesis parameter on the boundary of the parameter space in each, have asymptotic distributions of a 50:50 mixture of a chi-square distribution with one degree of freedom and a degenerate distribution that is a constant of 0 under the null hypothesis. Simulation results, primarily based on the DINA model (deterministic inputs, noisy ‘‘AND’’ gate), are used to investigate the following: (a) how accurately the statistical tests identify normal/aberrant behaviors, (b) how the power of the tests depends on the length of the cognitive exam and the degree of the inclination toward aberrance, and (c) how sensitive the tests are to inaccurate estimation of model parameters.