Model Similarity, Model Selection, and Attribute Classification.

Abstract

Selecting the most appropriate cognitive diagnosis model (CDM) for an item is a challenging process. Although general CDMs provide better model-data fit, specific CDMs have more straightforward interpretations, are more stable, and can provide more accurate classifications when used correctly. Recently, the Wald test has been proposed to determine at the item level whether a general CDM can be replaced by specific CDMs without a significant loss in model-data fit. The current study examines the practical consequence of the test by evaluating whether the attribute-vector classification based on CDMs selected by the Wald test is better than that based on general CDMs. Although the Wald test can detect the true underlying model for certain CDMs, it is yet unclear how effective it is at distinguishing among the wider range of CDMs found in the literature. This study investigates the relative similarity of the various CDMs through the use of the newly developed dissimiliarity index, and explores the implications for the Wald test. Simulations show that the Wald test cannot distinguish among additive models due to their inherent similarity, but this does not impede the ability of the test to provide higher correct classification rates than general CDMs, particularly when the sample size is small and items are of low quality. An empirical example is included to demonstrate the viability of the procedure.