Using machine learning to improve Q-matrix validation.

Abstract

The Q-matrix, which specifies the relationship between items and attributes, is a crucial component of cognitive diagnostic models (CDMs). A precisely specified Q-matrix allows for valid cognitive diagnostic assessments. In practice, a Q-matrix is usually developed by domain experts, and noted as being subjective and potentially containing misspecifications which can decrease the classification accuracy of examinees. To overcome this, some promising validation methods have been proposed, such as the general discrimination index (GDI) method and the Hull method. In this article, we propose four new methods for Q-matrix validation based on random forest and feed-forward neural network techniques. Proportion of variance accounted for (PVAF) and coefficient of determination (i.e., the McFadden pseudo-R2) are used as input features for developing the machine learning models. Two simulation studies are carried out to examine the feasibility of the proposed methods. Finally, a sub-dataset of the PISA 2000 reading assessment is analyzed as illustration.