Exploration of polytomous-attribute Q-matrix validation in cognitive diagnostic assessment

Abstract

Compared with typical binary attributes, polytomous attributes can take three or more values (corresponding to different levels of mastery of a respondent or measurement of an item). They can indicate whether a respondent possesses the attributes of interest and mastery levels. Therefore, the test with polytomous-attribute Q-matrix can become more informative and provide respondents with richer diagnostic information than the test based on the dichotomous-attribute Q-matrix. This paper extends the S-statistic and the residual method applicable for the Q-matrix of binary attributes to validate the polytomous-attribute Q-matrix. Under two common scenarios in real-world applications, two associated validation algorithms: the joint validation (JV) algorithm and the online validation (OV) algorithm, are proposed. Both simulation studies and an empirical data example were employed to assess the robustness and usefulness of these two methods under various conditions. Results indicate that the JV algorithm is suitable for validating a Q-matrix predefined by subject matter experts. Especially when the Q-matrix contains fewer misspecifications, while the OV algorithm can be applied to define the attribute vector of "new items". Based on a certain number of "operational items", the OV algorithm can achieve a promising performance for obtaining the specification of the new items.