Validation of Mathematics Construct and Diagnostic Profiles Using a Differential Item Functioning Approach

Abstract

The quality of diagnostic profiles and probability assignment depends on the validity of the proposed attributes and Q-matrix. The rule-space method (RSM), one of diagnostic classification models, provides the quality indices of diagnostic profiles, such as the classification rate and the squared Mahalanobis distance. The study aims to further validate the attributes and Q-matrix that represent construct validity of mathematics tests as well as diagnostic profiles by applying differential item functioning (DIF) analysis along with a different grouping technique. The grouping technique is based on the Q-matrix and diagnostic profiles (i.e., attribute mastery patterns and probabilities) from the RSM. For the validity evidence, it is expected to exhibit DIF for all test items in this study. The results indicated that approximately 90% of items were identified as DIF items that all favored the mastery groups. This suggests that the proposed attributes and Q-matrix with RSM work well to map Taiwanese students into appropriate attribute mastery patterns and to assign appropriate attribute mastery probabilities to them. This study also provides evidence that a DIF approach with the new grouping technique is a viable method for the validation research. More discussion and suggestions are presented in the paper.