The Development of CD-CAT with Polytomous Attributes

Abstract

Cognitive diagnostic computerized adaptive testing(CD-CAT) purports to combine the strengths of both CAT and cognitive diagnosis. The attributes in CD-CAT have been supposed to be dichotomous, "0" represents non-master for examinee and non-measurement for item while "1" represents master for examinee and measurement for item. But recently, polytomous attribute framework has been proposed by some studies(Karelitz, 2004; de la Torre, et al., 2010; Chen, et al., 2013). Based on the conventional computerized adaptive testing for cognitive diagnosis with dichotomous attributes, this study developed a new CD-CAT with polytomous attributes, called p CD-CAT, which is adaptive to polytomous or ordered category attributes framework. During the procedure, two key parts were involved in the p CD-CAT. One was parameters estimation. Here the cognitive diagnosis model with polytomous attributes was employed to estimate the individual's polytomous knowledge states with maximum likelihood estimator(MLE) algorithm. Another was the item selection strategy. In this paper, the PAKL and PAPWKL methods based on polytomous attributes were developed and implied to select items adaptively from item pool. Three Monte Carlo simulation studies with different experimental conditions were conducted here, which mainly focused on the performance of the proposed p CD-CAT by this paper. These experimental conditions were the fixed test length CAT(15, 20 and 25 items respectively), the variable test length CAT(the post probability of knowledge state is 0.75,0.80 and 0.85 respectively) and the compare between p CD-CAT and the conventional dichotomous CD-CAT, respectively. There studies showed: The classification accuracy, test security and test efficiency under p CD-CAT were all acceptable and reasonable. The PA-KL item selection strategy had low classification, test security and test efficiency, which indicated that PA-KL was unfit to the p CD-CAT. However PA-PWKL and PA-HKL item selection strategies had high classification accuracy, test security and test efficiency. In addition, while using conventional CD-CAT to fit the p CD-CAT with polytomous attributes, the PMA(attribute pattern match ration) was less than 30% and the test security indexes(e.g. item exposure and test overlap) were poorer than the p CD-CAT. All above results indicated that the conventional CD-CAT should not be employed to fit the polytomous attributes, while the method proposed by this paper is a good choose. All in all, the p CD-CAT overcame the shortcomings stemmed from dichotomous CD-CAT, thus they might expect a good prospect and application. And it provided a kind of new methods and techniques in cognitive diagnosis, which might extended the applicable area.