Attribute Hierarchy Method Based on Graded Response Model with Different Scoring-Weight for Attributes

Abstract

Compared to the traditional test, the value of test for diagnostic assessment test lies in its ability to reveal each student’s specific cognitive strengths and weaknesses and further helps design effective remedy for individual student. More information for cognitive diagnose could be provided by polytomous scoring than dichotomous scoring. So far, the Polytomous Extension of diagnostic assessment still remains at the stage that all the attributes share the same scoring-weight. It is contrary to the fact that attributes are very likely to have different weights. On the assumption that two students respectively grasp the same number of attributes in an item, but not the same attributes, rater should give more scores to the student who answers the more difficult or key attributes correctly, rather than give the same score. It’s imperative for us to study the Cognitive Diagnostic Models(CDM) based on the attributes with different scoring-weight. In this paper, a method derived from Bayesian Networks and Least Squares Distance theories is proposed to calculate the score weight of attributes. Additionally, this paper discovers and solves a problem that the weight of the same attributes in different items may not be the same. The cognitive diagnostic model in this paper is Weighted Attribute Hierarchy Method (WAHM) with score weights of attributes, which is based on Graded Response Model (GRM), briefly, it is called WAHM-GRM. Four kinds of attribute hierarchies were separately used as the basis for the simulation. A sample of 5000 expected item response vectors was generated based on each of the four expected response patterns which are normally distributed. Each of the four samples consists of expected response patterns which are free from slips, the observed item response patterns were generated by randomly adding slips to each of the expected response patterns. In this study, the percentage of random errors was manipulated to 5%, 10%,15% and 20% of the total number of item responses to examine whether the number of random errors has an impact on the accuracy of classification methods. Simulation results showed that under the condition that attributes with different weights, very high classification accuracy rates remain for all classification methods, including methods A and B, proposed by Leighton et al.(2004) and ration of logarithm likelihood method (LL),proposed by Zhu et al.(2009). Especially for A and B methods, classification accuracy rate of AHM-GRM remains above 90% even when slip is as high as 20%. In Conclusion, AHM-GRM with different weighted attributes has a very high classification accuracy rate. In addition, score weights of attribute can guide item builders to distribute scores to the item attributes at the stage of developing item tests.