Abstract
In recent decades, cognitive diagnostic models (CDMs) have been intensively researched and applied to various educational and psychological tests. However, because existing CDMs fail to consider rater effects, the application of CDMs to constructed-response (CR) items that involve human raters is seriously limited. Given the popularity of CR items, it is desirable to develop new CDMs that are capable of describing and estimating rater effects on CR items. In this study, we developed such new CDMs within the frameworks of facets models and hierarchical rater models, using the log-linear cognitive diagnosis model as a template. The parameters of the new models were estimated with the Markov chain Monte Carlo methods implemented in the freeware JAGS. Simulations were conducted to evaluate the parameter recovery of the new models. Results showed that the parameters were recovered fairly well and the more data there were, the better the recovery. Implications and applications of the new models were illustrated with an empirical study that adopted a fine-grained checklist to assess English academic essays.
Highlights
TO THE FACETS AND HRM APPROACHESThe Facets ApproachIn the facets approach, raters are treated as instruments to measure ratees, just like items are
Given the popularity of CR items, it is desirable to develop new cognitive diagnostic models (CDMs) that are capable of describing and estimating rater effects on CR items. We developed such new CDMs within the frameworks of facets models and hierarchical rater models, using the log-linear cognitive diagnosis model as a template
In the facets model (Linacre, 1989), the log-odds of scoring k over k – 1 on item j for ratee i judged by rater r is defined as: log(Pijkr/Pij(k−1)r) = θi − βjk − ηr where Pijkr and Pij(k−1)r are the probabilities of receiving a score of k and k - 1, respectively, for ratee i on item j from rater r; θi is the latent trait of ratee i and is often assumed to follow a normal distribution; βjk is the kth threshold of item j; ηr is the severity of rater r
Summary
TO THE FACETS AND HRM APPROACHESThe Facets ApproachIn the facets approach, raters are treated as instruments to measure ratees, just like items are. In the facets model (Linacre, 1989), the log-odds (logit) of scoring k over k – 1 on item j for ratee i judged by rater r is defined as: log(Pijkr/Pij(k−1)r) = θi − βjk − ηr (1). Where Pijkr and Pij(k−1)r are the probabilities of receiving a score of k and k - 1, respectively, for ratee i on item j from rater r; θi is the latent (continuous) trait of ratee i and is often assumed to follow a normal distribution; βjk is the kth threshold of item j; ηr is the severity of rater r. To account for the intra-rater fluctuations in severity, Wang and Wilson (2005) proposed adding a random-effect parameter to the facets model, which can be expressed as: log(Pijkr/Pij(k−1)r) = θi − βjk − (ηr + ζir)
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
