Modeling Randomness in Judging Rating Scales with a Random‐Effects Rating Scale Model

Abstract

This study presents the random‐effects rating scale model (RE‐RSM) which takes into account randomness in the thresholds over persons by treating them as random‐effects and adding a random variable for each threshold in the rating scale model (RSM) (Andrich, 1978). The RE‐RSM turns out to be a special case of the multidimensional random coefficients multinomial logit model (MRCMLM) (Adams, Wilson, & Wang, 1997) so that the estimation procedures for the MRCMLM can be directly applied. The results of the simulation indicated that when the data were generated from the RSM, using the RSM and the RE‐RSM to fit the data made little difference: both resulting in accurate parameter recovery. When the data were generated from the RE‐RSM, using the RE‐RSM to fit the data resulted in unbiased estimates, whereas using the RSM resulted in biased estimates, large fit statistics for the thresholds, and inflated test reliability. An empirical example of 10 items with four‐point rating scales was illustrated in which four models were compared: the RSM, the RE‐RSM, the partial credit model (Masters, 1982), and the constrained random‐effects partial credit model. In this real data set, the need for a random‐effects formulation becomes clear.