The Impact of Unobserved Extreme Categories on Item and Person Estimates – A Simulation Study

Abstract

For any polytomous items, it sometimes occurs that an extreme category, which is logically possible, is not observed in a particular sample. For example, in education when the performance tasks by the students from different year levels are judged by the same set of criteria, it is likely that none of the lower year level students would achieve the highest marks in some criteria. In health, it may happen when a group of generally healthy participants are measured by an instrument designed to detect some particular symptoms. This paper uses a simulation study to investigate the impact of unobserved extreme categories on item and person estimates. Based on the polytomous Rasch model, the Partial Credit Model (Masters, Psychometrika 47:149–174, 1982), data were simulated for 1,000 persons, N(0,1), and ten polytomous items with five categories under two scenarios, one with unobserved extreme high categories and the other with unobserved extreme low categories. The generated data sets were analysed with the RUMM2030 software (Andrich et al. 2009). The results show that unobserved extreme high categories in the data tend to lead to overestimated person means and unobserved extreme low categories tend to lead to underestimated person means. Both scenarios resulted in underestimated item standard deviations. The results suggest that collapsing unobserved extreme categories improves person and item estimation accuracy, especially when a large proportion of items have an unobserved extreme category. These results have implications for designing and measuring performance tasks that need to be carried out across a wide spectrum of ability groups. It may also affect the common-item equating procedure for polytomous items where the threshold values of common items are used.