Performance of Polytomous IRT Models With Rating Scale Data: An Investigation Over Sample Size, Instrument Length, and Missing Data

Cihan Demir,Thao Thu Vo,Haixia He,Olasunkanmi James Kehinde,Shenghai Dai,Yu Xue,Xiaolin Wang

doi:10.3389/feduc.2021.721963

Cihan Demir, Thao Thu Vo + Show 5 more

Open Access

https://doi.org/10.3389/feduc.2021.721963

Copy DOI

Abstract

The implementation of polytomous item response theory (IRT) models such as the graded response model (GRM) and the generalized partial credit model (GPCM) to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used. The performance of such models has not been fully investigated and compared across conditions with common survey-specific characteristics such as short test length, small sample size, and data missingness. The purpose of the current simulation study is to inform the literature and guide the implementation of GRM and GPCM under these conditions. For item parameter estimations, results suggest a sample size of at least 300 and/or an instrument length of at least five items for both models. The performance of GPCM is stable across instrument lengths while that of GRM improves notably as the instrument length increases. For person parameters, GRM reveals more accurate estimates when the proportion of missing data is small, whereas GPCM is favored in the presence of a large amount of missingness. Further, it is not recommended to compare GRM and GPCM based on test information. Relative model fit indices (AIC, BIC, LL) might not be powerful when the sample size is less than 300 and the length is less than 5. Synthesis of the patterns of the results, as well as recommendations for the implementation of polytomous IRT models, are presented and discussed.

Highlights

The implementation of polytomous item response theory (IRT) models to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used (e.g., Carle et al, 2009; Sharkness and DeAngelo, 2011; Cordier et al, 2019; French and Vo, 2020)
The current study investigated the performance of graded response model (GRM) and generalized partial credit model (GPCM) with rating scale data across various instrument lengths, sample sizes, item quality, and missing data rates
Synthesizing the results of item parameter estimations for both GRM and GPCM, we identified the following patterns: 1) The estimation of item parameters for GPCM was more stable than for GRM. 2) In general, a small sample size, a short instrument length, poor item quality, and a high missing rate, tended to adversely impact the estimation accuracy of both item discrimination and threshold parameters collectively, especially for the item thresholds

Summary

Introduction

The implementation of polytomous item response theory (IRT) models to inform instrument design and validation has been increasing across social and educational contexts where rating scales are usually used (e.g., Carle et al, 2009; Sharkness and DeAngelo, 2011; Cordier et al, 2019; French and Vo, 2020). The most commonly used polytomous IRT models include the graded response model (GRM; Samejima, 1969) and the generalized partial credit model (GPCM; Muraki, 1992). GRM (Samejima, 1969) is one of the most commonly used polytomous IRT models It extends the dichotomous twoparameter logistic (2PL) IRT model by allowing ordered and polytomous item responses. Unlike the 2PL dichotomous IRT models in which only one item difficulty parameter is defined, the GRM specifies category boundary and threshold parameters for the items according to the number of response categories. For an item with K response categories, a number of K-1 threshold parameters will be specified in GRM. Is the equation of GRM (Embretson and Reise, 2000): Ppjy(θ) exp aj θ − δjm 1 + exp aj θ − δjm (1)

Objectives

Methods

Results

Conclusion