Polytomous Item Response Theory Models Research Articles

Assessing fit of alternative unidimensional polytomous IRT models using posterior predictive model checking.

This article explored the application of the posterior predictive model checking (PPMC) method in assessing fit for unidimensional polytomous item response theory (IRT) models, specifically the divide-by-total models (e.g., the generalized partial credit model). Previous research has primarily focused on using PPMC in model checking for unidimensional and multidimensional IRT models for dichotomous data, and has paid little attention to polytomous models. A Monte Carlo simulation was conducted to investigate the performance of PPMC in detecting different sources of misfit for the partial credit model family. Results showed that the PPMC method, in combination with appropriate discrepancy measures, had adequate power in detecting different sources of misfit for the partial credit model family. Global odds ratio and item total correlation exhibited specific patterns in detecting the absence of the slope parameter, whereas Yen's Q1 was found to be promising in the detection of misfit caused by the constant category intersection parameter constraint across items. (PsycINFO Database Record

Computerized Adaptive Testing with R: Recent Updates of the Package catR

The purpose of this paper is to list the recent updates of the R package catR. This package allows for generating response patterns under a computerized adaptive testing (CAT) framework with underlying item response theory (IRT) models. Among the most important updates, well-known polytomous IRT models are now supported by catR; several item selection rules have been added; and it is now possible to perform post-hoc simulations. Some functions were also rewritten or withdrawn to improve the usefulness and performances of the package.

Asymptotic Standard Errors of Observed‐Score Equating With Polytomous IRT Models

In observed‐score equipercentile equating, the goal is to make scores on two scales or tests measuring the same construct comparable by matching the percentiles of the respective score distributions. If the tests consist of different items with multiple categories for each item, a suitable model for the responses is a polytomous item response theory (IRT) model. The parameters from such a model can be utilized to derive the score probabilities for the tests and these score probabilities may then be used in observed‐score equating. In this study, the asymptotic standard errors of observed‐score equating using score probability vectors from polytomous IRT models are derived using the delta method. The results are applied to the equivalent groups design and the nonequivalent groups design with either chain equating or poststratification equating within the framework of kernel equating. The derivations are presented in a general form and specific formulas for the graded response model and the generalized partial credit model are provided. The asymptotic standard errors are accurate under several simulation conditions relating to sample size, distributional misspecification and, for the nonequivalent groups design, anchor test length.

On the Finiteness of the Weighted Likelihood Estimator of Ability.

The purpose of this note is to focus on the finiteness of the weighted likelihood estimator (WLE) of ability in the context of dichotomous and polytomous item response theory (IRT) models. It is established that the WLE always returns finite ability estimates. This general result is valid for dichotomous (one-, two-, three- and four-parameter logistic) IRT models, the class of polytomous difference models and divide-by-total models, independently of the number of items, the item parameters and the response patterns. Further implications of this result are outlined.

Advancing the Bayesian Approach for Multidimensional Polytomous and Nominal IRT Models: Model Formulations and Fit Measures.

It is common to encounter polytomous and nominal responses with latent variables in social or behavior research, and a variety of polytomous and nominal item response theory (IRT) models are available for applied researchers across diverse settings. With its flexibility and scalability, the Bayesian approach using the Markov chain Monte Carlo (MCMC) method demonstrates its great advantages for polytomous and nominal IRT models. However, the potential of the Bayesian approach would not be fully realized without model formulations that can cover various models and effective fit measures for model assessment or criticism. This research first provided formulations for typical models that are representative of different modeling groups. Then, a series of discrepancy measures that can offer diagnostic information for model-data misfit were introduced. Simulation studies showed that the formulation worked as expected, and some of the fit measures were more useful than the others or across different situations.

Adjacent-Categories Mokken Models for Rater-Mediated Assessments.

Molenaar extended Mokken's original probabilistic-nonparametric scaling models for use with polytomous data. These polytomous extensions of Mokken's original scaling procedure have facilitated the use of Mokken scale analysis as an approach to exploring fundamental measurement properties across a variety of domains in which polytomous ratings are used, including rater-mediated educational assessments. Because their underlying item step response functions (i.e., category response functions) are defined using cumulative probabilities, polytomous Mokken models can be classified as cumulative models based on the classifications of polytomous item response theory models proposed by several scholars. In order to permit a closer conceptual alignment with educational performance assessments, this study presents an adjacent-categories variation on the polytomous monotone homogeneity and double monotonicity models. Data from a large-scale rater-mediated writing assessment are used to illustrate the adjacent-categories approach, and results are compared with the original formulations. Major findings suggest that the adjacent-categories models provide additional diagnostic information related to individual raters' use of rating scale categories that is not observed under the original formulation. Implications are discussed in terms of methods for evaluating rating quality.

Power and Sample Size Calculations in Clinical Trials with Patient-Reported Outcomes under Equal and Unequal Group Sizes Based on Graded Response Model: A Simulation Study

ObjectivesTo provide a valid sample size strategy based on simulation and to evaluate the statistical power in clinical trials with patient-reported outcomes (PROs) based on a polytomous item response theory model—the graded response model (GRM)—and to compare this framework with the classical test theory (CTT) approach. MethodsOne thousand randomized clinical trials were simulated using PRO based on the GRM and under various combinations of the number of patients in each arm, the group allocation ratio, the number of items and categories, and group effects. The power and sample size estimated in the simulations were then compared with those computed using the CTT framework. ResultsThe results indicated that the impact of the most influential factors, including the number of patients, group allocation ratio, group effects, and the number of categories, on the power and sample size of the GRM-based and CTT-based approaches was similar. Nevertheless, the strong impact of the number of items on these issues distinguished the two approaches. ConclusionsIt is crucial to use an adapted sample size formula in a GRM-based analysis because the classical formula designed for the CTT-based approach does not consider the impact of the number of items, which could result in an inadequately sized study and a decrease in power. Thus, when clinicians design a randomized clinical trial with polytomous PRO endpoints using classical sample size formula as the base, they should be aware of the possibility of making an incorrect clinical decision.

Effects of item type and estimation method on the accuracy of estimated personality trait scores: Polytomous item response theory models versus summated scoring

Despite an increased use in item response theory (IRT)-based personality testing there is little research documenting whether trait estimations are actually improved over those derived via simply summated scoring according to classical test theory (CTT). In this study personality scale validity was compared using a variety of estimation methods (CTT, adjusted-CTT, SGR, GGUM) and item types (monotonic vs. non-monotonic) for the traits of conscientiousness and extraversion. Regardless of item type or estimation method, trait estimates were highly correlated. Using job performance ratings as an external criterion within the nomological network of these traits, model fit was not related to scale validity, and all estimation procedures resulted in comparable validity coefficients. Implications are discussed.

The Development of IRT Based Attitude Scale towards Educational Measurement Course

In this study, the Scale of Attitude towards Educational Measurement and Evaluation (SAEM) developed by Demirtasli (2002) is reconstructed based on polytomous Item Response Theory (IRT) models and its psychometric features are identified. In this context, the best polythomous IRT model was investigated which is fitted SAEM data. IRT models gives invariant person and item parameters, when data-model fit. A version of SAEM has 41 Likert type items with four points was administered to 519 teacher candidates attending teacher education programs at several universities in Turkey. The data were analyzed according to polythomous IRT models: Samejima’s graded response model (S-GRM), the partial credit model (PCM) and a nominal response model (NRM). The results of the analysis showed that a new version of SAEM, which is based on S-GRM, consists of 33 items, has lower chi-square value than the other models and the classic internal reliability was found to be 0.97. The findings of the study indicate that the validity and reliability features of the scale are fairly good.

An item analysis of the Conditional Reasoning Test of Aggression.

This manuscript uses item response theory (IRT) to estimate item characteristics of the Conditional Reasoning Test of Aggression (CRT-A). Using a sample size of 5,511 respondents, the present analysis provides an accurate assessment of the capability of the CRT-A to measure latent aggression. The one-parameter logistic (1PL) model, two-parameter logistic (2PL) model, and three-parameter logistic (3PL) model are compared before the item analysis. Results suggest that the 2PL model is the most appropriate dichotomous IRT model for describing the item characteristics of the CRT-A. Potential multdimensionality in the CRT-A is also examined. Results suggest that CRT-A items work as theoretically intended, with the probability of selecting an aggressive response increasing with latent trait levels. Information curves indicate that the CRT-A is best suited for use with individuals who are high on latent aggression. Exploratory analyses include an examination of polytomous IRT models and DIF comparing student and employee respondents. The results have implications for future research using the CRT-A as well as the identification of populations appropriate for measurement using this assessment tool.

Using Testlet Response Theory to Examine Local Dependence in C-Tests

C-tests are gap-filling tests widely used to assess general language proficiency for purposes of placement, screening, or provision of feedback to language learners. C-tests consist of several short texts in which parts of words are missing. We addressed the issue of local dependence in C-tests using an explicit modeling approach based on testlet response theory. Data were provided by a total sample of 4,708 participants working on eight C-test texts with 20 gaps each. The resulting parameter estimates were compared to those obtained from (a) a polytomous item response theory (IRT) model and (b) a standard IRT model that ignored the dependence structure. Testlet effects proved to be very small and correspondence between parameters obtained from the different modeling approaches was high. When local dependence was ignored reliability of the C-test was slightly overestimated. Implications for the analysis of testlets in general, and C-tests in particular are discussed.

A Note on the Equivalence Between Observed and Expected Information Functions With Polytomous IRT Models

The purpose of this note is to study the equivalence of observed and expected (Fisher) information functions with polytomous item response theory (IRT) models. It is established that observed and expected information functions are equivalent for the class of divide-by-total models (including partial credit, generalized partial credit, rating scale, and nominal response models) but not for the class of difference models (including the graded response and modified graded response models). Yet, observed information function remains positive in both classes. Straightforward connections with dichotomous IRT models and further implications are outlined.

Effect of Item Response Theory (IRT) Model Selection on Testlet‐Based Test Equating

The local item independence assumption underlying traditional item response theory (IRT) models is often not met for tests composed of testlets. There are 3 major approaches to addressing this issue: (a) ignore the violation and use a dichotomous IRT model (e.g., the 2‐parameter logistic [2PL] model), (b) combine the interdependent items to form a polytomous item and apply a polytomous IRT model (e.g., the graded response model [GRM]), and (c) apply a model that explicitly takes into account the dependence at the item level (e.g., the testlet response theory [TRT] model). In this study, a simulation was conducted to compare the performance of these 3 approaches on number‐correct score equating when degrees of testlet effect were manipulated. The traditional equipercentile method was used as an evaluation baseline. The results show that the 2PL and the TRT approaches produce comparable results that more closely agree with the results of the equipercentile method than the GRM does. And the number‐correct equating using the 2PL is robust to the violation of local item independence.

An NCME Instructional Module on Polytomous Item Response Theory Models

A polytomous item is one for which the responses are scored according to three or more categories. Given the increasing use of polytomous items in assessment practices, item response theory (IRT) models specialized for polytomous items are becoming increasingly common. The purpose of this ITEMS module is to provide an accessible overview of polytomous IRT models. The module presents commonly encountered polytomous IRT models, describes their properties, and contrasts their defining principles and assumptions. After completing this module, the reader should have a sound understating of what a polytomous IRT model is, the manner in which the equations of the models are generated from the model's underlying step functions, how widely used polytomous IRT models differ with respect to their definitional properties, and how to interpret the parameters of polytomous IRT models.

Practical Guide to Conducting an Item Response Theory Analysis

Item response theory (IRT) is a psychometric technique used in the development, evaluation, improvement, and scoring of multi-item scales. This pedagogical article provides the necessary information needed to understand how to conduct, interpret, and report results from two commonly used ordered polytomous IRT models (Samejima’s graded response [GR] model and reduced GR model). Throughout this article, simulated data from a multi-item scale is used to illustrate IRT analyses. The simulated data and IRTPRO version 2.1 point-and-click commands needed to reproduce all analyses in this article are available as supplemental online materials at http://jea.sagepub.com/maint . The intent of this article is to provide an overview of essential components of an IRT analysis to enable increased access to this powerful tool for applied early adolescence researchers.

Estimating the Nominal Response Model Under Nonnormal Conditions

The nominal response model (NRM), a much understudied polytomous item response theory (IRT) model, provides researchers the unique opportunity to evaluate within-item category distinctions. Polytomous IRT models, such as the NRM, are frequently applied to psychological assessments representing constructs that are unlikely to be normally distributed in the population. Unfortunately, models estimated using estimation software with the MML/EM algorithm frequently employs a set of normal quadrature points, effectively ignoring the true shape of the latent trait distribution. To address this problem, the current research implements an alternative estimation approach, Ramsay Curve Item Response Theory (RC-IRT), to provide more accurate item parameter estimates modeled under the NRM under normal, skewed, and bimodal latent trait distributions for ordered polytomous items. Based on the results of improved item parameter recovery under RC-IRT, it is recommended that RC-IRT estimation be implemented whenever a researcher considers the construct being measured has the potential of being nonnormally distributed.

Development of a Short Version of the Visual Function Questionnaire Using Item-Response Theory

PurposeIn clinical ophthalmology as in other fields, measuring patient-reported outcomes imposes a burden on patients. To decrease that burden, we used item-response theory (IRT) to develop and test a short version of the National Eye Institute's Visual Function Questionnaire (VFQ).MethodsWe analyzed VFQ data from 276 adults in Japan. Most of them had glaucoma, cataract, or macular degeneration. Their visual acuity (Snellen fraction) averaged 20/120 (range: 20/13 to 20/2000) for the better eye, and 20/200 (range: 20/13 to 20/2000) for the worse eye. We used a polytomous IRT model, the Generalized Partial Credit Model as implemented in software for parameter scaling of rating data (PARSCALE). To select items for inclusion in the short version we examined each item's location on the latent-trait continuum, its slope, and its frequency of missing data. We also ensured representation of all 7 domains that are important in Japan. To examine the characteristics of the resulting scale, we computed its test information (an index of precision that can vary with the value of the latent trait), and carried out validation testing.ResultsFrom 32 of the original VFQ items, we selected 11. The scale comprising those 11 items (the VFQ-J11) had test information greater than 9 for values of the latent trait between −2.0 and +0.8. The item thresholds were well-targeted for patients with vision problems. Scores on the VFQ-J11 correlated strongly and in the expected direction with measures of visual field and corrected visual acuity. As expected for a valid measure, those scores also improved by a large amount (almost one standard deviation) after cataract surgery.ConclusionThis 11-item instrument can provide reliable and the valid data on visual functioning in patients with ophthalmic problems. It is expected to be less of a burden on respondents, while it maintains good psychometric properties.

Handbook of Polytomous Item Response Theory Models edited by Michael L. Nering and Remo Ostini

Handbook of Polytomous Item Response Theory Models edited by Michael L. Nering and Remo Ostini

Development of a brief mental health impairment scale using a nationally representative sample in the USA

A psychometric analysis was conducted to reduce the number of items needed to assess the disability associated with mental disorders using the World Health Organization Disability Assessment Schedule (WHODAS). The WHODAS was to be used in the Substance Abuse and Mental Health Services Administration National Survey on Drug Use and Health (NSDUH), beginning in 2008, as part of a screening algorithm to produce estimates of the prevalence of serious mental illness (SMI) in the US adult population. The goal of the work presented in this paper was to create a parsimonious screening scale from the full 16-item WHODAS that was administered to 24,156 respondents (aged 18+) in the 2002 NSDUH. Exploratory factor analysis showed that WHODAS responses were unidimensional. A two-parameter polytomous Item Response Theory model showed that all 16 WHODAS items had good item discrimination (slopes greater than 1.0) for each response option. Analysis of item difficulties and differential item function across socio-demographic categories was then used to select a subset of eight items to create a short version of the WHODAS. The Pearson correlation between scores in the original 16-item and reduced eight-item WHODAS scales was 0.97, documenting that the vast majority of variation in total scale scores was retained in the reduced scale.

Using a Multivariate Multilevel Polytomous Item Response Theory Model to Study Parallel Processes of Change: The Dynamic Association Between Adolescents' Social Isolation and Engagement With Delinquent Peers in the National Youth Survey

The application of multidimensional item response theory models to repeated observations has demonstrated great promise in developmental research. It allows researchers to take into consideration both the characteristics of item response and measurement error in longitudinal trajectory analysis, which improves the reliability and validity of the latent growth curve (LGC) model. The purpose of this study is to demonstrate the potential of Bayesian methods and the utility of a comprehensive modeling framework, the one combining a measurement model (e.g., a multidimensional graded response model, MGRM) with a structural model (e.g., an associative latent growth curve analysis, ALGC). All analyses are implemented in WinBUGS 1.4.3 (Spiegelhalter, Thomas, Best, & Lunn, 2003), which allows researchers to use Markov chain Monte Carlo simulation methods to fit complex statistical models and circumvent intractable analytic or numerical integrations. The utility of this MGRM-ALGC modeling framework was investigated with both simulated and empirical data, and promising results were obtained. As the results indicate, being a flexible multivariate multilevel model, this MGRM-ALGC model not only produces item parameter estimates that are readily estimable and interpretable but also estimates the corresponding covariation in the developmental dimensions. In terms of substantive interpretation, as adolescents perceived themselves more socially isolated, the chance that they are engaged with delinquent peers becomes profoundly larger. Generally, boys have a higher initial exposure extent than girls. However, there is no gender difference associated with other latent growth parameters.