Traditional Item Response Theory Models Research Articles

Measuring Learning Progress Based on Cognitive Operations

Measuring students’ growth and change is considered one of the main ways for evidence-based development of educational systems. However, it is a non-trivial methodological task, despite the numerous approaches available for its conceptualization and statistical realization. In this article, we describe the main features of measuring students' growth and change using Item Response Theory (IRT) in detail. We then expand this approach to allow for the modeling of cognitive operations with the Linear Logistic Test Model (LLTM). We show that the synthesis of traditional IRT models for measuring growth and change with LLTM significantly enriches the interpretability of ability estimates while preserving the advantages of the traditional approach. To illustrate this approach, we use a set of monitoring tests to measure educational progress in mathematics in secondary school.

Item parameter recovery via traditional 2PL, Testlet and Bi-factor models for Testlet-Based tests

The testlet comprises a set of items based on a common stimulus. When the testlet is used in the tests, there may violate the local independence assumption, and in this case, it would not be appropriate to use traditional item response theory models in the tests in which the testlet is included. When the testlet is discussed, one of the most frequently used models is the testlet response theory (TRT) model. In addition, the bi-factor model and traditional 2PL models are also used for testlet-based tests. This study aims to examine the item parameters estimated by these three calibration models of the data properties produced under different conditions and to compare the performances of the models. For this purpose, data were generated under three conditions: sample size (500, 1000, and 2000), testlet variance (.25, .50, and 1), and testlet size (4 and 10). For each simulation condition, the number of items in the test was fixed at i = 40 and 100 replications were made under each condition. Among these models, it was concluded that the TRT model gave less biased results than the other two models, but the results of the bi-factor model and the TRT were more similar as the sample size increased. Among the examined conditions, it was determined that the most effective variable in parameter recovery was the sample size.

Matching IRT Models to Patient-Reported Outcomes Constructs: The Graded Response and Log-Logistic Models for Scaling Depression

Item response theory (IRT) model applications extend well beyond cognitive ability testing, and various patient-reported outcomes (PRO) measures are among the more prominent examples. PRO (and like) constructs differ from cognitive ability constructs in many ways, and these differences have model fitting implications. With a few notable exceptions, however, most IRT applications to PRO constructs rely on traditional IRT models, such as the graded response model. We review some notable differences between cognitive and PRO constructs and how these differences can present challenges for traditional IRT model applications. We then apply two models (the traditional graded response model and an alternative log-logistic model) to depression measure data drawn from the Patient-Reported Outcomes Measurement Information System project. We do not claim that one model is “a better fit” or more “valid” than the other; rather, we show that the log-logistic model may be more consistent with the construct of depression as a unipolar phenomenon. Clearly, the graded response and log-logistic models can lead to different conclusions about the psychometrics of an instrument and the scaling of individual differences. We underscore, too, that, in general, explorations of which model may be more appropriate cannot be decided only by fit index comparisons; these decisions may require the integration of psychometrics with theory and research findings on the construct of interest.

Parameter Recovery and Classification Accuracy Under Conditions of Testlet Dependency: A Comparison of the Traditional 2PL, Testlet, and Bi-Factor Models

ABSTRACTTestlets, or groups of related items, are commonly included in educational assessments due to their many logistical and conceptual advantages. Despite their advantages, testlets introduce complications into the theory and practice of educational measurement. Responses to items within a testlet tend to be correlated even after controlling for latent ability, which violates the assumption of conditional independence made by traditional item response theory models. The present study used Monte Carlo simulation methods to evaluate the effects of testlet dependency on item and person parameter recovery and classification accuracy. Three calibration models were examined, including the traditional 2PL model with marginal maximum likelihood estimation, a testlet model with Bayesian estimation, and a bi-factor model with limited-information weighted least squares mean and variance adjusted estimation. Across testlet conditions, parameter types, and outcome criteria, the Bayesian testlet model outperformed, or performed equivalently to, the other approaches.

IRT Model Misspecification and Measurement of Growth in Vertical Scaling

Functional form misfit is frequently a concern in item response theory (IRT), although the practical implications of misfit are often difficult to evaluate. In this article, we illustrate how seemingly negligible amounts of functional form misfit, when systematic, can be associated with significant distortions of the score metric in vertical scaling contexts. Our analysis uses two‐ and three‐parameter versions of Samejima's logistic positive exponent model (LPE) as a data generating model. Consistent with prior work, we find LPEs generally provide a better comparative fit to real item response data than traditional IRT models (2PL, 3PL). Further, our simulation results illustrate how 2PL‐ or 3PL‐based vertical scaling in the presence of LPE‐induced misspecification leads to an artificial growth deceleration across grades, consistent with that commonly seen in vertical scaling studies. The results raise further concerns about the use of standard IRT models in measuring growth, even apart from the frequently cited concerns of construct shift/multidimensionality across grades.

MultiLCIRT: An R package for multidimensional latent class item response models

A class of Item Response Theory (IRT) models for binary and ordinal polytomous items is illustrated and an R package for dealing with these models, named MultiLCIRT, is described. The models at issue extend traditional IRT models allowing for multidimensionality and discreteness of latent traits. They also allow for different parameterizations of the conditional distribution of the response variables given the latent traits, depending on both the type of link function and constraints imposed on the discriminating and difficulty item parameters. These models may be estimated by maximum likelihood via an Expectation–Maximization algorithm, which is implemented in the MultiLCIRT package. Issues related to model selection are also discussed in detail. In order to illustrate this package, two datasets are analyzed: one concerning binary items and referred to the measurement of ability in mathematics and the other one coming from the administration of ordinal polytomous items for the assessment of anxiety and depression. In the first application, aggregation of items in homogeneous groups is illustrated through a model-based hierarchical clustering procedure which is implemented in the proposed package. In the second application, the steps to select a specific model having the best fit in the class of IRT models at issue are described.

Simple imputation methods versus direct likelihood analysis for missing item scores in multilevel educational data

Missing data, such as item responses in multilevel data, are ubiquitous in educational research settings. Researchers in the item response theory (IRT) context have shown that ignoring such missing data can create problems in the estimation of the IRT model parameters. Consequently, several imputation methods for dealing with missing item data have been proposed and shown to be effective when applied with traditional IRT models. Additionally, a nonimputation direct likelihood analysis has been shown to be an effective tool for handling missing observations in clustered data settings. This study investigates the performance of six simple imputation methods, which have been found to be useful in other IRT contexts, versus a direct likelihood analysis, in multilevel data from educational settings. Multilevel item response data were simulated on the basis of two empirical data sets, and some of the item scores were deleted, such that they were missing either completely at random or simply at random. An explanatory IRT model was used for modeling the complete, incomplete, and imputed data sets. We showed that direct likelihood analysis of the incomplete data sets produced unbiased parameter estimates that were comparable to those from a complete data analysis. Multiple-imputation approaches of the two-way mean and corrected item mean substitution methods displayed varying degrees of effectiveness in imputing data that in turn could produce unbiased parameter estimates. The simple random imputation, adjusted random imputation, item means substitution, and regression imputation methods seemed to be less effective in imputing missing item scores in multilevel data settings.

Multidimensional Scoring of Abilities: The Ordered Polytomous Response Case

Recent work has shown that multidimensionally scoring responses from different tests can provide better ability estimates. For educational assessment data, applications of this approach have been limited to binary scores. Of the different variants, the de la Torre and Patz model is considered more general because implementing the scoring procedure does not require prior knowledge of the correlational structure of the abilities, and existing item parameter estimates from traditional item response theory models can be utilized. This article extends the application of this method to data scored on ordered polytomous scales. A simulation study systematically examines how improvement in ability estimates is affected by factors such as the number of score categories, number of tests, test length, and correlation between abilities. Application of the method is illustrated using real data. Estimates of the abilities and the correlational structure are obtained using the Markov chain Monte Carlo method.

An Extension of the IRT to a Network Model

The traditional Item Response Theory (IRT) models assume local independence, which is equivalent to the assumption of unidimensionality. This assumption states that a subject’s responses to different items in a test are statistically independent. For the assumption to be true, a subject’s performance on one item must not affect, either for better or for worse, his or her responses to any other items in the test. The main purpose of this paper is to relax the local independence assumption in the traditional IRT models by extending to a network model. A new IRT model is defined which assumes probabilistic network structures for the assumption of local independence. Another unique feature of the model proposed is that it is a new probabilistic network model with the conditional probability parameters depending on a latent trait variable. Information criteria AIC and BIC are used to evaluate the performance of the model proposed, using actual test. data. It shows that the proposed model provides better results than the traditional model. In addition, this paper proposes an item selection criterion from the decision theoretic approach. The amount of test information is defined as the amount of mutual information between a variable for the item and all variables over the test, to maximize the prediction efficiency of the subject’s responses. The new item selection method is used to compare the prediction efficiency between the proposed model and the traditional IRT model. The proposed model is shown to be more efficient.