Standard Item Response Theory Research Articles

Item Response Theory Models for Carry-Over Effect Across Different Scales.

It is common in educational and psychological tests or social surveys that the same statement is judged on multiple scales. These multiple responses are linked by the same statement, which may cause local dependence. Considering the way a statement is judged on multiple scales, a new class of item response theory (IRT) models is developed to account for the nonrecursive carry-over effect, in which a response can be affected only by its preceding response rather than by a subsequent response. The parameters of the models can be estimated with the freeware WinBUGS. Two simulation studies were conducted to evaluate the parameter recovery of the new models and the consequences of model misspecification. Results showed that the parameters of the new models were recovered fairly well; fitting unnecessarily complicated models to data that did not have the carry-over effect did little harm to parameter estimation; and ignoring the carry-over effect by fitting standard IRT models yielded biased estimates for the item parameters, the correlation between latent traits, and the test reliability. Two empirical examples with parallel design and sequential design are provided to demonstrate the implications and applications of the new models.

Item Response Theory Models for Performance Decline During Testing

Sometimes, test‐takers may not be able to attempt all items to the best of their ability (with full effort) due to personal factors (e.g., low motivation) or testing conditions (e.g., time limit), resulting in poor performances on certain items, especially those located toward the end of a test. Standard item response theory (IRT) models fail to consider such testing behaviors. In this study, a new class of mixture IRT models was developed to account for such testing behavior in dichotomous and polytomous items, by assuming test‐takers were composed of multiple latent classes and by adding a decrement parameter to each latent class to describe performance decline. Parameter recovery, effect of model misspecification, and robustness of the linearity assumption in performance decline were evaluated using simulations. It was found that the parameters in the new models were recovered fairly well by using the freeware WinBUGS; the failure to account for such behavior by fitting standard IRT models resulted in overestimation of difficulty parameters on items located toward the end of the test and overestimation of test reliability; and the linearity assumption in performance decline was rather robust. An empirical example is provided to illustrate the applications and the implications of the new class of models.

IRT Model Misspecification and Measurement of Growth in Vertical Scaling

Functional form misfit is frequently a concern in item response theory (IRT), although the practical implications of misfit are often difficult to evaluate. In this article, we illustrate how seemingly negligible amounts of functional form misfit, when systematic, can be associated with significant distortions of the score metric in vertical scaling contexts. Our analysis uses two‐ and three‐parameter versions of Samejima's logistic positive exponent model (LPE) as a data generating model. Consistent with prior work, we find LPEs generally provide a better comparative fit to real item response data than traditional IRT models (2PL, 3PL). Further, our simulation results illustrate how 2PL‐ or 3PL‐based vertical scaling in the presence of LPE‐induced misspecification leads to an artificial growth deceleration across grades, consistent with that commonly seen in vertical scaling studies. The results raise further concerns about the use of standard IRT models in measuring growth, even apart from the frequently cited concerns of construct shift/multidimensionality across grades.

Polytomous multilevel testlet models for testlet‐based assessments with complex sampling designs

Applications of standard item response theory models assume local independence of items and persons. This paper presents polytomous multilevel testlet models for dual dependence due to item and person clustering in testlet-based assessments with clustered samples. Simulation and survey data were analysed with a multilevel partial credit testlet model. This model was compared with three alternative models - a testlet partial credit model (PCM), multilevel PCM, and PCM - in terms of model parameter estimation. The results indicated that the deviance information criterion was the fit index that always correctly identified the true multilevel testlet model based on the quantified evidence in model selection, while the Akaike and Bayesian information criteria could not identify the true model. In general, the estimation model and the magnitude of item and person clustering impacted the estimation accuracy of ability parameters, while only the estimation model and the magnitude of item clustering affected the item parameter estimation accuracy. Furthermore, ignoring item clustering effects produced higher total errors in item parameter estimates but did not have much impact on the accuracy of ability parameter estimates, while ignoring person clustering effects yielded higher total errors in ability parameter estimates but did not have much effect on the accuracy of item parameter estimates. When both clustering effects were ignored in the PCM, item and ability parameter estimation accuracy was reduced.

Testlet Response Theory: An Introduction and New Developments

A testlet is a cluster of items that share a common passage, stimulus, or other context, and the dependence between testlet-based item responses that caused by the common stimulus called testlet effect. This article introduced and combed the concepts of testlet, local independence and local dependence, and proposed the concepts of testlet effect in accordance with previous research. Applying standard item response models to testlet responses leaded to many problems such as biased parameter estimation and overestimation of test reliability, hence, testlet response model was introduced to deal with testlet effect better. 6 testlet response models were described in detail including Higher-order testlet model and Multilevel testlet model, and two issues were discussed:(a) the connotation of slope parameter in testlet response model is different from that in standard item response theory which is inappropriate to contrast the two directly;(b) testlet response model belongs to general multidimensional item response model and the main distinction with special multidimensional item response model is the latent factors needed to control to satisfy general local dependence hypothesis.

Generalized IRT Models for Extreme Response Style

Extreme response style (ERS) is a systematic tendency for a person to endorse extreme options (e.g., strongly disagree, strongly agree) on Likert-type or rating-scale items. In this study, we develop a new class of item response theory (IRT) models to account for ERS so that the target latent trait is free from the response style and the tendency of ERS is quantified. Parameters of these new models can be estimated with marginal maximum likelihood estimation methods or Bayesian methods. In this study, we use the freeware program WinBUGS, which implements Bayesian methods. In a series of simulations, we find that the parameters are recovered fairly well; ignoring ERS by fitting standard IRT models resulted in biased estimates, and fitting the new models to data without ERS did little harm. Two empirical examples are provided to illustrate the implications and applications of the new models.

Simultaneous multifactor DIF analysis and detection in Item Response Theory

Item Response Theory (IRT) is a psychometric theory widely used in educational assessment and cognitive psychology to analyse data emerged from answers given to items contained in exams, questionnaires, etc. Standard IRT, however, is based on models which assume that items behave equally to all individuals. This may not be a reasonable assumption, especially when the individuals taking the test have different social and/or cultural backgrounds. Differential Item Functioning (DIF) is an area of IRT which allows an item to be perceived differently by distinct groups, respecting its usual characteristics. DIF hypothesis avoids neglecting items that may behave differently among groups and may also be used to provide important information about differences in the populations involved in the study. In this paper, two integrated Bayesian models for DIF analysis in IRT are proposed and compared. Both models are based on a two component mixture with one component describing DIF and the other accounting for the absence of DIF. Another contribution of this paper is the approach of the simultaneous presence of multiple factors causing DIF. Ideas from ANOVA models are used to characterize different possibilities associated with these factors. The models are also extended to account for explanation and detection in each factor. A simulation study was conducted to assess the model’s capabilities and to compare it against existing alternatives. Special attention has been directed to the conditions required to ensure model identification. An analysis of a Mathematics exam applied nationally to Brazilian elementary school students is made considering two DIF factors: geographical region and type of school. The results highlight the relevance of the proposed methodology to address important issues in educational studying and testing.

Using the GLIMMIX Procedure in SAS 9.3 to Fit a Standard Dichotomous Rasch and Hierarchical 1-PL IRT Model

Although Rasch models have been shown to be a sound methodological approach to develop and validate measures of psychological constructs for more than 50 years, they remain underutilized in psychology and other social sciences. Until recently, one reason for this underutilization was the lack of syntactically simple procedures to fit Rasch and item response theory (IRT) models in general statistical software packages. In this article, the authors demonstrate how to fit the standard dichotomous Rasch model and a dichotomous one-parameter logistic IRT model with nested random effects via the easy-to-use GLIMMIX procedure in SAS 9.3. For comparison purposes, the standard dichotomous Rasch model was also fit using the Rasch specialized software, WINSTEPS 3.68.2. The SAS code used to simulate the data on which the Rasch model was fit is provided to allow replication of estimates. Findings suggest that the GLIMMIX procedure may be a viable option for fitting the standard dichotomous Rasch and dichotomous IRT models.

APPLICATION OF A GENERAL POLYTOMOUS TESTLET MODEL TO THE READING SECTION OF A LARGE‐SCALE ENGLISH LANGUAGE ASSESSMENT

ABSTRACTMany standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet‐based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large‐scale English‐language test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.

Test Equating with Testlets

The research on test equating is very important for fairness of examnation, item banking, teaching quality assessing and computerized adaptive test. Along with the development of research on examination, testlets have appeared in different examnations increasingly, such as reading comprehension, mathematics, map etc. How to equate tests composed of testlets is a problem we are facing. When item response theory (IRT) models are applied in test equating, strong statistical assumptions—local independence (LI)—must be met. However, previous studies have shown that local independence is likely to be violated when testlets are contained in test. Hence, when equating tests composed of testlets, that local dependence is ignored can lead to distortion of equating coefficients using standard IRT model. In order to solve this problem, we use a testlets-based model—2 Parameters Testlet Model (2PTM), which derives from IRT 2 Parameters Logistic Model by adding random-effect parameters associated with each testlet. Local dependence is considered in 2PTM. IRT characteristic curve equating methods and specific procedures for calculating equating coefficients were presented in this paper. In terms of the recovery of estimating the equating coefficients and based on Wilcoxon sign-rank test, a lot of experiments was done using Monte Carlo simulation method. The effectiveness of equating tests containing testlets was investigated under the several conditions, including the accuracy of the estimation of item parameters (AEIP), the number of examinees and the degree of local dependence. The findings of equating tests made up of testlets using 2PTM were compared with standard IRT model—2PLM, which not account for local dependence among items from a common testlet. Results suggest that 2PTM is better than 2PLM in recovery and have significant differences mostly, so 2PTM is suitable for equating tests based testlets. In addition, the findings of using six different equating criterions for 2PTM were also compared with each other. The results showed that, generally speaking, when the value of the coefficient A is between 0.5 and 0.9, the performance of SLcrit is the best, SQRcrit is proper for 0.9A1.5 and Hcrit is proper for 1.5A2.0. The higher AEIP is, the better SQRcrit and SLcrit perform. Hcrit and SQRcrit are proper for large testlet effect. LCcrit, Wcrit and SREcrit are rarely better than others.

Assessing the accuracy and consistency of language proficiency classification under competing measurement models

This article investigates how measurement models and statistical procedures can be applied to estimate the accuracy of proficiency classification in language testing. The paper starts with a concise introduction of four measurement models: the classical test theory (CTT) model, the dichotomous item response theory (IRT) model, the testlet response theory (TRT) model, and the polytomous item response theory (Poly-IRT) model. Following this, two classification procedures are presented: the Livingston and Lewis method for CTT and the Rudner method for the three IRT-based models. The utility of these models and procedures are then evaluated by examining the accuracy of classifying 5000 language test takers from a large-scale language certification examination into two proficiency categories. The most important finding is that the testlet format (multiple questions based on one prompt), which language tests usually rely on, has a great impact on the proficiency classification. All testlets in this study show a strong testlet effect. Hence, the TRT model is recommended for proficiency classification. Using the standard IRT model would inflate the classification accuracy due to the underestimated measurement error. Meanwhile, using the Poly-IRT model would give slightly less accurate classification results. Concerning the CTT model, while its classification accuracy is comparable to that of the TRT, there exists considerable inconsistency between their classification results.

Difficulty, Discrimination, and Information Indices in the Linear Factor Analysis Model for Continuous Item Responses

Spearman's factor-analytic model has been proposed as a unidimensional linear item response theory (IRT) model for continuous item responses. This article first proposes a reexpression of the model that leads to a form similar to that of standard IRT models for binary responses and discusses the item indices of difficulty discrimination and information. Next, the extension to the multiple-factor model is considered, and multidimensional counterparts of the item indices are proposed. The interpretation of the proposed measures and their usefulness in personality and attitude measurement are discussed and illustrated by means of two empirical examples.

Log-Multiplicative Association Models as Item Response Models

Log-multiplicative association (LMA) models, which are special cases of log-linear models, have interpretations in terms of latent continuous variables. Two theoretical derivations of LMA models based on item response theory (IRT) arguments are presented. First, we show that Anderson and colleagues (Anderson & Vermunt, 2000; Anderson & Bockenholt, 2000; Anderson, 2002), who derived LMA models from statistical graphical models, made the equivalent assumptions as Holland (1990) when deriving models for the manifest probabilities of response patterns based on an IRT approach. We also present a second derivation of LMA models where item response functions are specified as functions of rest-scores. These various connections provide insights into the behavior of LMA models as item response models and point out philosophical issues with the use of LMA models as item response models. We show that even for short tests, LMA and standard IRT models yield very similar to nearly identical results when data arise from standard IRT models. Log-multiplicative association models can be used as item response models and do not require numerical integration for estimation.

Comparison of the Performance of Varimax and Promax Rotations: Factor Structure Recovery for Dichotomous Items

Nonlinear factor analysis is a tool commonly used by measurement specialists to identify both the presence and nature of multidimensionality in a set of test items, an important issue given that standard Item Response Theory models assume a unidimensional latent structure. Results from most factor‐analytic algorithms include loading matrices, which are used to link items with factors. Interpretation of the loadings typically occurs after they have been rotated in order to amplify the presence of simple structure. The purpose of this simulation study is to compare the ability of two commonly used methods of rotation, Varimax and Promax, in terms of their ability to correctly link items to factors and to identify the presence of simple structure. Results suggest that the two approaches are equally able to recover the underlying factor structure, regardless of the correlations among the factors, though the oblique method is better able to identify the presence of a “simple structure.” These results suggest that for identifying which items are associated with which factors, either approach is effective, but that for identifying simple structure when it is present, the oblique method is preferable.

A Comparison of Alternative Models for Testlets

When tests are made up of testlets, standard item response theory (IRT) models are often not appropriate due to the local dependence present among items within a common testlet. A testlet-based IRT model has recently been developed to model examinees' responses under such conditions (Bradlow, Wainer, & Wang, 1999). The Bradlow, Wainer, and Wang model introduces separate testlet factors to account for this dependence and applies a common item discrimination parameter to both the general ability and testlet factor. This study investigates several alternative ways of accounting for local dependence that make different assumptions regarding the influence of testlet factors on item performance. The authors implement several Bayesian model selection criteria to compare models using several real test data sets that have a testlet structure. Results suggest that an alternative model in which separate discrimination parameters are applied to the general ability and testlet factors provides a better fit to these data despite its greater complexity. Index terms: item response theory, Bayesian model comparison, Markov chain Monte Carlo, testlets

Adaptive Testing With Regression Trees in the Presence of Multidimensionality

It is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all the new and currently considered computer-based tests. In addition to developing new models, we also need to give attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This article introduces a nonparametric, tree-based algorithm for adaptive testing and shows that it may be superior to conventional, IRT-based adaptive testing in cases where the IRT assumptions are not satisfied. In particular, it shows that the tree-based approach clearly outperformed (one-dimensional) IRT when the pool was strongly two-dimensional.

Bayesian Factor Analysis for Mixed Ordinal and Continuous Responses

Many situations exist in which a latent construct has both ordinal and continuous indicators. This presents a problem for the applied researcher because standard measurement models are not designed to accommodate mixed ordinal and continuous data. I address this problem by formulating a measurement model that is appropriate for such mixed multivariate responses. This model unifies standard normal theory factor analysis and item response theory models for ordinal data. I detail a Markov chain Monte Carlo algorithm for model fitting. I apply the model to cross-national data on political-economic risk and find that the model works well. Software for fitting this model is publicly available in theMCMCpack(Martin and Quinn 2004, “MCMCpack 0.4–8”)Rpackage.

ADAPTIVE TESTING WITHOUT IRT IN THE PRESENCE OF MULTIDIMENSIONALITY

ABSTRACTIt is unrealistic to suppose that standard item response theory (IRT) models will be appropriate for all of the new and currently considered computer‐based tests. In addition to developing new models, we also need to give some attention to the possibility of constructing and analyzing new tests without the aid of strong models. Computerized adaptive testing currently relies heavily on IRT. Alternative, empirically based, nonparametric adaptive testing algorithms exist, but their properties are little known. This paper introduces a nonparametric, tree‐based algorithm for adaptive testing and shows that it may be superior to conventional, IRT‐based adaptive testing in cases where the IRT assumptions are not satisfied. In particular, it shows that the tree‐based approach clearly outperformed (one‐dimensional) IRT when the pool was strongly two‐dimensional.

USING A NEW STATISTICAL MODEL FOR TESTLETS TO SCORE TOEFL

ABSTRACTStandard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimuli (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, and Du, 2000). This parameter, γ, characterizes the amount of local dependence in a testlet.We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters [discrimination (a), difficulty (b), and guessing (c)] with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over‐estimated when conditional independence was incorrectly assumed.

Using a New Statistical Model for Testlets to Score TOEFL

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimulus (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ended examinations in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, & Du, 2000). This parameter, γ characterizes the amount of local dependence in a testlet.We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters (discrimination (a), difficulty (b) and guessing (c)) with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over‐estimated when conditional independence was incorrectly assumed.