Item Parameter Recovery Research Articles

A Mixture Item Response Model for Multiple-Choice Data

A mixture item response model is proposed for investigating individual differences in the selection of response categories in multiple-choice items. The model accounts for local dependence among response categories by assuming that examinees belong to discrete latent classes that have different propensities towards those responses. Varying response category propensities are captured by allowing the category intercept parameters in a nominal response model ( Bock, 1972 ) to assume different values across classes. A Markov Chain Monte Carlo algorithm for the estimation of model parameters and classification of examinees is described. A real-data example illustrates how the model can be used to distinguish examinees that are disproportionately attracted to different types of distractors in a test of English usage. A simulation study evaluates item parameter recovery and classification accuracy in a hypothetical multiple-choice test designed to be diagnostic. Implications for test construction and the use of multiple-choice tests to perform cognitive diagnoses of item response patterns are discussed.

A Comparative Study of On‐line Pretest Item—Calibration/Scaling Methods in Computerized Adaptive Testing

The purpose of this study was to compare and evaluate five on‐line pretest item‐calibration/scaling methods in computerized adaptive testing (CAT): marginal maximum likelihood estimate with one EM cycle (OEM), marginal maximum likelihood estimate with multiple EM cycles (MEM), Stocking's Method A, Stocking's Method B, and BILOG/Prior. The five methods were evaluated in terms of item‐parameter recovery, using three different sample sizes (300, 1000 and 3000). The MEM method appeared to be the best choice among these, because it produced the smallest parameter‐estimation errors for all sample size conditions. MEM and OEM are mathematically similar, although the OEM method produced larger errors. MEM also was preferable to OEM, unless the amount of time involved in iterative computation is a concern. Stocking's Method B also worked very well, but it required anchor items that either would increase test lengths or require larger sample sizes depending on test administration design. Until more appropriate ways of handling sparse data are devised, the BILOG/Prior method may not be a reasonable choice for small sample sizes. Stocking's Method A had the largest weighted total error, as well as a theoretical weakness (i.e., treating estimated ability as true ability); thus, there appeared to be little reason to use it.

An Investigation of the Item Parameter Recovery Characteristics of a Gibbs Sampling Procedure

The item parameter recovery characteristics of a Gibb's sampling method (Albert, 1992) for IRT item parameter estimation were investigated using a simulation study. The item parameters were estimated, under a normal ogive item response function model, using Gibbs sampling and BILOG (Mislevy & Bock, 1989). The item parameter estimates were then equated to the metric of the underlying item parameters for tests with 10, 20, 30, and 50 items, and samples of 30, 60, 120, and 500 examinees. Summary statistics of the equating coefficients showed that Gibbs sampling and BILOG both produced trait scale metrics with units of measurement that were too small, but yielding a proper midpoint of the metric. When expressed in a common metric, the biases of the BILOG estimates of the item discriminations were uniformly smaller and less variable than those from Gibbs sampling. The biases of the item difficulty estimates yielded by the two estimation procedures were small and similar to each other. In addition, the item parameter recovery characteristics were comparable for the largest dataset of 50 items and 500 examinees. However, for short tests and sample sizes the item parameter recovery characteristics of BILOG were superior to those of the Gibbs sampling approach.