Fixed Item Parameter Research Articles

Bias and Linking Error in Fixed Item Parameter Calibration

The two-parameter logistic (2PL) item response theory (IRT) model is frequently applied to analyze group differences for multivariate binary random variables. The item parameters in the 2PL model are frequently fixed when estimating the mean and the standard deviation for a group of interest. This method is also called fixed item parameter calibration (FIPC). In this article, the bias and the linking error of the FIPC approach are analytically derived in the presence of random uniform differential item functioning (DIF). The adequacy of the analytical findings was validated in a simulation study. It turned out that the extent of the bias and the variance in distribution parameters increases with increasing variance of random DIF effects.

The irtQ R package: a user-friendly tool for item response theory-based test data analysis and calibration.

Computerized adaptive testing (CAT) has become a widely adopted test design for high-stakes licensing and certification exams, particularly in the health professions in the United States, due to its ability to tailor test difficulty in real time, reducing testing time while providing precise ability estimates. A key component of CAT is item response theory (IRT), which facilitates the dynamic selection of items based on examinees' ability levels during a test. Accurate estimation of item and ability parameters is essential for successful CAT implementation, necessitating convenient and reliable software to ensure precise parameter estimation. This paper introduces the irtQ R package, which simplifies IRT-based analysis and item calibration under unidimensional IRT models. While it does not directly simulate CAT, it provides essential tools to support CAT development, including parameter estimation using marginal maximum likelihood estimation via the expectation-maximization algorithm, pretest item calibration through fixed item parameter calibration and fixed ability parameter calibration methods, and examinee ability estimation. The package also enables users to compute item and test characteristic curves and information functions necessary for evaluating the psychometric properties of a test. This paper illustrates the key features of the irtQ package through examples using simulated datasets, demonstrating its utility in IRT applications such as test data analysis and ability scoring. By providing a user-friendly environment for IRT analysis, irtQ significantly enhances the capacity for efficient adaptive testing research and operations. Finally, the paper highlights additional core functionalities of irtQ, emphasizing its broader applicability to the development and operation of IRT-based assessments.

Expanding NAEP and TIMSS Analysis to Include Additional Variables or a New Scoring Model Using the R Package Dire

The R packages Dire and EdSurvey allow analysts to make a conditioning model with new variables and then draw new plausible values. This is important because results for a variable not in the conditioning model are biased. For regression-type analyses, users can also use direct estimation to estimate parameters without generating new plausible values. Dire is distinct from other available software in R in that it requires fixed item parameters and simplifies calculation of high-dimensional integrals necessary to calculate composite or subscales. When used with EdSurvey, it is very easy to use published item parameters to estimate a new conditioning model. We show the theory behind the methods in Dire and a coding example where we perform an analysis that includes simple process data variables. Because the process data is not used in the conditioning model, the estimator is biased if a new conditioning model is not added with Dire.

Linking Error in the 2PL Model

The two-parameter logistic (2PL) item response model is likely the most frequently applied item response model for analyzing dichotomous data. Linking errors quantify the variability in means or standard deviations due to the choice of items. Previous research presented analytical work for linking errors in the one-parameter logistic model. In this article, we present linking errors for the 2PL model using the general theory of M-estimation. Linking errors are derived in the case of log-mean-mean linking for linking two groups. The performance of the newly proposed formulas is evaluated in a simulation study. Furthermore, the linking error estimation in the 2PL model is also treated in more complex settings, such as chain linking, trend estimation, fixed item parameter calibration, and concurrent calibration.

Characterizing Sampling Variability for Item Response Theory Scale Scores in a Fixed-Parameter Calibrated Projection Design.

A common practice of linking uses estimated item parameters to calculate projected scores. This procedure fails to account for the carry-over sampling variability. Neglecting sampling variability could consequently lead to understated uncertainty for Item Response Theory (IRT) scale scores. To address the issue, we apply a Multiple Imputation (MI) approach to adjust the Posterior Standard Deviations of IRT scale scores. The MI procedure involves drawing multiple sets of plausible values from an approximate sampling distribution of the estimated item parameters. When two scales to be linked were previously calibrated, item parameters can be fixed at their original published scales, and the latent variable means and covariances of the two scales can then be estimated conditional on the fixed item parameters. The conditional estimation procedure is a special case of Restricted Recalibration (RR), in which the asymptotic sampling distribution of estimated parameters follows from the general theory of pseudo Maximum Likelihood (ML) estimation. We evaluate the combination of RR and MI by a simulation study to examine the impact of carry-over sampling variability under various simulation conditions. We also illustrate how to apply the proposed method to real data by revisiting Thissen et al. (2015).

Detecting Noneffortful Responses Based on a Residual Method Using an Iterative Purification Process

The prevalence and serious consequences of noneffortful responses from unmotivated examinees are well-known in educational measurement. In this study, we propose to apply an iterative purification process based on a response time residual method with fixed item parameter estimates to detect noneffortful responses. The proposed method is compared with the traditional residual method and noniterative method with fixed item parameters in two simulation studies in terms of noneffort detection accuracy and parameter recovery. The results show that when severity of noneffort is high, the proposed method leads to a much higher true positive rate with a small increase of false discovery rate. In addition, parameter estimation is significantly improved by the strategies of fixing item parameters and iteratively cleansing. These results suggest that the proposed method is a potential solution to reduce the impact of data contamination due to severe low test-taking effort and to obtain more accurate parameter estimates. An empirical study is also conducted to show the differences in the detection rate and parameter estimates among different approaches.

The Benefits of Fixed Item Parameter Calibration for Parameter Accuracy in Small Sample Situations in Large‐Scale Assessments

AbstractLarge‐scale assessments such as the Programme for International Student Assessment (PISA) have field trials where new survey features are tested for utility in the main survey. Because of resource constraints, there is a trade‐off between how much of the sample can be used to test new survey features and how much can be used for the initial item response theory (IRT) scaling. Utilizing real assessment data of the PISA 2015 Science assessment, this article demonstrates that using fixed item parameter calibration (FIPC) in the field trial yields stable item parameter estimates in the initial IRT scaling for samples as small as n = 250 per country. Moreover, the results indicate that for the recovery of the county‐specific latent trait distributions, the estimates of the trend items (i.e., the information introduced into the calibration) are crucial. Thus, concerning the country‐level sample size of n = 1,950 currently used in the PISA field trial, FIPC is useful for increasing the number of survey features that can be examined during the field trial without the need to increase the total sample size. This enables international large‐scale assessments such as PISA to keep up with state‐of‐the‐art developments regarding assessment frameworks, psychometric models, and delivery platform capabilities.

A comparison of linear and equipercentile equating and IRT equating with FIPC across multidimensional test forms for non-equivalent groups

The aim of this study was to compare linear equating (Levine and Tucker), equipercentile equating, and true and observed score IRT equating with fixed item parameter calibration methods when applied to non-equivalent groups taking forms which have various multidimensional structures: equal or unequal total test difficulty and similar or dissimilar difficulty within dimensions across forms. This situation may be common for large-scale test forms that are composed of multiple sub-content areas and are being administered to examinees of mixed abilities at different times. Within the specifications of this study, when forms differ in total difficulty but the difference in difficulty within dimensions is consistent across forms, the linear methods may be preferred. When forms differ in average item difficulty within dimensions, regardless of equal or unequal total test difficulty, the IRT method with fixed item parameter calibration is favoured when data are correlated.

A comparison of linear and equipercentile equating and IRT equating with FIPC across multidimensional test forms for non-equivalent groups

The aim of this study was to compare linear equating (Levine and Tucker), equipercentile equating, and true and observed score IRT equating with fixed item parameter calibration methods when applied to non-equivalent groups taking forms which have various multidimensional structures: equal or unequal total test difficulty and similar or dissimilar difficulty within dimensions across forms. This situation may be common for large-scale test forms that are composed of multiple sub-content areas and are being administered to examinees of mixed abilities at different times. Within the specifications of this study, when forms differ in total difficulty but the difference in difficulty within dimensions is consistent across forms, the linear methods may be preferred. When forms differ in average item difficulty within dimensions, regardless of equal or unequal total test difficulty, the IRT method with fixed item parameter calibration is favoured when data are correlated.

Visual acuity and contrast sensitivity are two important factors affecting vision-related quality of life in advanced age-related macular degeneration.

PurposeVision loss from age-related macular degeneration (AMD) has a profound effect on vision-related quality of life (VRQoL). The pupose of this study is to identify clinical factors associated with VRQoL using the Rasch- calibrated NEI VFQ-25 scales in bilateral advanced AMD patients.MethodsWe retrospectively reviewed 47 patients (mean age 83.2 years) with bilateral advanced AMD. Clinical assessment included age, gender, type of AMD, high contrast visual acuity (VA), history of medical conditions, contrast sensitivity (CS), central visual field loss, report of Charles Bonnet Syndrome, current treatment for AMD and Rasch-calibrated NEI VFQ-25 visual function and socioemotional function scales. The NEI VFQ visual function scale includes items of general vision, peripheral vision, distance vision and near vision-related activity while the socioemotional function scale includes items of vision related-social functioning, role difficulties, dependency, and mental health. Multiple regression analysis (structural regression model) was performed using fixed item parameters obtained from the one-parameter item response theory model.ResultsMultivariate analysis showed that high contrast VA and CS were two factors influencing VRQoL visual function scale (β = -0.25, 95% CI-0.37 to -0.12, p<0.001 and β = 0.35, 95% CI 0.25 to 0.46, p<0.001) and socioemontional functioning scale (β = -0.2, 95% CI -0.37 to -0.03, p = 0.023, and β = 0.3, 95% CI 0.18 to 0.43, p = 0.001). Central visual field loss was not assoicated with either VRQoL visual or socioemontional functioning scale (β = -0.08, 95% CI-0.28 to 0.12,p = 0.44 and β = -0.09, 95% CI -0.03 to 0.16, p = 0.50, respectively).ConclusionIn patients with vision impairment secondary to bilateral advanced AMD, high contrast VA and CS are two important factors affecting VRQoL.

FIPC Linking Across Multidimensional Test Forms: Effects of Confounding Difficulty within Dimensions

This study investigated the effects of linking potentially multidimensional test forms using the fixed item parameter calibration. Forms had equal or unequal total test difficulty with and without confounding difficulty. The mean square errors and bias of estimated item and ability parameters were compared across the various confounding tests. The estimated discrimination parameters were influenced by the levels of correlation between dimensions. The mean square errors (MSEs) of the average of the true discrimination parameters with the estimated value were smallest when the correlation equaled 0; however, the MSEs of the multidimensional discrimination parameter were smallest when the correlation was larger than 0. The estimated difficulty parameters were highly affected by different amount of confounding difficulty within dimensions. Furthermore, the MSEs of the average of the true ability parameters on the first and second dimensions with the estimated ability were smaller than those from the ability parameter on each dimension for all conditions. The pattern varied according to the number of common items, and the measures of MSE and squared bias were relatively consistent across forms at the same level of correlation, except for the condition where the correlation was 0 and the number of common items was 8.

A Note on the Item Information Function of the Four-Parameter Logistic Model

This article focuses on four-parameter logistic (4PL) model as an extension of the usual three-parameter logistic (3PL) model with an upper asymptote possibly different from 1. For a given item with fixed item parameters, Lord derived the value of the latent ability level that maximizes the item information function under the 3PL model. The purpose of this article is to extend this result to the 4PL model. A generic and algebraic method is developed for that purpose. The result is practically illustrated by an example and several potential applications of this result are outlined.

Linking item parameters to a base scale

This paper compares three methods of item calibration—concurrent calibration, separate calibration with linking, and fixed item parameter calibration—that are frequently used for linking item parameters to a base scale. Concurrent and separate calibrations were implemented using BILOG-MG. The Stocking and Lord in Appl Psychol Measure 7:201–210, (1983) characteristic curve method of parameter linking was used in conjunction with separate calibration. The fixed item parameter calibration (FIPC) method was implemented using both BILOG-MG and PARSCALE because the method is carried out differently by the two programs. Both programs use multiple EM cycles, but BILOG-MG does not update the prior ability distribution during FIPC calibration, whereas PARSCALE updates the prior ability distribution multiple times. The methods were compared using simulations based on actual testing program data, and results were evaluated in terms of recovery of the underlying ability distributions, the item characteristic curves, and the test characteristic curves. Factors manipulated in the simulations were sample size, ability distributions, and numbers of common (or fixed) items. The results for concurrent calibration and separate calibration with linking were comparable, and both methods showed good recovery results for all conditions. Between the two fixed item parameter calibration procedures, only the appropriate use of PARSCALE consistently provided item parameter linking results similar to those of the other two methods.

A Comparative Study of IRT Fixed Parameter Calibration Methods

This article provides technical descriptions of five fixed parameter calibration (FPC) methods, which were based on marginal maximum likelihood estimation via the EM algorithm, and evaluates them through simulation. The five FPC methods described are distinguished from each other by how many times they update the prior ability distribution and by how many EM cycles they use. Specifically, the five FPC methods included no prior weights updating and one EM cycle (NWU‐OEM) or multiple EM cycles (NWU‐MEM), one prior weights updating and one EM cycle (OWU‐OEM) or multiple EM cycles (OWU‐MEM), and multiple weights updating and multiple EM cycles (MWU‐MEM) methods. All the five FPC methods were evaluated in terms of recovery of the underlying ability distribution and item parameters. An important factor in the simulation was three different ability (normal) distributions—N(0, 1), N(0.5, 1.22), and N(1, 1.42)—for FPC groups, with the fixed item parameters obtained with a reference N(0, 1) group. Only the MWU‐MEM method appeared to perform properly under all the three distributions. Under the N(0, 1) distribution, the NWU‐MEM and OWU‐MEM methods also appeared to perform properly. Under the N(0.5, 1.22), and N(1, 1.42) distributions, however, the four methods other than the MWU‐MEM method resulted in some or severe under‐estimation in the recovery.

A UNIFIED APPROACH TO IRT SCALE LINKING AND SCALE TRANSFORMATIONS

ABSTRACTThis paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.

Stochastic order in dichotomous item response models for fixed, adaptive, and multidimensional tests

Dichotomous IRT models can be viewed as families of stochastically ordered distributions of responses to test items. This paper explores several properties of such distributions. In particular, it is examined under what conditions stochastic order in families of conditional distributions is transferred to their inverse distributions, from two families of related distributions to a third family, or from multivariate conditional distributions to a marginal distribution. The main results are formulated as a series of theorems and corollaries which apply to dichotomous IRT models. One part of the results holds for unidimensional models with fixed item parameters. The other part holds for models with random item parameters as used, for example, in adaptive testing or for tests with multidimensional abilities.