Common-item Nonequivalent Groups Design Research Articles

다차원 IRT 모형 적합 혼합형 검사를 위한 공통문항 척도 연계 방법

Under the context of the common-item nonequivalent groups design, item response theory (IRT) scale
 linking methods have been presented for each of the multidimensional extensions of the three-parameter
 logistic (M3PL), graded response (MGR), generalized partial credit (MGPC), and nominal response (MNR)
 models. The common-item multidimensional IRT (MIRT) linking methods that have been proved effective
 for single-format tests include the direct least squares (DLS), item category response function (ICRF), and
 test response function (TRF) methods. This paper extends the three MIRT linking methods to any mixture
 of the exploratory M3PL, MGR, MGPC, and MNR models for mixed-format tests. Each linking method
 extended attempts to find the linking coefficients in the rotation matrix and translation vector that
 constitute a linear transformation between two multidimensional ability scales. Simulations are conducted to
 investigate the performance of the three linking methods under various linking conditions. The results
 indicated that overall, the ICRF method performed best in estimating the rotation matrix and translation
 vector of the linear transformation between the new and true simulation scales. In not a few conditions, the
 DLS method resulted in the second best linking solutions. The TRF method tended to estimate the rotation
 matrix well but the translation vector poorly and thus showed the worst performance among the linking
 methods in some conditions.

Bootstrap Equating Errors for the Common-item Nonequivalent Groups Design: A Comparison of Rasch Equating Methods

The nonequivalent groups with anchor test (NEAT) equating design are traditionally based on using a single anchor to adjust for differences in test difficulty which is critical to equating test forms in most large-scale testing programs. When tests differ somewhat in content and length, methods based on the item response theory (IRT) model leads to greater stability of equating results. The current study compared standard errors, bias, and root mean square errors using four Rasch IRT equating methods for the nonequivalent groups with anchor test design. The sizes of the equating anchor were employed in all four different Rasch equating methods to investigate how different anchor sizes may impact the test accuracy of the tests by conducting a simulation study.

Effect of Item Parameter Drift in Mixed Format Common Items on Test Equating

The aim of the study was to examine the common items in the mixed format (e.g., multiple-choices and essay items) contain parameter drifts in the test equating processes performed with the common item non-equivalent groups design. In this study, which was carried out using Monte Carlo simulation with a fully crossed design, the factors of test length (30 and 50), sample size (1000 and 3000), common item ratio (30 and 40%), ratio of items with item parameter drift (IPD) in common items (20 and 30%), location of common items in tests (at the beginning, randomly distributed, and at the end) and IPD size in multiple-choice items (low [0.2] and high [1.0]) were studied. Four test forms were created, and two test forms do not contain parameter drifts. After the parameter drift was performed on the first of the other two test forms, the parameter drift was again performed on the second test form. Test equating results were compared using the root mean squared error (RMSE) value. As a result of the research, ratio of items with IPD in common items, IPD size in multiple-choice items, common item ratio, sample size and test length on equating errors were found to be significant.

Investigation of the Missing Data Imputation Methods on Characteristic Curve Transformation Methods Used in Test Equating

In this research, the aim was to evaluate the effect of zero imputation and multiple imputation missing data handling methods on item response theory (IRT) based test equating methods under different conditions. Data in this study was obtained from the administration of the TIMSS 2019 eighth-grade science test. Data sets were formed by randomly selecting a sample of 1000 students with full data from booklets 7 and 8. By deleting data under a completely random missing data mechanism within the scope of common-item nonequivalent groups (CINEG) design, four different data sets were obtained with the missing data rates of 10% or 20% in the new test or in both tests. The missing da ta problem was solved by using zero imputation and multiple imputation methods from these data sets. In this way, 8 different data sets were formed. Then, scaling transformation was performed by using characteristic curve transformation methods (Haebara, Stocking-Lord). Test equating results were reported in terms of observed scores. The root mean square error (RMSE) was used as the evaluation criterion to determine the error involved in test equating. As a result, it was determined that in the case of 10% missing data in both tests, generally lower RMSE values were obtained. It was observed that the multiple imputation method, one of the methods for handling missing data, was the method that produced RMSE values that were both the lowest and closer to the full data set as a reference value compared to the zero-imputation method. In addition, it was determined that, when compared to the Haebara method, Stocking-Lord method, one of the characteristic curve transformation methods, produced lower RMSE values and these values were closer to the full data set, which was taken as a reference value.

Item Response Theory True Score Equating for the Bifactor Model Under the Common-Item Nonequivalent Groups Design.

Applying item response theory (IRT) true score equating to multidimensional IRT models is not straightforward due to the one-to-many relationship between a true score and latent variables. Under the common-item nonequivalent groups design, the purpose of the current study was to introduce two IRT true score equating procedures that adopted different dimension reduction strategies for the bifactor model. The first procedure, which was referred to as the integration procedure, linked the latent variable scales for the bifactor model and integrated out the specific factors from the item response function of the bifactor model. Then, IRT true score equating was applied to the marginalized bifactor model. The second procedure, which was referred to as the PIRT-based procedure, projected the specific dimensions onto the general dimension to obtain a locally dependent unidimensional IRT (UIRT) model and linked the scales of the UIRT model, followed by the application of IRT true score equating to the locally dependent UIRT model. Equating results obtained with the two equating procedures along with those obtained with the unidimensional three-parameter logistic (3PL) model were compared using both simulated and real data. In general, the integration and PIRT-based procedures provided equating results that were not practically different. Furthermore, the equating results produced by the two bifactor-based procedures became more accurate than the results returned by the 3PL model as tests became more multidimensional.

Comparison of IRT observed score equating methods for CR items under the common-item nonequivalent groups design

이 연구는 공통문항 비동등집단 설계에서 구성형 문항의 IRT 관찰점수 동등화의 정확성을 알아보기 위한 모의실험 연구로, 다음 세 가지 조건을 사용하여 동등화의 정확성을 확인하였다. 첫째, 구성형 문항만을 공통문항으로 사용하여 일차원 IRT 모형을 적용하였다. 둘째, 구성형 문항과 외부(external) 선택형 문항을 공통문항으로 사용하여 일차원 IRT 모형을 적용하였다. 셋째, 구성형 문항과 외부 선택형 문항을 공통문항으로 사용하여 다차원 IRT 모형을 적용하였다. 연구 결과, 다음의 제한된 조건에서 공통문항에 구성형 문항만을 사용하는 것보다 선택형 문항을 함께 사용했을 때 동등화 결과가 더 정확하였다. 즉, 1) 다차원 IRT 모형을 적용하였을 때, 2) 표본 크기가 클 경우, 그리고 3) 선택형 문항에서 집단 간의 능력 차이가 작을 때 구성형 문항의 동등화의 정확성이 향상되었다. 또한, 일반적으로 문항의 수가 많고 점수 범주의 수가 적을 때 구성형 문항의 동등화 결과가 정확하였다.

Comparing Performance of Different Equating Methods in Presence and Absence of DIF Items in Anchor Test

This study aims to compare the performance of different small sample equating methods in the presence and absence of differential item functioning (DIF) in common items. In this research, Tucker linear equating, Levine linear equating, unsmoothed and presmoothed (C=4) chained equipercentile equating, and simplified circle arc equating methods were considered. The data used in this study is 8th-grade mathematics test item responses which obtained from Trends in International Mathematics and Science Study (TIMSS) 2015 Turkey sample. Item responses from Booklet-1 (N=199) and Booklet-14 (N=224) are chosen for this study. Data analyses were completed in four steps. In the first step, assumptions for DIF detection and test equating methods were checked. In the second step, DIF analyses were conducted with Mantel Haenszel and logistic regression methods. In the third step, Booklet 1 was chosen as base form and Booklet 14 chosen as a new form, then test equating was conducted under common item nonequivalent groups design. Test equating was done in two phases: the presence and absence of DIF items in the common items. Equating results were evaluated based on standard error of equating (se), bias and RMSE indexes. DIF analyses showed that there were two sizeable DIF items in anchor test. Equating results showed that performances of equating methods are similar in presence and absence of DIF items from anchor test and there is no notable change in se, bias and RMSE values. While the circle arc equating method outperformed other equating methods based on se, 4-moment presmoothed chained equipercentile equating method outperformed other methods based on bias and RMSE evaluation criteria.

Common-Item Scale-Linking Methods for the Bi-factor Graded Response Model in MIRT

다차원 문항반응이론에서, 이요인 등급반응(BGR) 모형은 순서화된 다분 문항반응을 검사 문항 전체에 연관된 일반 요인과 각 문항집단에 독립적인 특수 요인으로 분석하기 위해 사용되어 왔다. 본 연구의 목적은 BGR 모형을 위한 공통-문항 기반 척도연계 방법들을 제시하고 모의실험을 통해 이 방법들의 특성과 상대적 기능을 탐구하는 데 있다. 제시되는 척도연계 방법들은 최소제곱(LSQ) 방법, 평균-최소제곱(MLS) 방법, 문항유목반응함수(IRF) 방법, 그리고 검사반응함수(TRF) 방법이다. 각 척도연계 방법은, 모든 요인들 간의 독립성 가정에 따라, 각 차원에 대해 임의의 두 능력척도를 연계하기 위한 팽창 계수와 이동 계수를 추정한다. 네 가지 척도연계 방법들의 기능을 평가하기 위해 피험자 집단 간 능력의 비동등성 수준, 표본의 크기, 공통 문항의 수 등을 요인으로 포함하는 모의실험을 수행하였다. 주요 결과로, 네 가지 척도연계 방법은 고려된 모의실험 조건 모두에서 팽창 계수를 적절히 추정하였다. 그러나 이동 계수를 추정함에 있어서, IRF, LSQ 및 MLS 방법은 적절히 기능한 반면 TRF 방법은 그렇지 못하였다. 척도연계를 통한 문항모수의 복원에 있어서, 전반적으로 IRF 방법이 가장 우수하였으나, 표본의 크기가 작거나 공통 문항의 수가 작을 경우 LSQ 방법과 MLS 방법 또한 우수한 수행을 보였다.In applications of multidimensional item response theory (MIRT), the bi-factor graded response (BGR) model can be used to analyze the ordered response data of bi-factor latent structures. The purpose of this study is to present four scale-linking methods for the BGR model under the context of the common-item nonequivalent groups design and investigate their characteristics and performances through computer simulations. The four scale-linking methods presented are the least squares (LSQ), mean-least squares (MLS), item-category response function (IRF), and test response function (TRF) methods. The four methods estimate the dilation and translation linking coefficients with the assumption of independence among all general and group factors. Computer simulations that included three factors of (a) nonequivalence level between two examinee groups, (b) sample size, and (c) number of common items between linked test forms were conducted to examinee the characteristics and relative performances of the four scale-linking methods. Overall, the four methods properly estimated the dilation coefficients. In estimation of the translation coefficients, however, the IRF, LSQ, and MLS methods functioned properly but the TRF method did not. For the recovery of item parameters, the IRF method performed best in mose cases, but the LSQ and MLS methods also performed well in the cases of small sample size and small number of common items.

HOW TO TREAT OMITTED RESPONSES IN RASCH MODEL-BASED EQUATING

This study investigated the impact of the coding scheme on IRT-based true score equating under a common-item nonequivalent groups design. Two different coding schemes under investigation were carried out by assigning either a zero or a blank to a missing item response in the equating data. The investigation involved a comparison study using actual large scale data and then Monte Carlo simulations for a systematic inspection on the topic. The recommendations on the basis of the findings of the study were made to treat omitted responses as not-administered rather than as wrong, and use a large sample size to ensure the accuracy of the screening tools such as the displacement index and the robust-z statistic during equating.

Physical Reasoning Ability Development of High School Students

This study has purpose to reveal: (1) the characteristics of physics subject test of high school students of grade X and XI, (2) the equality of vertical test of physics subject of high school students of grade X and XI, (3) the development of the ability of high school students of grade X to grade XI physics subject. The data which was in the form of scores were then processed and synchronized by using a common item non-equivalent groups design by the method of equalizing of mean and sigma based on the Rasch model. The results of this study are: (1) the validity of the physics subjects test of grade X is 0.92 and grade XI is 0.96 (2) the reliability tests is 0.928 for grade X and 0.926 for grade XI; (3) the results of the vertical equalization based on the approach of Rasch model showed the conversion equation for grade X to grade XI was b * = bx - 0.36; (4) there is a development of the ability of grade X students to grade XI students or the ability of grade XI students is higher than the ability of grade X students.

Test Eşitlemede Çok Boyutluluğun Eş Zamanlı ve Ayrı Kalibrasyona Etkisi

The aim of this study is to investigate the effects of multidimensionality on equating results which obtained from separate and concurrent calibration methods. The study was conducted with using simulated data. In the scope of research, totally 40 simulation conditions [5 (degree of multidimensionality: 0.90, 0.75, 0.50, 0.25, and 0.00) x 2 calibration methods (separate and concurrent) x 2 (scale transformation methods: Stocking-Lord and Haebara) x 2 (test equating methods: IRT true score equating and observed score equating)] were examined. Multidimensionality was constructed as assuming the two test forms measuring Ɵ1 and Ɵ2 abilities. While the simulation condition which has correlation between abilities 0.90 represents weak multidimensional case, the correlation between abilities 0.00 represents the severe multidimensional case. Tests were equated under common-item non-equivalent groups design. Equating results were evaluated by using bias, standard deviation and RMSE evaluation criteria. The results showed that, generally under all conditions equating results provided from concurrent calibration more biased and had higher RMSE values than equating results provided by separate calibration. Based on standard deviation criteria, when the degree of multidimensionality was low, equating results which got from concurrent calibration and separate calibration with Stocking-Lord or Haebara scale transformation methods showed similar performance but when the degree of multidimensionality was severe equating results which had lowest random error were provided by concurrent calibration.

Software Review of IRTEQ, STUIRT, and POLYEQUATE for Item Response Theory Scale Linking and Equating

ABSTRACTThis article reviews several software programs designed to conduct item response theory (IRT) scale linking and equating. The programs reviewed include IRTEQ, STUIRT, and POLYEQUATE. Features and functionalities of each program are discussed and an example analysis using the common-item non-equivalent groups design in IRTEQ is provided.

Çok Boyutlu Eşitleme Yöntemlerinin Eşdeğer Olmayan Gruplarda Ortak Madde Deseni için Performanslarının İncelenmesi

Bu çalı ş mada çok boyutlu veri için adapte edilen Stocking-Lord, ortlama/ortlama ve ortalama/sigma eşitleme yöntemlerinin performansları eşdeğer olmayan gruplarda ortak madde deseni göz önüne alınarak incelenmiştir. Bu üç eşitleme yöntemin performansları orneklem büyüklüğünün, yetenek dağılımının, boyutlar arasındaki korelasyon değerlerinin ve testteki ortak madde yüzdelerinin kombinasyonlari altinda araştırılmıştır. Madde parametre kestirimlerinin değerlendirilmesinde RMSE ve yanlılık değerleri kullanılmıştır. Bu çalışmada çoğu koşul icin hem madde ayırt edicilik parametre kestirimlerinde hem de madde güçlük parametre kestirimlerinde Stocking-Lord yönteminin diger iki yönteme gore daha kucuk RMSE ve yanlılık değerleri verdiği bulunmuştur.

Investigating Separate and Concurrent Approaches for Item Parameter Drift in 3PL Item Response Theory Equating

This study examines separate and concurrent approaches to combine the detection of item parameter drift (IPD) and the estimation of scale transformation coefficients in the context of the common item nonequivalent groups design with the three-parameter item response theory equating. The study uses real and synthetic data sets to compare the two approaches based on IPD flagging rates, type I error and power rates, and recovery of scale transformation coefficients. Results indicate that the two approaches render similar outcomes with stable anchor sets. However, they can produce dissimilar results with unstable anchor sets because of differences in the performance of their IPD components. Further, the findings of this study caution about working backward from equated cut scores to motivate the selection of an anchor set.

The Impact of Test Dimensionality, Common-Item Set Format, and Scale Linking Methods on Mixed-Format Test Equating

The purpose of this study was to examine the impact of dimensionality, common-item set format, and different scale linking methods on preserving equity property with mixed-format test equating. Item response theory (IRT) true-score equating (TSE) and IRT observed-score equating (OSE) methods were used under common-item nonequivalent groups design. The equity property was evaluated based on firstorder equity (FOE) and second-order equity (SOE) properties. A simulation study was conducted based on actual item parameter estimates obtained from the TIMSS 2011 8th grade mathematics assessment. The results showed that: (i) The FOE and SOE properties were best preserved under the unidimensional condition, were poorly preserved when the degree of multidimensionality was severe. (ii) The TSE and OSE results, which were provided by using a mixed-format common-item set, preserved FOE better compared to equating results, which provided only a multiple-choice common item set. (iii) Under the unidimensional and multidimensional test structure, characteristic curve methods performed significantly better than moment scale linking methods in terms of preserving FOE and SOE properties.

Differentiation among Equated Scores for General Abilities Test Forms Using Classical Equating Methods with Common-Item Non-Equivalent Groups Design

Differentiation among Equated Scores for General Abilities Test Forms Using Classical Equating Methods with Common-Item Non-Equivalent Groups Design

New Robust Scale Transformation Methods in the Presence of Outlying Common Items.

Common items play an important role in item response theory (IRT) true score equating under the common-item nonequivalent groups design. Biased item parameter estimates due to common item outliers can lead to large errors in equated scores. Current methods used to screen for common item outliers mainly focus on the detection and elimination of those items, which may lead to inadequate content representation for the common items. To reduce the impact of inconsistency in item parameter estimates while maintaining content representativeness, the authors propose two robust scale transformation methods based on two weighting methods: the Area-Weighted method and the Least Absolute Values (LAV) method. Results from two simulation studies indicate that these robust scale transformation methods performed as well as the Stocking-Lord method in the absence of common item outliers and, more importantly, outperformed the Stocking-Lord method when a single outlying common item was simulated.

Using a Linear Regression Method to Detect Outliers in IRT Common Item Equating

Common test items play an important role in equating alternate test forms under the common item nonequivalent groups design. When the item response theory (IRT) method is applied in equating, inconsistent item parameter estimates among common items can lead to large bias in equated scores. It is prudent to evaluate inconsistency in parameter estimates of common items before conducting IRT equating. The evaluation of inconsistency in parameter estimates is typically achieved through detecting outliers in the common item set. In this study, a linear regression method is proposed as a detection method. The newly proposed method was compared with a traditional method in various conditions. The results of this study confirmed the necessity of detecting and removing outlying common items. The results also show that the newly proposed method performed better than did the traditional method in most conditions.

A Graphical Approach to Evaluating Equating Using Test Characteristic Curves

An essential concern in the application of any equating procedure is determining whether tests can be considered equated after the tests have been placed onto a common scale. This article clarifies one equating criterion, the first-order equity property of equating, and develops a new method for evaluating equating that is linked to this criterion. The new approach involves graphically examining the difference in test characteristic curves (TCCs), calculating the maximum absolute difference between the TCCs, and comparing the differences in TCCs to the difference that matters (DTM). The new approach is applied to evaluate the equating of the Multistate Bar Exam (MBE) for six different item response theory (IRT) scaling approaches in the common item nonequivalent groups design. The new approach is also contrasted with Tong and Kolen’s (2005) index for assessing first-order equity. The empirical investigations indicated that the Stocking-Lord and fixed-parameter equating methods appear to perform the best for equating the MBE and that the use of concurrent calibration is not desirable.

A New Concurrent Calibration Method for Nonequivalent Group Design under Nonrandom Assignment

This study proposes a new item parameter linking method for the common-item nonequivalent groups design in item response theory (IRT). Previous studies assumed that examinees are randomly assigned to either test form. However, examinees can frequently select their own test forms and tests often differ according to examinees’ abilities. In such cases, concurrent calibration or multiple group IRT modeling without modeling test form selection behavior can yield severely biased results. We proposed a model wherein test form selection behavior depends on test scores and used a Monte Carlo expectation maximization (MCEM) algorithm. This method provided adequate estimates of testing parameters.