Common-item Nonequivalent Groups Research Articles

다차원 IRT 모형 적합 혼합형 검사를 위한 공통문항 척도 연계 방법

Under the context of the common-item nonequivalent groups design, item response theory (IRT) scale
 linking methods have been presented for each of the multidimensional extensions of the three-parameter
 logistic (M3PL), graded response (MGR), generalized partial credit (MGPC), and nominal response (MNR)
 models. The common-item multidimensional IRT (MIRT) linking methods that have been proved effective
 for single-format tests include the direct least squares (DLS), item category response function (ICRF), and
 test response function (TRF) methods. This paper extends the three MIRT linking methods to any mixture
 of the exploratory M3PL, MGR, MGPC, and MNR models for mixed-format tests. Each linking method
 extended attempts to find the linking coefficients in the rotation matrix and translation vector that
 constitute a linear transformation between two multidimensional ability scales. Simulations are conducted to
 investigate the performance of the three linking methods under various linking conditions. The results
 indicated that overall, the ICRF method performed best in estimating the rotation matrix and translation
 vector of the linear transformation between the new and true simulation scales. In not a few conditions, the
 DLS method resulted in the second best linking solutions. The TRF method tended to estimate the rotation
 matrix well but the translation vector poorly and thus showed the worst performance among the linking
 methods in some conditions.

한국판 정서·행동평가시스템(K·BASC-3), 교사보고형 검사-아동용(TRS-C)의 타당화 연구

The purpose of this study was to evaluate the validity of the Korean Behavior Assessment System for Children 3rd Edition, Teacher Rating Scales-Child Form (K·BASC-3, TRS-C). The 156-item K·BASC-3, TRS-C was administered to 318 teachers of children. Equating using common item non-equivalent group designs, a general norm was made for a total of 954. The generalized partial credit model based on item response theory analysis was used to conduct internal validity. Additionally, as a measure of external validity, concurrent validity was tested using the Teacher’s Report Form for Ages 6-18 (TRF 6-18). Results from the analysis indicated that items included in clinical subscale provided appropriate information at the ability levels higher than the mean. Also, the results revealed that the most item difficulty and discrimination were appropriate. The items that yielded less than desirable discrimination scores correspond to those measuring symptoms related to the DSM diagnostic categories or protective and risk factors for childhood emotional/behavioral disorders. The anxiety, hyperactivity, and attention problems subscales of the K·BASC-3, TRS-C demonstrated strong concurrent validity statistics with the related subscales of the TRF 6-18. Other relevant findings, practical implications regarding the use of the K·BASC-3, TRS-C, strengths and weaknesses of the current study and suggestions for future research are included in the discussion.

Evaluating Equating Methods for Varying Levels of Form Difference

Equating is a statistical procedure used to adjust for the difference in form difficulty such that scores on those forms can be used and interpreted comparably. In practice, however, equating methods are often implemented without considering the extent to which two forms differ in difficulty. The study aims to examine the effect of the magnitude of a form difficulty difference on equating results under random group (RG) and common-item nonequivalent group (CINEG) designs. Specifically, this study evaluates the performance of six equating methods under a set of simulation conditions including varying levels of form difference. Results revealed that, under the RG design, mean equating was proven to be the most accurate method when there is no or small form difference, whereas equipercentile is the most accurate method when the difficulty difference is medium or large. Under the CINEG design, Tucker Linear was found to be the most accurate method when the difficulty difference is medium or small, and either chained equipercentile or frequency estimation is preferred with a large difficulty level. This study would provide practitioners with research evidence–based guidance in the choice of equating methods with varying levels of form difference. As the condition of no form difficulty difference is also included, this study would inform testing companies of appropriate equating methods when two forms are similar in difficulty level.

혼합형 가교문항 구성 방법이 혼합형 검사 동등화에 미치는 영향

With the introduction of innovative items in the computer-based NAEA, the item format became diversified, and the current method of selecting anchor items from multiple-choice (MC) items became burdensome for test developers. The purpose of this study is to examine whether a mixed-format anchor items including both MC and constructed-response (CR) items could hold the stability of equating in the NAEA, and give desirable direction to expand the range of anchor items and to make test construct flexible. A simulation study was conducted based on a common-item non-equivalent group design with the test conditions of the Korean subject in the NAEA. The treatment conditions include (1) a type of anchor item set (a single MC or a mixed), (2) a composition type of mixed anchor item (MC:CR = 1:1, 3:2, or 3:4), (3) a length of mixed anchor items within a test (25%, 22%, or 19%), and (4) the presence of multidimensionality according to the item format were assumed. The main result is that there was no significant difference in the equating error between the single MC and the mixed anchor item condition. Specifically the equating error of the single MC was smaller than that of the mixed-format cases where the ratios between MC and CR were 1:1 and 3:2. In the 3:4-ratio mixed condition, the equating error was smaller than that of the single MC condition. A length of mixed anchor items within a test showed somewhat mixed results, probably because there was not much variation between the conditions in the composition ratio, about 20% in test. When there was multidimensionality according to the item format in the test, the equating error increased significantly with mixed anchor items compared to that of unidimensional test condition. In sum, about 20% of the mixed anchor items with slightly higher CR ratio was found to be suitable in the unidimensional test and caution was called for in using mixed anchor items in multidimensional test.

Evaluating the Effects of Missing Data Handling Methods on Scale Linking Accuracy.

For large-scale assessments, data are often collected with missing responses. Despite the wide use of item response theory (IRT) in many testing programs, however, the existing literature offers little insight into the effectiveness of various approaches to handling missing responses in the context of scale linking. Scale linking is commonly used in large-scale assessments to maintain scale comparability over multiple forms of a test. Under a common-item nonequivalent group design (CINEG), missing data that occur to common items potentially influence the linking coefficients and, consequently, may affect scale comparability, test validity, and reliability. The objective of this study was to evaluate the effect of six missing data handling approaches, including listwise deletion (LWD), treating missing data as incorrect responses (IN), corrected item mean imputation (CM), imputing with a response function (RF), multiple imputation (MI), and full information likelihood information (FIML), on IRT scale linking accuracy when missing data occur to common items. Under a set of simulation conditions, the relative performance of the six missing data treatment methods under two missing mechanisms was explored. Results showed that RF, MI, and FIML produced less errors for conducting scale linking whereas LWD was associated with the most errors regardless of various testing conditions.

Bootstrap Equating Errors for the Common-item Nonequivalent Groups Design: A Comparison of Rasch Equating Methods

The nonequivalent groups with anchor test (NEAT) equating design are traditionally based on using a single anchor to adjust for differences in test difficulty which is critical to equating test forms in most large-scale testing programs. When tests differ somewhat in content and length, methods based on the item response theory (IRT) model leads to greater stability of equating results. The current study compared standard errors, bias, and root mean square errors using four Rasch IRT equating methods for the nonequivalent groups with anchor test design. The sizes of the equating anchor were employed in all four different Rasch equating methods to investigate how different anchor sizes may impact the test accuracy of the tests by conducting a simulation study.

Effect of Item Parameter Drift in Mixed Format Common Items on Test Equating

The aim of the study was to examine the common items in the mixed format (e.g., multiple-choices and essay items) contain parameter drifts in the test equating processes performed with the common item non-equivalent groups design. In this study, which was carried out using Monte Carlo simulation with a fully crossed design, the factors of test length (30 and 50), sample size (1000 and 3000), common item ratio (30 and 40%), ratio of items with item parameter drift (IPD) in common items (20 and 30%), location of common items in tests (at the beginning, randomly distributed, and at the end) and IPD size in multiple-choice items (low [0.2] and high [1.0]) were studied. Four test forms were created, and two test forms do not contain parameter drifts. After the parameter drift was performed on the first of the other two test forms, the parameter drift was again performed on the second test form. Test equating results were compared using the root mean squared error (RMSE) value. As a result of the research, ratio of items with IPD in common items, IPD size in multiple-choice items, common item ratio, sample size and test length on equating errors were found to be significant.

Investigation of the Missing Data Imputation Methods on Characteristic Curve Transformation Methods Used in Test Equating

In this research, the aim was to evaluate the effect of zero imputation and multiple imputation missing data handling methods on item response theory (IRT) based test equating methods under different conditions. Data in this study was obtained from the administration of the TIMSS 2019 eighth-grade science test. Data sets were formed by randomly selecting a sample of 1000 students with full data from booklets 7 and 8. By deleting data under a completely random missing data mechanism within the scope of common-item nonequivalent groups (CINEG) design, four different data sets were obtained with the missing data rates of 10% or 20% in the new test or in both tests. The missing da ta problem was solved by using zero imputation and multiple imputation methods from these data sets. In this way, 8 different data sets were formed. Then, scaling transformation was performed by using characteristic curve transformation methods (Haebara, Stocking-Lord). Test equating results were reported in terms of observed scores. The root mean square error (RMSE) was used as the evaluation criterion to determine the error involved in test equating. As a result, it was determined that in the case of 10% missing data in both tests, generally lower RMSE values were obtained. It was observed that the multiple imputation method, one of the methods for handling missing data, was the method that produced RMSE values that were both the lowest and closer to the full data set as a reference value compared to the zero-imputation method. In addition, it was determined that, when compared to the Haebara method, Stocking-Lord method, one of the characteristic curve transformation methods, produced lower RMSE values and these values were closer to the full data set, which was taken as a reference value.

Item Response Theory True Score Equating for the Bifactor Model Under the Common-Item Nonequivalent Groups Design.

Applying item response theory (IRT) true score equating to multidimensional IRT models is not straightforward due to the one-to-many relationship between a true score and latent variables. Under the common-item nonequivalent groups design, the purpose of the current study was to introduce two IRT true score equating procedures that adopted different dimension reduction strategies for the bifactor model. The first procedure, which was referred to as the integration procedure, linked the latent variable scales for the bifactor model and integrated out the specific factors from the item response function of the bifactor model. Then, IRT true score equating was applied to the marginalized bifactor model. The second procedure, which was referred to as the PIRT-based procedure, projected the specific dimensions onto the general dimension to obtain a locally dependent unidimensional IRT (UIRT) model and linked the scales of the UIRT model, followed by the application of IRT true score equating to the locally dependent UIRT model. Equating results obtained with the two equating procedures along with those obtained with the unidimensional three-parameter logistic (3PL) model were compared using both simulated and real data. In general, the integration and PIRT-based procedures provided equating results that were not practically different. Furthermore, the equating results produced by the two bifactor-based procedures became more accurate than the results returned by the 3PL model as tests became more multidimensional.

Scale Banking for Patient-Reported Outcome Measures That Measure Functioning in Rheumatoid Arthritis: A Daily Activities Metric.

Functioning is an important outcome for the management of rheumatoid arthritis (RA). Heterogeneity of respective patient-reported outcome measures (PROMs) challenges direct comparisons between their results. This study aimed to standardize reporting of such PROMs measuring functioning in RA to facilitate comparability. Common-item nonequivalent group design with the Health Assessment Questionnaire (HAQ) as a common scale across data sets from various countries (including the UK, Turkey, and Germany) to establish a common metric was used. Other PROMs included are the physical function items of the Multidimensional HAQ (MDHAQ), the Disabilities of the Arm, Shoulder, and Hand questionnaire, the Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), the World Health Organization Disability Assessment Schedule II (WHODAS II), the Medical Outcomes Study Short Form 36 (SF-36) health survey, and 4 short forms (20, 10, 6, and 4 physical function items) from the Patient-Reported Outcomes Measurement Information System. As the HAQ includes mobility, self-care, and domestic life items, this study focuses on these 3 domains. PROMs were described using standard error of measurement (SEM) and smallest detectable difference (SDD). A Rasch measurement model was used to create the common metric. The range of the SEM was 0.2 (MDHAQ) to 7.4 (SF-36 health survey physical functioning domain). The SDD revealed a range from 9.7% (WOMAC rating scale) to 33.5% (WHODAS physical functioning domain). PROMs co-calibration revealed fit to the Rasch measurement model. A transformation table was developed to allow exchange between PROM scores. Scores between the daily activity PROMs commonly used in RA can now be compared. Factors such as SEM and SDD help to determine the choice of a PROM in clinical practice and research.

Comparison of IRT observed score equating methods for CR items under the common-item nonequivalent groups design

이 연구는 공통문항 비동등집단 설계에서 구성형 문항의 IRT 관찰점수 동등화의 정확성을 알아보기 위한 모의실험 연구로, 다음 세 가지 조건을 사용하여 동등화의 정확성을 확인하였다. 첫째, 구성형 문항만을 공통문항으로 사용하여 일차원 IRT 모형을 적용하였다. 둘째, 구성형 문항과 외부(external) 선택형 문항을 공통문항으로 사용하여 일차원 IRT 모형을 적용하였다. 셋째, 구성형 문항과 외부 선택형 문항을 공통문항으로 사용하여 다차원 IRT 모형을 적용하였다. 연구 결과, 다음의 제한된 조건에서 공통문항에 구성형 문항만을 사용하는 것보다 선택형 문항을 함께 사용했을 때 동등화 결과가 더 정확하였다. 즉, 1) 다차원 IRT 모형을 적용하였을 때, 2) 표본 크기가 클 경우, 그리고 3) 선택형 문항에서 집단 간의 능력 차이가 작을 때 구성형 문항의 동등화의 정확성이 향상되었다. 또한, 일반적으로 문항의 수가 많고 점수 범주의 수가 적을 때 구성형 문항의 동등화 결과가 정확하였다.

Asymptotic Standard Errors of Generalized Partial Credit Model True Score Equating Using Characteristic Curve Methods.

In this study, the delta method was applied to estimate the standard errors of the true score equating when using the characteristic curve methods with the generalized partial credit model in test equating under the context of the common-item nonequivalent groups equating design. Simulation studies were further conducted to compare the performance of the delta method with that of the bootstrap method and the multiple imputation method. The results indicated that the standard errors produced by the delta method were very close to the criterion empirical standard errors as well as those yielded by the bootstrap method and the multiple imputation method under all the manipulated conditions.

Asymptotic Standard Errors of Equating Coefficients Using the Characteristic Curve Methods for the Graded Response Model

ABSTRACT The characteristic curve methods have been applied to estimate the equating coefficients in test equating under the graded response model (GRM). However, the approaches for obtaining the standard errors for the estimates of these coefficients have not been developed and examined. In this study, the delta method was applied to derive the mathematical formulas for computing the asymptotic standard errors for the parameter scale transformation coefficients and the true score equating coefficients that are estimated using the characteristic curve methods in test equating under the GRM in the context of the common-item nonequivalent groups equating design. Simulated and real data were further used to examine the accuracy of the derivations and compare the performance of the newly developed delta method with that of the multiple imputation method. The results indicated that the standard errors produced by the delta method were extremely close to the criterion empirical standard errors as well as those yielded by the multiple imputation method. The development of the standard error expressions by the delta method in the study has important practical implications.

Comparing Performance of Different Equating Methods in Presence and Absence of DIF Items in Anchor Test

This study aims to compare the performance of different small sample equating methods in the presence and absence of differential item functioning (DIF) in common items. In this research, Tucker linear equating, Levine linear equating, unsmoothed and presmoothed (C=4) chained equipercentile equating, and simplified circle arc equating methods were considered. The data used in this study is 8th-grade mathematics test item responses which obtained from Trends in International Mathematics and Science Study (TIMSS) 2015 Turkey sample. Item responses from Booklet-1 (N=199) and Booklet-14 (N=224) are chosen for this study. Data analyses were completed in four steps. In the first step, assumptions for DIF detection and test equating methods were checked. In the second step, DIF analyses were conducted with Mantel Haenszel and logistic regression methods. In the third step, Booklet 1 was chosen as base form and Booklet 14 chosen as a new form, then test equating was conducted under common item nonequivalent groups design. Test equating was done in two phases: the presence and absence of DIF items in the common items. Equating results were evaluated based on standard error of equating (se), bias and RMSE indexes. DIF analyses showed that there were two sizeable DIF items in anchor test. Equating results showed that performances of equating methods are similar in presence and absence of DIF items from anchor test and there is no notable change in se, bias and RMSE values. While the circle arc equating method outperformed other equating methods based on se, 4-moment presmoothed chained equipercentile equating method outperformed other methods based on bias and RMSE evaluation criteria.

Estimating standard errors of IRT true score equating coefficients using imputed item parameters

Reporting standard errors of equating has been advocated as a standard practice when conducting test equating. The two most widely applied procedures for standard errors of equating including the bootstrap method and the delta method are either computationally intensive or confined to the derivations of complicated formulas. In the current study, a hypothetical example was used to illustrate how the multiple imputation method could be taken as an alternative procedure for obtaining the standard errors for the item response theory (IRT) true score equating coefficients in the context of the common-item nonequivalent groups equating design under the three-parameter logistic IRT model. This method makes use of multiple sets of imputed item parameter values. By using the simulated and real data, the performance of the multiple imputation method was examined and compared with that of the bootstrap and delta methods. The results indicated that the multiple imputation method performed as effectively as the bootstrap method and the delta method when using the characteristic curve methods. The multiple imputation method produced very similar results to the delta method when the moment methods were used.

Common-Item Scale-Linking Methods for the Bi-factor Graded Response Model in MIRT

다차원 문항반응이론에서, 이요인 등급반응(BGR) 모형은 순서화된 다분 문항반응을 검사 문항 전체에 연관된 일반 요인과 각 문항집단에 독립적인 특수 요인으로 분석하기 위해 사용되어 왔다. 본 연구의 목적은 BGR 모형을 위한 공통-문항 기반 척도연계 방법들을 제시하고 모의실험을 통해 이 방법들의 특성과 상대적 기능을 탐구하는 데 있다. 제시되는 척도연계 방법들은 최소제곱(LSQ) 방법, 평균-최소제곱(MLS) 방법, 문항유목반응함수(IRF) 방법, 그리고 검사반응함수(TRF) 방법이다. 각 척도연계 방법은, 모든 요인들 간의 독립성 가정에 따라, 각 차원에 대해 임의의 두 능력척도를 연계하기 위한 팽창 계수와 이동 계수를 추정한다. 네 가지 척도연계 방법들의 기능을 평가하기 위해 피험자 집단 간 능력의 비동등성 수준, 표본의 크기, 공통 문항의 수 등을 요인으로 포함하는 모의실험을 수행하였다. 주요 결과로, 네 가지 척도연계 방법은 고려된 모의실험 조건 모두에서 팽창 계수를 적절히 추정하였다. 그러나 이동 계수를 추정함에 있어서, IRF, LSQ 및 MLS 방법은 적절히 기능한 반면 TRF 방법은 그렇지 못하였다. 척도연계를 통한 문항모수의 복원에 있어서, 전반적으로 IRF 방법이 가장 우수하였으나, 표본의 크기가 작거나 공통 문항의 수가 작을 경우 LSQ 방법과 MLS 방법 또한 우수한 수행을 보였다.In applications of multidimensional item response theory (MIRT), the bi-factor graded response (BGR) model can be used to analyze the ordered response data of bi-factor latent structures. The purpose of this study is to present four scale-linking methods for the BGR model under the context of the common-item nonequivalent groups design and investigate their characteristics and performances through computer simulations. The four scale-linking methods presented are the least squares (LSQ), mean-least squares (MLS), item-category response function (IRF), and test response function (TRF) methods. The four methods estimate the dilation and translation linking coefficients with the assumption of independence among all general and group factors. Computer simulations that included three factors of (a) nonequivalence level between two examinee groups, (b) sample size, and (c) number of common items between linked test forms were conducted to examinee the characteristics and relative performances of the four scale-linking methods. Overall, the four methods properly estimated the dilation coefficients. In estimation of the translation coefficients, however, the IRF, LSQ, and MLS methods functioned properly but the TRF method did not. For the recovery of item parameters, the IRF method performed best in mose cases, but the LSQ and MLS methods also performed well in the cases of small sample size and small number of common items.

HOW TO TREAT OMITTED RESPONSES IN RASCH MODEL-BASED EQUATING

This study investigated the impact of the coding scheme on IRT-based true score equating under a common-item nonequivalent groups design. Two different coding schemes under investigation were carried out by assigning either a zero or a blank to a missing item response in the equating data. The investigation involved a comparison study using actual large scale data and then Monte Carlo simulations for a systematic inspection on the topic. The recommendations on the basis of the findings of the study were made to treat omitted responses as not-administered rather than as wrong, and use a large sample size to ensure the accuracy of the screening tools such as the displacement index and the robust-z statistic during equating.

Physical Reasoning Ability Development of High School Students

This study has purpose to reveal: (1) the characteristics of physics subject test of high school students of grade X and XI, (2) the equality of vertical test of physics subject of high school students of grade X and XI, (3) the development of the ability of high school students of grade X to grade XI physics subject. The data which was in the form of scores were then processed and synchronized by using a common item non-equivalent groups design by the method of equalizing of mean and sigma based on the Rasch model. The results of this study are: (1) the validity of the physics subjects test of grade X is 0.92 and grade XI is 0.96 (2) the reliability tests is 0.928 for grade X and 0.926 for grade XI; (3) the results of the vertical equalization based on the approach of Rasch model showed the conversion equation for grade X to grade XI was b * = bx - 0.36; (4) there is a development of the ability of grade X students to grade XI students or the ability of grade XI students is higher than the ability of grade X students.

Test Eşitlemede Çok Boyutluluğun Eş Zamanlı ve Ayrı Kalibrasyona Etkisi

The aim of this study is to investigate the effects of multidimensionality on equating results which obtained from separate and concurrent calibration methods. The study was conducted with using simulated data. In the scope of research, totally 40 simulation conditions [5 (degree of multidimensionality: 0.90, 0.75, 0.50, 0.25, and 0.00) x 2 calibration methods (separate and concurrent) x 2 (scale transformation methods: Stocking-Lord and Haebara) x 2 (test equating methods: IRT true score equating and observed score equating)] were examined. Multidimensionality was constructed as assuming the two test forms measuring Ɵ1 and Ɵ2 abilities. While the simulation condition which has correlation between abilities 0.90 represents weak multidimensional case, the correlation between abilities 0.00 represents the severe multidimensional case. Tests were equated under common-item non-equivalent groups design. Equating results were evaluated by using bias, standard deviation and RMSE evaluation criteria. The results showed that, generally under all conditions equating results provided from concurrent calibration more biased and had higher RMSE values than equating results provided by separate calibration. Based on standard deviation criteria, when the degree of multidimensionality was low, equating results which got from concurrent calibration and separate calibration with Stocking-Lord or Haebara scale transformation methods showed similar performance but when the degree of multidimensionality was severe equating results which had lowest random error were provided by concurrent calibration.

Software Review of IRTEQ, STUIRT, and POLYEQUATE for Item Response Theory Scale Linking and Equating

ABSTRACTThis article reviews several software programs designed to conduct item response theory (IRT) scale linking and equating. The programs reviewed include IRTEQ, STUIRT, and POLYEQUATE. Features and functionalities of each program are discussed and an example analysis using the common-item non-equivalent groups design in IRTEQ is provided.