Mixed-format Tests Research Articles

IRT Characteristic Curve Linking Methods Weighted by Information for Mixed-Format Tests

IRT Characteristic Curve Linking Methods Weighted by Information for Mixed-Format Tests

Customizing Bayesian multivariate generalizability theory to mixed-format tests.

Mixed-format tests, which typically include dichotomous items and polytomously scored tasks, are employed to assess a wider range of knowledge and skills. Recent behavioral and educational studies have highlighted their practical importance and methodological developments, particularly within the context of multivariate generalizability theory. However, the diverse response types and complex designs of these tests pose significant analytical challenges when modeling data simultaneously. Current methods often struggle to yield reliable results, either due to the inappropriate treatment of different types of response data separately or the imposition of identical covariates across various response types. Moreover, there are few software packages or programs that offer customized solutions for modeling mixed-format tests, addressing these limitations. This tutorial provides a detailed example of using a Bayesian approach to model data collected from a mixed-format test, comprising multiple-choice questions and free-response tasks. The modeling was conducted using the Stan software within the R programming system, with Stan codes tailored to the structure of the test design, following the principles of multivariate generalizability theory. By further examining the effects of prior distributions in this example, this study demonstrates how the adaptability of Bayesian models to diverse test formats, coupled with their potential for nuanced analysis, can significantly advance the field of psychometric modeling.

Impact of Violating Unidimensionality on Rasch Calibration for Mixed-Format Tests

ABSTRACT Mixed-format tests that include both multiple-choice (MC) and constructed-response (CR) items have become widely used in many large-scale assessments. When an item response theory (IRT) model is used to score a mixed-format test, the unidimensionality assumption may be violated if the CR items measure a different construct from that measured by MC items. In this simulation study, we explore the impact of different degrees of violation on the unidimensionality assumption on Rasch item parameter recovery and model-data fit statistics. Our results suggest that, when the unidimensionality assumption is violated for mixed-format tests, both estimated Rasch item parameters and model-data fit statistics are distorted, with greater impact corresponding to higher degrees of violations. The findings from this study may provide guidance for test practitioners for working with mixed-format tests within the Rasch context.

Parallel Optimal Calibration of Mixed-Format Items for Achievement Tests

When large achievement tests are conducted regularly, items need to be calibrated before being used as operational items in a test. Methods have been developed to optimally assign pretest items to examinees based on their abilities. Most of these methods, however, are intended for situations where examinees arrive sequentially to be assigned to calibration items. In several calibration tests, examinees take the test simultaneously or in parallel. In this article, we develop an optimal calibration design tailored for such parallel test setups. Our objective is both to investigate the efficiency gain of the method as well as to demonstrate that this method can be implemented in real calibration scenarios. For the latter, we have employed this method to calibrate items for the Swedish national tests in Mathematics. In this case study, like in many real test situations, items are of mixed format and the optimal design method needs to handle that. The method we propose works for mixed-format tests and accounts for varying expected response times. Our investigations show that the proposed method considerably enhances calibration efficiency.

다차원 IRT 모형 적합 혼합형 검사를 위한 공통문항 척도 연계 방법

Under the context of the common-item nonequivalent groups design, item response theory (IRT) scale
 linking methods have been presented for each of the multidimensional extensions of the three-parameter
 logistic (M3PL), graded response (MGR), generalized partial credit (MGPC), and nominal response (MNR)
 models. The common-item multidimensional IRT (MIRT) linking methods that have been proved effective
 for single-format tests include the direct least squares (DLS), item category response function (ICRF), and
 test response function (TRF) methods. This paper extends the three MIRT linking methods to any mixture
 of the exploratory M3PL, MGR, MGPC, and MNR models for mixed-format tests. Each linking method
 extended attempts to find the linking coefficients in the rotation matrix and translation vector that
 constitute a linear transformation between two multidimensional ability scales. Simulations are conducted to
 investigate the performance of the three linking methods under various linking conditions. The results
 indicated that overall, the ICRF method performed best in estimating the rotation matrix and translation
 vector of the linear transformation between the new and true simulation scales. In not a few conditions, the
 DLS method resulted in the second best linking solutions. The TRF method tended to estimate the rotation
 matrix well but the translation vector poorly and thus showed the worst performance among the linking
 methods in some conditions.

랜덤집단 설계 IRT 검사 동등화의 표준오차 추정을 위한 델타 방법의 기능 진단

In the academic achievement and ability testing programs, item response theory (IRT) methods have been widely used for test equating. The standard error (SE) of equating indicates the amount of random error that is due to the sampling of examinees in estimating the population equivalent scores relationship. The purpose of the present study was to use computer simulations to examine the accuracy and performance of the delta method formulas for estimating SEs of IRT true score and observed score equating that have been presented under the random groups design. Test type (3PL model test, 3PL+GPC model test, and 3PL+GR model test) and sample size were considered as simulation factors. Main results were as follows. First, for all the test types, the theoretical SEs of IRT equating estimated by the delta method were very close to the empirical SEs computed using the simulated equated scores. Second, the SEs of IRT equating were reduced as the sample size increased, and they were, approximately, inversely proportional to the square root of the sample size. Third, except for the extreme (lowest or highest) test scores, the SEs of IRT observed score equating tended to be slightly smaller than the SEs of IRT true score equating. Fourth, on the average, the SEs of IRT equating for the 3PL model test, which consisted of 30 multiple-choice items, were smaller than those for the mixed-format tests, which consisted of the same 30 multiple-choice items and 10 constructed-response items.

Efficiency Analysis of Item Response Theory Kernel Equating for Mixed-Format Tests.

This study aims to evaluate the performance of Item Response Theory (IRT) kernel equating in the context of mixed-format tests by comparing it to IRT observed score equating and kernel equating with log-linear presmoothing. Comparisons were made through both simulations and real data applications, under both equivalent groups (EG) and non-equivalent groups with anchor test (NEAT) sampling designs. To prevent bias towards IRT methods, data were simulated with and without the use of IRT models. The results suggest that the difference between IRT kernel equating and IRT observed score equating is minimal, both in terms of the equated scores and their standard errors. The application of IRT models for presmoothing yielded smaller standard error of equating than the log-linear presmoothing approach. When test data were generated using IRT models, IRT-based methods proved less biased than log-linear kernel equating. However, when data were simulated without IRT models, log-linear kernel equating showed less bias. Overall, IRT kernel equating shows great promise when equating mixed-format tests.

이요인 모형 기반 혼합형 검사를 위한 다차원 IRT 척도 연계 방법

Like unidimensional item response theory (IRT) models, bifactor models in multidimensional IRT have a scale indeterminacy problem, and due to this problem scale linking methods are needed to place all bifactor model parameter estimates from separate calibrations on a common ability scale. Four bifactor scale linking methods including the direct least squares (DLS), mean/least squares (MLS), item category response function (ICRF), and test response function (TRF) methods have been presented for use with single-format tests. Parallel to the 2006 paper of Kim and Lee, this paper extends the four scale linking methods to a mixture of bifactor models for mixed-format tests. Each linking method extended is intended to handle mixed-format tests using any mixture of the following bifactor extensions of four unidimensional IRT models: the bifactor three-parameter logistic, bifactor graded response, bifactor generalized partial credit, and bifactor nominal response models. For generality, symmetric criterion functions are proposed for the ICRF and TRF methods. Given two sets of parameter estimates for the common items linking two test forms, each linking method estimates the dilation (slope) and translation (intercept) coefficients of a linear transformation. Simulations are conducted to investigate the performance of the four linking methods. The results indicate that overall, the ICRF method performs very well, the MLS and DLS methods perform well (the MLS method is slightly better than the DLS method), and the TRF method performs poorly in estimating the linking coefficients. The inferiority of the TRF method is mainly due to its poor estimation of the translation coefficients.

The Optimal Design of Bifactor Multidimensional Computerized Adaptive Testing with Mixed-format Items.

Multidimensional computerized adaptive testing (MCAT) using mixed-format items holds great potential for the next-generation assessments. Two critical factors in the mixed-format test design (i.e., the order and proportion of polytomous items) and item selection were addressed in the context of mixed-format bifactor MCAT. For item selection, this article presents the derivation of the Fisher information matrix of the bifactor graded response model and the application of the bifactor dimension reduction method to simplify the computation of the mutual information (MI) item selection method. In a simulation study, different MCAT designs were compared with varying proportions of polytomous items (0.2-0.6, 1), different item-delivering formats (DPmix: delivering polytomous items at the final stage; RPmix: random delivering), three bifactor patterns (low, middle, and high), and two item selection methods (Bayesian D-optimality and MI). Simulation results suggested that a) the overall estimation precision increased with a higher bifactor pattern; b) the two item selection methods did not show substantial differences in estimation precision; and c) the RPmix format always led to more precise interim and final estimates than the DPmix format. The proportions of 0.3 and 0.4 were recommended for the RPmix and DPmix formats, respectively.

Extended Multivariate Generalizability Theory With Complex Design Structures.

This article extends multivariate generalizability theory (MGT) to tests with different random-effects designs for each level of a fixed facet. There are numerous situations in which the design of a test and the resulting data structure are not definable by a single design. One example is mixed-format tests that are composed of multiple-choice and free-response items, with the latter involving variability attributable to both items and raters. In this case, two distinct designs are needed to fully characterize the design and capture potential sources of error associated with each item format. Another example involves tests containing both testlets and one or more stand-alone sets of items. Testlet effects need to be taken into account for the testlet-based items, but not the stand-alone sets of items. This article presents an extension of MGT that faithfully models such complex test designs, along with two real-data examples. Among other things, these examples illustrate that estimates of error variance, error-tolerance ratios, and reliability-like coefficients can be biased if there is a mismatch between the user-specified universe of generalization and the complex nature of the test.

A Comparison of Common IRT Model-selection Methods with Mixed-Format Tests

ABSTRACT To date, only frequentist model-selection methods have been studied with mixed-format data in the context of IRT model-selection, and it is unknown how popular Bayesian model-selection methods such as DIC, WAIC, and LOO perform. In this study, we present the results of a comprehensive simulation study that compared the performances of eight model-selection methods with mixed-format data to select the correct combination of IRT models. Findings of the simulation study indicate that DIC, WAIC, and LOO had excellent statistical power to choose the correct IRT model combination. They performed comparably with LRT and slightly preferably than AIC, and considerably better than BIC, AICc, and SABIC. In addition, the performances of the three Bayesian methods were more stable than those of AIC and LRT regardless of the sample size and ability distribution. The eight model-selection methods were applied to a real dataset for demonstration purpose.

The Effects of Proportion and Composition of Common Items on Observed-Score Equating with the Bi-Factor MIRT model for Mixed-Format Tests

이 연구에서는 Bi-Factor 다차원 문항반응모형을 공통문항 비동등집단 설계에서 수집되었다고 가정한 혼합형 검사 자료에 적용하여 관찰점수 동등화를 진행하고, 그 결과가 공통문항의 구성 조건 및 문항 유형 요인의 상관 정도에 따라 어떠한 차이를 보이는지 살펴보았다. 이를 평가하기 위하여 선행연구에서 많이 사용되었던 2모수 일차원 문항반응모형을 비교 모형으로 설정하였다. 연구 결과, 문항 유형 간 상관이 제일 낮은 조건인 0.5와 중간 조건인 0.7인 경우에서는 공통문항의 비율 및 구성 방식에 상관없이 모든 조건에서 Bi-Factor MIRT 모형의 동등화 오차가 일차원 모형보다 더 양호한 것으로 나타났다. 이에 반해 문항 유형 간 상관이 0.9인 경우에서는 2모수 일차원 문항반응모형의 동등화 오차가 더 낮은 것으로 나타났다. 공통문항의 구성에 따라서는 전체 검사의 선택형 문항과 서답형 문항의 비율과 유사하게 공통문항이 구성되었을 때 동등화 오차가 가장 낮은 것으로 나타났고, 전체 검사 문항에 대한 공통문항의 비율이 증가함에 따라 동등화 오차는 감소하는 것으로 나타났다.

Comparison of the Standard Errors of Equating by the IRT and Equipercentile Methods Under the Random Groups Design

검사 동등화 연구에서 자료 수집 설계로 랜덤집단 설계가 종종 사용된다. 최근 들어, 두 검사형 간 동등점수 관계를 찾기 위해 전통적인 동백분위 동등화 방법보다는 문항반응이론(IRT) 기반 동등화 방법이 더 많이 활용되고 있다. 본 연구의 첫째 목적은 랜덤집단 설계하에서 IRT 동등화 방법에 대한 동등화 표준오차의 점근 공식을 혼합형 검사에 적용할 수 있도록 일반적으로 제시하는 데 있다. 둘째 목적은 점근 공식에 기초하여 IRT 방법과 동백분위 방법에 의해 산출되는 동등화 표준오차가 어떻게 다른지를 경험적 자료를 통해 검토하는 것이다. 둘째 목적과 관련하여, 다양한 동등화 조건들을 포함하여 실시한 비교 연구를 통해 다음의 주요 결과를 얻었다. 첫째, IRT 관찰점수 동등화 방법은 IRT 진점수 동등화 방법보다 평균적으로 더 작은 동등화 표준오차를 산출하였으며, 두 IRT 동등화 방법은 동백분위 동등화 방법보다 더 작은 동등화 표준오차를 산출하였다. 둘째, 각 동등화 방법은 검사의 길이가 증가할수록 더 큰 동등화 표준오차를 산출하였다. 셋째, 각 동등화 방법은 표본의 크기가 커질수록 더 작은 동등화 표준오차를 산출하였으며, 평균 동등화 표준오차는 대략적으로 표본 크기의 제곱근에 반비례하였다.The random groups design is often used for collecting sample data for test equating. Recently, the item response theory (IRT) based equating methods have been more widely used than the traditional equipercentile equating method. One of the two purposes of this study is to present general formulas for the asymptotic standard errors of equated scores estimated by the IRT true score and observed score equating methods under the random groups design so that they can be used for equating with mixed-format tests. The other purpose is to use empirical test data to examine how the standard errors of equating produced by the IRT equating methods differ from those by the equipercentile equating method. Concerning the second purpose, a comparative study, including various equating conditions, was conducted and the following were found. First, on the average, the IRT observed score equating method resulted in slightly smaller standard errors of equating than the IRT true score equating method, and the two IRT equating methods produced smaller standard errors than the equipercentile equating method which was based on unsmoothed score distributions. Second, the standard errors produced by the three equating methods increased as the lengths of the test forms equated increased and thus the relative frequencies for test scores were lowered. Third, the mean standard errors of equating for the three equating methods decreased as the sample size increased and the amounts of the errors were inversely proportional to the square root of the sample size.

Automated Essay Scoring Effect on Test Equating Errors in Mixed-format Test

Scoring constructed-response items can be highly difficult, time-consuming, and costly in practice. Improvements in computer technology have enabled automated scoring of constructed-response items. However, the application of automated scoring without an investigation of test equating can lead to serious problems. The goal of this study was to score the constructed-response items in mixed-format tests automatically with different test/training data rates and to investigate the indirect effect of these scores on test equating compared with human raters. Bidirectional long-short term memory (BLSTM) was selected as the automated scoring method for the best performance. During the test equating process, methods based on classical test theory and item response theory were utilized. In most of the equating methods, errors of the equating resulting from automated scoring were close to the errors occurring in equating processes conducted by human raters. It was concluded that automated scoring can be applied because it is convenient in terms of equating.

Examining of Internal Consistency Coefficients in Mixed-Format Tests in Different Simulation Conditions

Examining of Internal Consistency Coefficients in Mixed-Format Tests in Different Simulation Conditions

Using Bayesian Nonparametric Item Response Function Estimation to Check Parametric Model Fit.

Previous studies indicated that the assumption of logistic form of parametric item response functions (IRFs) is violated often enough to be worth checking. Using nonparametric item response theory (IRT) estimation methods with the posterior predictive model checking method can obtain significance probabilities of fit statistics in a Bayesian framework by accounting for the uncertainty of the parameter estimation and can indicate the location and magnitude of misfit for an item. The purpose of this study is to check the performance of the Bayesian nonparametric method to assess the IRF fit of parametric IRT models for mixed-format tests and compare it with the existing bootstrapping nonparametric method under various conditions. The simulation study results show that the Bayesian nonparametric method can detect misfit items with higher power and lower type I error rates when the sample size is large and with lower type I error rates compared with the bootstrapping method for the conditions with nonmonotonic items. In the real-data study, several dichotomous and polytomous misfit items were identified and the location and magnitude of misfit were indicated.

A Comparison of IRT Model Combinations for Assessing Fit in a Mixed Format Elementary School Science Test

Open-ended and multiple choice questions are commonly placed on the same tests; however, there is a discussion on the effects of using different item types on the test and item statistics. This study aims to compare model and item fit statistics in a mixed format test where multiple choice and constructed response items are used together. In this 25-item fourth-grade science test administered to 2351 students in 35 schools in Turkey, items are calibrated separately and concurrently utilizing different IRT models. An important aspect of this study is that the effect of the calibration method on model and item fit is investigated on real data. Firstly, while the 1-, 2-, and 3-Parameter Logistic models are utilized to calibrate the binary coded items, the Graded Response Model and the Generalized Partial Credit Model are used to calibrate the open-ended ones. Then, combinations of dichotomous and polytomous models are employed concurrently. The results based on model comparisons revealed that the combination of the 3PL and the Graded Response Model produced the best fit statistics.

Classification Consistency and Accuracy for Mixed-Format Tests

ABSTRACT This study explores classification consistency and accuracy for mixed-format tests using real and simulated data. In particular, the current study compares six methods of estimating classification consistency and accuracy for seven mixed-format tests. The relative performance of the estimation methods is evaluated using simulated data. Study results from real data analysis showed that the procedures exhibited similar patterns across various exams, but some tended to produce lower estimates of classification consistency and accuracy than others. As data became more multidimensional, unidimensional and multidimensional item response theory (IRT) methods tended to produce different results, with the unidimensional approach yielding lower estimates than the multidimensional approach. Results from simulated data analysis demonstrated smaller estimation error for the multidimensional IRT methods than for the unidimensional IRT method. The unidimensional approach yielded larger error as tests became more multidimensional, whereas a reverse relationship was observed for the multidimensional IRT approach. Among the non-IRT approaches, the normal approximation and Livingston-Lewis methods performed well, whereas the compound multinomial method tended to produce relatively larger error.

An Investigation of the Factors Affecting the Vertical Scaling of Multidimensional Mixed-Format Tests

This study examined the effect of the structure of a common item set (only dichotomous common items – mixed-format common item sets), parameter estimation methods and scale shrinkage on vertical scaling results when multidimensional datasets were used within the context of Common Item Nonequivalent Group (CINEG) design. Interactions between these variables were also investigated. The study was performed using simulated data. Measurement error and bias indexes were used to evaluate the quality of vertical scaling. All the procedures used in the data analysis were replicated 50 times to increase the generalizability of the results. R program was used for the data generation, calibration of the parameters and vertical scaling procedures. Possible interactions were investigated with factorial analysis of variance by using SPSS. The results showed a consistent effect of the common item format in all conditions. In addition, some interactions between the variables were observed. These findings are discussed and some recommendations are provided.

Contextual Collaborative Filtering for Student Response Prediction in Mixed-Format Tests

The purpose of this study is to design a machine learning approach to predict the student response in mixed-format tests. Particularly, a novel contextual collaborative filtering model is proposed to extract latent factors for students and test items, by exploiting the item information. Empirical results from a simulation study validate the effectiveness of the proposed method.