Maximum Information Method Research Articles

Simultaneous Estimation of Overall Score and Subscores Using MIRT, HO-IRT and Bi-factor Model on TIMSS Data

In educational testing, there is an increasing interest in the simultaneous estimation of the overall scores and subscores. This study aims to compare the reliability and precision of the simultaneous estimation of overall scores and sub-scores using MIRT, HO-IRT and Bi-factor models. TIMSS 2015 mathematics scores have been used as a data set in this study. The TIMSS 2015 mathematics test consists of 35 items, four of which are polytomously scored (0-1-2), and the rest of the items are dichotomously scored (0-1). The four content domains include number (14 items), algebra (9 items), geometry (6 items), and data and change (6 items). Ability parameters were estimated using the BMIRT software. The results showed that the MIRT and HO-IRT methods performed similarly in terms of precision and reliability for subscore estimates. The MIRT maximum information method had the smallest standard error of measurement for the overall score estimates. All three methods performed similarly in terms of the overall score reliability. The findings suggest that among the three methods compared, HO-IRT appears to be a better choice in the simultaneous estimation of the overall score and subscores for the data from TIMSS 2015. Recommendations for the testing practices and future research are provided.

A simplified version of the maximum information per time unit method in computerized adaptive testing.

In this article, we propose a simplified version of the maximum information per time unit method (MIT; Fan, Wang, Chang, & Douglas, Journal of Educational and Behavioral Statistics 37: 655-670, 2012), or MIT-S, for computerized adaptive testing. Unlike the original MIT method, the proposed MIT-S method does not require fitting a response time model to the individual-level response time data. It is also computationally efficient. The performance of the MIT-S method was compared against that of the maximum information (MI) method in terms of measurement precision, testing time saving, and item pool usage under various item response theory (IRT) models. The results indicated that when the underlying IRT model is the two- or three-parameter logistic model, the MIT-S method maintains measurement precision and saves testing time. It performs similarly to the MI method in exposure control; both result in highly skewed item exposure distributions, due to heavy reliance on the highly discriminating items. If the underlying model is the one-parameter logistic (1PL) model, the MIT-S method maintains the measurement precision and saves a considerable amount of testing time. However, its heavy reliance on time-saving items leads to a highly skewed item exposure distribution. This weakness can be ameliorated by using randomesque exposure control, which successfully balances the item pool usage. Overall, the MIT-S method with randomesque exposure control is recommended for achieving better testing efficiency while maintaining measurement precision and balanced item pool usage when the underlying IRT model is 1PL.

The Effect of Upper and Lower Asymptotes of IRT Models on Computerized Adaptive Testing.

In this article, the effect of the upper and lower asymptotes in item response theory models on computerized adaptive testing is shown analytically. This is done by deriving the step size between adjacent latent trait estimates under the four-parameter logistic model (4PLM) and two models it subsumes, the usual three-parameter logistic model (3PLM) and the 3PLM with upper asymptote (3PLMU). The authors show analytically that the large effect of the discrimination parameter on the step size holds true for the 4PLM and the two models it subsumes under both the maximum information method and the b-matching method for item selection. Furthermore, the lower asymptote helps reduce the positive bias of ability estimates associated with early guessing, and the upper asymptote helps reduce the negative bias induced by early slipping. Relative step size between modeling versus not modeling the upper or lower asymptote under the maximum Fisher information method (MI) and the b-matching method is also derived. It is also shown analytically why the gain from early guessing is smaller than the loss from early slipping when the lower asymptote is modeled, and vice versa when the upper asymptote is modeled. The benefit to loss ratio is quantified under both the MI and the b-matching method. Implications of the analytical results are discussed.

PEMILIHAN BUTIR SOAL PADA RANCANGAN TES ADAPTIF BERDASARKAN EFFICIENCY BALANCED INFORMATION

One of the most popular item selection methods in the design of adaptive testing is Maximum Information Method. This method provides the items with the maximum information at a certain level of selecting ability. The lack of this method is rarely accurate in estimating the level of ability of the examinees at the beginning of the test and tends to select items with higher discrimination-parameter value than items with lower discrimination-parameter. It creates problem in maintaining item bank. Therefore, another method should be found. The objective of the study is to determine the performance of the application of the estimation method Balanced Efficiency Information (EBI) on the design of an adaptive test. This research was carried out through a simulation study in the setting of organizing the Open University final exams. Item bank for the purposes of simulation models based on 3 parameters Item Response Theory was contructed a total of 900 items were generated base on the ideal parameter of the item specifications. Two selection criteria items were simulated, namely Information Maximum and EBI Maximum. Those two criteria were also designed to meet the content balancing. This is to ensure that the algorithm is appropriate with the applicable UT modular learning, meaning items of each module were proportionally represented and suited the blueprint. The tests will be stopped at when the standard error of estimate ( standard error of estimation = SEE ) is 0.3.The study summarized that the algorithm of EBI was more accurate than the Maximum Information criteria in estimating performance capabilities of participants. This is indicated by the value of the bias and the standard deviation of EBI is smaller than Maximum Information criterias. Another advantage of the application of the EBI Maximum is optimally utilizing of the item bank. The items with low level of the discrimination-parameter will also be chosen at the begining of the test. The maximum information criterion is more efficient in terms of test length but less optimally of the item bank utilization.
 Salah satu metode pemilihan butir soal yang popular digunakan dalam rancangan tes adaptif adalah metode Informasi Maksimum. Melalui metode ini, butir soal yang memiliki informasi maksimum pada tingkat kemampuan tertentu akan dipilih dan diberikan kepada peserta tes. Namun kelemahan dari metode ini adalah kurang akurat dalam mengestimasi tingkat kemampuan peserta pada awal tes dan memiliki kecenderungan untuk memilih butir dengan nilai daya pembeda parameter butir yang tinggi dibandingkan butir dengan nilai parameter daya pembeda yang rendah, sehingga menimbulkan masalah pemeliharaan butir soal dalam bank soal. Karena itu dicari cara lain untuk mengatasi masalah tersebut. Penelitian ini bertujuan untuk mengetahui performa hasil estimasi dari penerapan metode Efficiency Balanced Information (EBI) pada rancangan tes adaptif. Penelitian ini dilakukan melalui studi simulasi dalam setting penyelenggaraan ujian akhir semester Universitas Terbuka. Bank soal untuk keperluan simulasi dibangkitkan berdasarkan model Item Response Theory 3 parameter. Sebanyak 900 butir soal dalam bank soal bangkitan dengan spesifikasi parameter butir yang ideal. Dua kriteria pemilihan butir soal yang disimulasikan yaitu Informasi Maksimum dan EBI Maksimum yang juga dirancang agar memenuhi keseimbangan isi. Hal ini agar menjamin bahwa algoritma yang dihasilkan sesuai dengan pembelajaran moduler yang diterapkan UT, artinya butir soal setiap modul secara proporsional terwakili dan sesuai kisi-kisi. Aturan pemberhentian tes menggunakan kesalahan baku estimasi (standard error of estimation=SEE) sebesar 0,3. Hasil penelitian menyimpulkan bahwa algoritma rancangan tes adaptif dengan kriteria EBI menghasilkan performa hasil estimasi kemampuan peserta yang lebih akurat dibandingkan kriteria Informasi Maksimum. Hal ini ditunjukkan oleh nilai bias dan simpangan baku pengukuran yang lebih kecil dibandingkan kriteria Informasi Maksimum. Kelebihan lain dari penerapan kriteria EBI Maksimum adalah kebermanfaatan bank soal lebih optimal karena butir-butir soal dengan tingkat daya beda rendah juga dimunculkan khususnya pada awal tes. Sedangkan kriteria Informasi Maksimum walaupun lebih efisien dari sisi panjang tes tetapi kurang optimal dalam memanfaatkan bank soal.

Multidimensional Linking for Domain Scores and Overall Scores for Nonequivalent Groups

The No Child Left Behind Act requires state assessments to report not only overall scores but also domain scores. To see the information on students’ overall achievement, progress, and detailed strengths and weaknesses, and thereby identify areas for improvement in educational quality, students’ performances across years or across forms need to be made comparable in terms of both overall scores and domain scores. Multidimensional item response theory (MIRT) and its linking procedures for item parameter recovery have been studied; however, the effects of those linking procedures on the recovery of domain scores and overall scores have not been addressed, especially with regard to MIRT models of higher dimensions. The relationship between an examinee’s overall score and domain scores can be complex and may not be adequately represented by a linear function. This article proposes using the MIRT maximum information method to obtain the overall scores from MIRT domain scores. A simulation study was conducted to investigate the accuracy and effects of using the MIRT matching test response function (TRF) linking procedure to link the five-dimensional domain scores and their overall scores under varying conditions of anchor set lengths, population distribution types, and sample sizes. The results show that the TRF method recovered the domain scores and overall scores well for an anchor set of 10 items, with 2 items in each domain. Domain scores and overall scores have high reliabilities when the correlations between domains are high; reliability is higher than .9 for overall scores and higher than .8 for domain scores under all conditions.

Reporting Valid and Reliable Overall Scores and Domain Scores

In educational assessment, overall scores obtained by simply averaging a number of domain scores are sometimes reported. However, simply averaging the domain scores ignores the fact that different domains have different score points, that scores from those domains are related, and that at different score points the relationship between overall score and domain score may be different. To report reliable and valid overall scores and domain scores, I investigated the performance of four methods using both real and simulation data: (a) the unidimensional IRT model; (b) the higher‐order IRT model, which simultaneously estimates the overall ability and domain abilities; (c) the multidimensional IRT (MIRT) model, which estimates domain abilities and uses the maximum information method to obtain the overall ability; and (d) the bifactor general model. My findings suggest that the MIRT model not only provides reliable domain scores, but also produces reliable overall scores. The overall score from the MIRT maximum information method has the smallest standard error of measurement. In addition, unlike the other models, there is no linear relationship assumed between overall score and domain scores. Recommendations for sizes of correlations between domains and the number of items needed for reporting purposes are provided.

Item Selection in Computerized Adaptive Testing: Should More Discriminating Items be Used First?

During computerized adaptive testing (CAT), items are selected continuously according to the test‐taker's estimated ability. The traditional method of attaining the highest efficiency in ability estimation is to select items of maximum Fisher information at the currently estimated ability. Test security has become a problem because high‐discrimination items are more likely to be selected and become overexposed. So, there seems to be a tradeoff between high efficiency in ability estimations and balanced usage of items. This series of four studies with simulated data addressed the dilemma by focusing on the notion of whether more or less discriminating items should be used first in CAT. The first study demonstrated that the common maximum information method with Sympson and Hetter (1985) control resulted in the use of more discriminating items first. The remaining studies showed that using items in the reverse order (i.e., less discriminating items first), as described in Chang and Ying's (1999) stratified method had potential advantages: (a) a more balanced item usage and (b) a relatively stable resultant item pool structure with easy and inexpensive management. This stratified method may have ability‐estimation efficiency better than or close to that of other methods, particularly for operational item pools when retired items cannot be totally replenished with similar highly discriminating items. It is argued that the judicious selection of items, as in the stratified method, is a more active control of item exposure, which can successfully even out the usage of all items.

A Comparison of Item Exposure Control Methods in Computerized Adaptive Testing

Two new methods for item exposure control were proposed. In the Progressive method, as the test progresses, the influence of a random component on item selection is reduced and the importance of item information is increasingly more prominent. In the Restricted Maximum Information method, no item is allowed to be exposed in more than a predetermined proportion of tests. Both methods were compared with six other item‐selection methods (Maximum Information, One Parameter, McBride and Martin, Randomesque, Sympson and Hetter, and Random Item Selection) with regard to test precision and item exposure variables. Results showed that the Restricted method was useful to reduce maximum exposure rates and that the Progressive method reduced the number of unused items. Both did well regarding precision. Thus, a combined Progressive‐Restricted method may be useful to control item exposure without a serious decrease in test precision.