Controlling Item Exposure Research Articles

Simulated computer adaptive testing method choices for ability estimation with empirical evidence

<p><span lang="EN-US">Computer adaptive testing (CAT) is a technological advancement for educational assessments that requires thorough feasibility studies through computer simulations to ensure strong testing foundations. This advancement is especially germane in Africa being adopters of technology, and this should not be done blindly without empirical evidence. A quasi-experimental design was adopted for this study to establish methodological choices for CAT ability estimation. Five thousand candidates were simulated with 100 items simulate through the three-parameter logistic model. The simulation design stipulated a fixed-length test of 30 items, while examinee characteristics were drawn from a normal distribution with a mean of 0 and a standard deviation of 1. Also, controls for the simulation were set not to control item exposure or to use the progressive restricted method. Data gathered were analyzed using descriptive statistics (mean and standard deviation) and inferential statistics (Two-way multivariate analysis of variance: MANOVA) for testing the generated hypotheses. This study provided empirical evidence for choosing ability estimation methods for CAT as part of the efforts geared towards designing accurate testing programs for use in higher education.</span></p>

New Item Selection Method Accommodating Practical Constraints in Cognitive Diagnostic Computerized Adaptive Testing: Maximum Deviation and Maximum Limitation Global Discrimination Indexes.

Maximum deviation global discrimination index (MDGDI) is a new item selection method for cognitive diagnostic computerized adaptive testing that allows for attribute coverage balance. We developed the maximum limitation global discrimination index (MLGDI) from MDGDI, which allows for both attribute coverage balance and item exposure control. MLGDI can realize the attribute coverage balance and exposure control of the item. Our simulation study aimed to evaluate the performance of our new method against maximum global discrimination index (GDI), modified maximum GDI (MMGDI), standardized weighted deviation GDI (SWDGDI), and constraint progressive with SWDGDI (CP_SWDGDI). The results indicated that (1a) under the condition of realizing the attribute coverage balance, MDGDI had the highest attribute classification accuracy; (1b) when the selection strategy accommodated the practical constraints of the attribute coverage balance and item exposure control, MLGDI had the highest attribute classification accuracy; (2) adding the item exposure control mechanism to the item selection method reduces the classification accuracy of the attributes of the item selection method; and (3) compared with GDI, MMGDI, SWDGDI, CP_SWDGDI, and MDGDI, MLGDI can better achieve the attribute-coverage requirement, control item exposure rate, and attribute correct classification rate.

Investigating Item Exposure Control Methods in Computerized Adaptive Testing

This study aims to investigate the effects of item exposure control methods on measurement precision and on test security under various item selection methods and item pool characteristics. In this study, the Randomesque (with item group sizes of 5 and 10), Sympson-Hetter, and Fade-Away methods were used as item exposure control methods. Moreover, in order to establish a comparison baseline for other methods, the noexposure control condition was also included in the research. As item selection methods, Maximum Fisher Information, a-Stratification, and Gradual Maximum Information Ratio were employed. While a parameters were generated from a uniform distribution ranging between 0.50 and 2.00 and c parameters were generated from a uniform distribution ranging between 0.05 and 0.20 for both item pools, b parameters were generated from a uniform distribution ranging between -3.00 and +3.00 for medium difficulty item pool, and from a standard normal N(2, 1.5) distribution for high difficulty item pool. Based on the research findings, there were no great differences between the values of indicators used in determining measurement precision when the item exposure control methods were used. On the other hand, in terms of test security, it was found that in general, the Fade-Away Method for controlling item exposure yielded better results than did the other methods in reducing the skewness of item pool utilization and the test overlap.

Variable-Length Computerized Adaptive Testing Based on Cognitive Diagnosis Models

Interest in developing computerized adaptive testing (CAT) under cognitive diagnosis models (CDMs) has increased recently. CAT algorithms that use a fixed-length termination rule frequently lead to different degrees of measurement precision for different examinees. Fixed precision, in which the examinees receive the same degree of measurement precision, is a major advantage of CAT over nonadaptive testing. In addition to the precision issue, test security is another important issue in practical CAT programs. In this study, the authors implemented two termination criteria for the fixed-precision rule and evaluated their performance under two popular CDMs using simulations. The results showed that using the two criteria with the posterior-weighted Kullback–Leibler information procedure for selecting items could achieve the prespecified measurement precision. A control procedure was developed to control item exposure and test overlap simultaneously among examinees. The simulation results indicated that in contrast to no method of controlling exposure, the control procedure developed in this study could maintain item exposure and test overlap at the prespecified level at the expense of only a few more items.

Investigating the relationship between item exposure and test overlap: Item sharing and item pooling

To date, exposure control procedures that are designed to control item exposure and test overlap simultaneously are based on the assumption of item sharing between pairs of examinees. However, examinees may obtain test information from more than one examinee in practice. This larger scope of information sharing needs to be taken into account in refining exposure control procedures. To control item exposure and test overlap among a group of examinees larger than two, the relationship between the two indices needs to be identified first. The purpose of this paper is to analytically derive the relationships between item exposure rate and each of the two forms of test overlap, item sharing and item pooling, for fixed-length computerized adaptive tests. Item sharing is defined as the number of common items shared by all examinees in a group, while item pooling is the number of overlapping items that an examinee has with a group of examinees. The accuracy of the derived relationships was verified using numerical examples. The relationships derived will lay the foundation for future development of procedures to simultaneously control item exposure and item sharing or item pooling among a group of examinees larger than two.

Controlling item exposure and test overlap on the fly in computerized adaptive testing

This paper proposes an on-line version of the Sympson and Hetter procedure with test overlap control (SHT) that can provide item exposure control at both the item and test levels on the fly without iterative simulations. The on-line procedure is similar to the SHT procedure in that exposure parameters are used for simultaneous control of item exposure rates and test overlap rate. The exposure parameters for the on-line procedure, however, are updated sequentially on the fly, rather than through iterative simulations conducted prior to operational computerized adaptive tests (CATs). Unlike the SHT procedure, the on-line version can control item exposure rate and test overlap rate without time-consuming iterative simulations even when item pools or examinee populations have been changed. Moreover, the on-line procedure was found to perform better than the SHT procedure in controlling item exposure and test overlap for examinees who take tests earlier. Compared with two other on-line alternatives, this proposed on-line method provided the best all-around test security control. Thus, it would be an efficient procedure for controlling item exposure and test overlap in CATs.

Predicting item exposure parameters in computerized adaptive testing

The purpose of this study is to find a formula that describes the relationship between item exposure parameters and item parameters in computerized adaptive tests by using genetic programming (GP) - a biologically inspired artificial intelligence technique. Based on the formula, item exposure parameters for new parallel item pools can be predicted without conducting additional iterative simulations. Results show that an interesting formula between item exposure parameters and item parameters in a pool can be found by using GP. The item exposure parameters predicted based on the found formula were close to those observed from the Sympson and Hetter (1985) procedure and performed well in controlling item exposure rates. Similar results were observed for the Stocking and Lewis (1998) multinomial model for item selection and the Sympson and Hetter procedure with content balancing. The proposed GP approach has provided a knowledge-based solution for finding item exposure parameters.

A Strategy for Controlling Item Exposure in Multidimensional Computerized Adaptive Testing

Although computerized adaptive tests have enjoyed tremendous growth, solutions for important problems remain unavailable. One problem is the control of item exposure rate. Because adaptive algorithms are designed to select optimal items, they choose items with high discriminating power. Thus, these items are selected more often than others, leading to both overexposure and underutilization of some parts of the item pool. Overused items are often compromised, creating a security problem that could threaten the validity of a test. Building on a previously proposed stratification scheme to control the exposure rate for one-dimensional tests, the authors extend their method to multidimensional tests. A strategy is proposed based on stratification in accordance with a functional of the vector of the discrimination parameter, which can be implemented with minimal computational overhead. Both theoretical and empirical validation studies are provided. Empirical results indicate significant improvement over the commonly used method of controlling exposure rate that requires only a reasonable sacrifice in efficiency.

Automated Simultaneous Assembly of Multistage Testlets for a High-Stakes Licensing Examination

Many challenges exist for high-stakes testing programs offering continuous computerized administration. The automated assembly of test questions to exactly meet content and other requirements, provide uniformity, and control item exposure can be modeled and solved by mixed-integer programming (MIP) methods. A case study of the computerized licensing examination of the American Institute of Certified Public Accountants is offered as one application of MIP techniques for test assembly. The solution illustrates assembly for a computer-adaptive multistage testing design. However, the general form of the constraint-based solution can be modified to generate optimal test designs for paper-based or computerized administrations, regardless of the specific psychometric model. An extension of this methodology allows for long-term planning for the production and use of test content on the basis of an exact psychometric test designs and administration schedules.

Controlling Item Exposure and Test Overlap in Computerized Adaptive Testing

This article proposes an item exposure control method, which is the extension of the Sympson and Hetter procedure and can provide item exposure control at both the item and test levels. Item exposure rate and test overlap rate are two indices commonly used to track item exposure in computerized adaptive tests. By considering both indices, item exposure can be monitored at both the item and test levels. To control the item exposure rate and test overlap rate simultaneously, the modified procedure attempted to control not only the maximum value but also the variance of item exposure rates. Results indicated that the item exposure rate and test overlap rate could be controlled simultaneously by implementing the modified procedure. Item exposure control was improved and precision of trait estimation decreased when a prespecified maximum test overlap rate was stringent.

Strategies for Controlling Item Exposure in Computerized Adaptive Testing With the Generalized Partial Credit Model

Choosing a strategy for controlling item exposure has become an integral part of test development for computerized adaptive testing (CAT). This study investigated the performance of six procedures for controlling item exposure in a series of simulated CATs under the generalized partial credit model. In addition to a no-exposure control baseline condition, the randomesque, modified-within-.10-logits, Sympson-Hetter, conditional Sympson-Hetter, a-stratified with multiple-stratification, and enhanced a-stratified with multiple-stratification procedures were implemented to control exposure rates. Two variations of the randomesque and modified-within-.10-logits procedures were examined, which varied the size of the item group from which the next item to be administered was randomly selected. The results indicate that although the conditional Sympson-Hetter provides somewhat lower maximum exposure rates, the randomesque and modified-within-.10-logits procedures with the six-item group variation have great utility for controlling overlap rates and increasing pool utilization and should be given further consideration.

A-Stratified Multistage Computerized Adaptive Testing

Computerized adaptive tests (CAT) commonly use item selection methods that select the item which provides maximum information at an examinees estimated trait level. However, these methods can yield extremely skewed item exposure distributions. For tests based on the three-parameter logistic model, it was found that administering items with low discrimination parameter (a) values early in the test and administering those with high a values later was advantageous; the skewness of item exposure distributions was reduced while efficiency was maintained in trait level estimation. Thus, a new multistage adaptive testing approach is proposed that factors a into the item selection process. In this approach, the items in the item bank are stratified into a number of levels based on their a values. The early stages ofa test use items with lower as and later stages use items with higher as. At each stage, items are selected according to an optimization criterion from the corresponding level. Simulation studies were performed to compare a-stratified CATs with CATs based on the Sympson-Hetter method for controlling item exposure. Results indicated that this new strategy led to tests that were well-balanced, with respect to item exposure, and efficient. The a-stratifiedCATs achieved a lower average exposure rate than CATs based on Bayesian or information-based item selection and the Sympson-Hetter method.

A Comparison of Item Exposure Control Methods in Computerized Adaptive Testing

Two new methods for item exposure control were proposed. In the Progressive method, as the test progresses, the influence of a random component on item selection is reduced and the importance of item information is increasingly more prominent. In the Restricted Maximum Information method, no item is allowed to be exposed in more than a predetermined proportion of tests. Both methods were compared with six other item‐selection methods (Maximum Information, One Parameter, McBride and Martin, Randomesque, Sympson and Hetter, and Random Item Selection) with regard to test precision and item exposure variables. Results showed that the Restricted method was useful to reduce maximum exposure rates and that the Progressive method reduced the number of unused items. Both did well regarding precision. Thus, a combined Progressive‐Restricted method may be useful to control item exposure without a serious decrease in test precision.

Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing

The interest in the application of large-scale adaptive testing for secure tests has served to focus attention on issues that arise when theoretical advances are made operational. One such issue is that of ensuring item and pool security in the continuous testing environment made possible by the computerized admin-istration of a test, as opposed to the more periodic testing environment typically used for linear paper-and-pencil tests. This article presents a new method of controlling the exposure rate of items conditional on ability level in this continuous testing environment. The properties of such conditional control on the exposure rates of items, when used in conjunction with a particular adaptive testing algorithm, are explored through studies with simulated data.

Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing

Controlling Item Exposure Conditional on Ability in Computerized Adaptive Testing

Effects of Controlling Item Exposure in Achievement Testing

Effects of Controlling Item Exposure in Achievement Testing