Abstract
In many high-stakes testing programs, testlets are used to increase efficiency. Since responses to items belonging to the same testlet not only depend on the latent ability, but also on correct reading, understanding, and interpretation of the stimulus, the assumption of Local Independence does not hold. Testlet Response Theory models have been developed to deal with this dependency. For both logit and probit testlet models, a random testlet effect is added to the standard logit and probit IRT models. Even though this testlet effect might make the IRT models more realistic, application of these models in practice leads to new questions, for example in automated test assembly. In many test assembly models, goals have been formulated for the amount of information the test should provide about the candidates. The amount of Fisher Information is often maximized or it has to meet a pre-specified target. Since Testlet Response Theory models have a random testlet effect, Fisher Information contains a random effect as well. The question arises as to how this random effect in automated test assembly should be dealt with. A method based on robust optimization techniques for dealing with uncertainty in test assembly due to random testlet effects is presented. The method is applied in the context of a high-stakes testing program and the impact of this robust test assembly method is studied. Results are discussed, advantages of the use of robust test assembly are mentioned, and recommendations about the use of the new method are given.
Highlights
In many tests, a reading passage, graph, video fragment, or simulation is presented to a test taker; and after reading the passage, studying the graph, watching the video fragment, or participating in the simulation, the test taker is presented with a number of items pertaining to the stimulus
In the test assembly process, both a lower and an upper bound for the target information function (TIF) had to be met
It has to be mentioned that any test that met the specifications would have been acceptable as a solution to the first test assembly model
Summary
A reading passage, graph, video fragment, or simulation is presented to a test taker; and after reading the passage, studying the graph, watching the video fragment, or participating in the simulation, the test taker is presented with a number of items pertaining to the stimulus. ATA methods typically model the test assembly problem as a mathematical programming problem that maximizes an objective function, for example, related to the amount of information in the test, while a number of constraints, for example, related to the test specifications, have to be met. The first step in ATA is to formulate the test assembly problem as a linear programming model These models are characterized by decision variable xi ∈ {0,1} for i = 1, ..., I, which denotes whether an item i is selected for the test (xi = 1) or not (xi = 0). This implies that the uncertainty in at most five of the items is assumed to impact the test assembly problem.
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
