Automated Test-Form Generation

Abstract

In automated test assembly (ATA), the methodology of mixed-integer programming is used to select test items from an item bank to meet the specifications for a desired test form and optimize its measurement accuracy. The same methodology can be used to automate the formatting of the set of selected items into the actual test form. Three different cases are discussed: (i) computerized test forms in which the items are presented on a screen one at a time and only their optimal order has to be determined; (ii) paper forms in which the items need to be ordered and paginated and the typical goal is to minimize paper use; and (iii) published test forms with the same requirements but a more sophisticated layout (e.g., double-column print). For each case, a menu of possible test-form specifications is identified, and it is shown how they can be modeled as linear constraints using 0–1 decision variables. The methodology is demonstrated using two empirical examples. Test development is a comprehensive process involving the typical stages of itembank design, item writing, field testing, and calibration, as well as test assembly and test-form generation. Guidelines for the various activities in these stages can be found, for instance, in Downing and Haladyna (2006) and Schmeiser and Welch (2006). Any step to automation of these stages is welcome, especially when it leads to optimization of the results and reduction of the amount of work. An area where such automation has made important progress lately is test assembly. Automated test assembly (ATA) involves the selection of the items for one or more test forms from an item bank subject to sets of content and statistical specifications while optimizing their measurement properties. The automation has become possible through the application of the methodology of mixed-integer programming (MIP). The core of the application consists of the definition of 0‐1 decision variables for the selection of the items, the modeling of the test specifications as a set of constraints with an objective function that is linear in the variables, and the use of a software package with an MIP solver to find the optimal solution, which is the combination of 0s and 1s for the variables for the items that identifies the optimal set in the bank. The use of MIP is extensively reviewed for a wide variety of test assembly problems in van der Linden (2005, chaps. 1‐9). In addition, introductions to ATA and a few worked examples can be found in Cor, Alves, and Gierl (2008, 2009). For readers not familiar with the methodology, a brief example is given in the Appendix. The same methodology can be used to design item banks and manage the process of item writing. The process then involves the definition of a design space for the items (i.e., the Cartesian product of all relevant item attributes) and the introduction