AN EXPERIMENT IN THE APPLICATION OF AN AUTOMATED ITEM SELECTION METHOD TO REAL DATA*

Abstract

A previously developed method of automatically selecting items for inclusion in a test subject to constraints on item content and statistical properties is applied to real data. Two tests are first assembled by experts in test construction who normally assemble such tests on a routine basis. Using the same pool of items and constraints articulated by test construction experts the same two tests are reassembled automatically. The manual and automatic assemblies are compared by test specialists who were not involved in the original manual assembly. Based on this experiment, barriers to future progress in the improvement of automatic test assembly methods seem not to be in the development of different algorithms, nor in the improvement of computer time and cost. Rather, the focus of future improvements in the process of automatic test assembly lies more in the direction of complete specifications of constraints on item selection and detailed coding of item properties.