Evaluating the Impact of Releasing an Item Pool on a Test's Empirical Characteristics

Abstract

Protecting the security of examination questions is an important task for high-stakes examining boards/agencies and university programs. To maintain the security of questions, examining boards and university programs use a combination of prevention, detection, and enforcement strategies. A common prevention strategy is to establish a number of controls on access to questions; however, restricting access can motivate examinees to try harder to reconstruct questions that may appear on future versions of the test. Moreover, access to study materials by some groups and not others can present a challenge to the fairness of examinations. The aim of this study was to evaluate the empirical stability of test characteristics. Specifically, the primary research objective was to investigate the empirical stability of the items and test forms of a written examination before and after a specific policy decision was implemented. As a response to both of these concerns, this article describes a study that evaluated how psychometric (i.e., statistical) properties of test forms and individual questions might be affected by publicly releasing a larger number of questions from an item question pool. A series of analyses were conducted, including item drift to evaluate stability of the characteristics. The results suggest that empirical characteristics of the test forms and individual questions have remained relatively stable since the release policy was implemented. Specifically, statistical properties of the test forms have continued to perform similarly to test forms that were constructed prior to the release. Although the results of this study were promising, the context of this specific testing program may have offered additional protections such as a limited number of administrations that others may not. Therefore, testing/examining agencies and university programs may want to consider this strategy with appropriate caution.