Monte Carlo detection of examinees with item preknowledge

Abstract

The general case of item preknowledge (IP) is studied, where groups of examinees had access to compromised subsets of items from an administered test prior to the exam. Nothing is known about these groups and subsets (terms groups and subsets are chosen to clearly distinguish between subsets of examinees and subsets of items). When only item scores are given, according to study by Karabatsos (Appl Meas Educ 16:277–298, 2003), the highest detection rate of examinees with IP is observed with TW statistic developed by Trabin and Weiss (in: Weiss (ed) New horizons in testing, Academic Press, New York, 1983). This paper develops a new Monte Carlo approach for detecting examinees with IP by estimating mean of a performance gain on a random sample of items (drawn from the administered test) relative to another random sample. Two samples are constructed such that for an examinee without IP, the gain should be low; meanwhile, for an examinee with IP, if the first sample has more compromised items than the second sample, the gain should not be low. Comparison study with TW using data simulating the general case of IP demonstrated a dramatic improvement of detection rates (over five times on average) when using the Monte Carlo approach. Even higher improvement (over 25 times) was observed in experiments with two publicly available real datasets. Recommendations for practitioners and extensions of the Monte Carlo approach are provided.