Testing for Bias in Order Assignment

Abstract

Many real-life situations require a set of items to be repeatedly placed in a random sequence. In such circumstances, it is often desirable to test whether such randomization indeed obtains, yet this problem has received very limited attention in the literature. This paper articulates the key features of this problem and presents three “untargeted” tests that require no a priori information from the analyst. These methods are used to analyze the order in which lottery numbers are drawn in Powerball, the order in which contestants perform on American Idol, and the order of candidates on primary election ballots in Texas and West Virginia. In this last application, multiple deviations from full randomization are detected, with potentially serious political and legal consequences. The form these deviations take varies, depending on institutional factors. This sometimes necessitates the use of tests that exchange power for increased robustness.