Counternull sets in randomized experiments

Abstract

Consider a statistical analysis for a randomized experiment that draws inferences based on hypothesis testing. In such settings, the plausibility of a null hypothesis is often examined using a p-value associated with a test statistic. In controlled experiments such as the ones that Bob conducts, Fisher-exact p-values are available and should be used to help evaluate results rather than the more commonly reported asymptotic p-values associated with common statistical tests (e.g., t-tests). Low p-value typically indicates some evidence against the null hypothesis, and when p-values are large, no definite statistical conclusions should be drawn! To avoid misinterpretations when testing a null hypothesis, we will present the "counternull value”, which was first defined by Rosenthal and Rubin (1994) and illustrate these concepts using some of Bob's chamber experiments.