Neyman's Repeated Sampling Approach to Completely Randomized Experiments

Abstract

INTRODUCTION In the last chapter we introduced the Fisher Exact P-value (FEP) approach for assessing sharp null hypotheses. As we saw, a sharp null hypothesis allowed us to fill in the values for all missing potential outcomes in the experiment. This was the basis for deriving the randomization distributions of various statistics, that is, the distributions induced by the random assignment of the treatments given fixed potential outcomes under that sharp null hypothesis. During the same period in which Fisher was developing this method, Neyman (1923, 1990) was focused on methods for the estimation of, and inference for, average treatment effects, also using the distribution induced by randomization, sometimes in combination with repeated sampling of the units in the experiment from a larger population of units. At a general level, he was interested in the long-run operating characteristics of statistical procedures under both repeated sampling from the population and randomized assignment of treatments to the units in the sample. Specifically, he attempted to find point estimators that were unbiased, and also interval estimators that had the specified nominal coverage in large samples. As noted before, his focus on average effects was different from the focus of Fisher; the average effect across a population may be equal to zero even when some, or even all, unit-level treatment effects differ from zero. Neyman's basic questions were the following. What would the average outcome be if all units were exposed to the active treatment, ȳ(1) in our notation? How did that compare to the average outcome if all units were exposed to the control treatment, ȳ(0) in our notation? Most importantly, what is the difference between these averages, the average treatment effect (Here we use the subscript fs to be explicit about the fact that the estimand is the finite-sample average treatment effect. Later we use the notation τ sp to denote the super-population average treatment effect.) Neyman's approach was to develop an estimator of the average treatment effect and derive its mean and variance under repeated sampling.