Sampling and Assignment Mechanisms in Experiments, Surveys and Observational Studies, Correspondent Paper

Abstract

Summary A general framework is given for examining the role of mechanisms for treatment assignment and unit selection in experiments, surveys and observational studies. Conditions are established under which these mechanisms can be ignored for model-based inference. Examples are presented to show how inference can incorporate the mechanisms when the conditions do not hold. Two of the most important tasks facing an applied statistician are the design and analysis of sample surveys, and the design and analysis of experiments. Despite the many apparent links between surveys and experiments identified by Fienberg & Tanur (1987), and the contributions to both subjects by statisticians such as Yates and Cochran, the theory and methods of these two subjects appear quite different. Why should this be? Sample survey design and analysis has grown from the need for descriptive statistics in the government and commercial stores. In contrast experiments use statistics analytically to test hypotheses for example, and are carried out most frequently in the industrial and scientific sectors. Government surveys are still used primarily to describe the properties of a finite population at a fixed point in time but many commercial surveys attempt to explain relationships between variables that will hold in the future, so they are explanatory or analytic rather than descriptive. In the analysis of an explanatory survey or of an experiment one aim is to find evidence for a law-like relationship, for causal explanations rather than descriptions. Wold (1967) addressed the problems of reaching causal conclusions from nonexperimental data. He argued that experimental knowledge is reproducible knowledge, the reproducibility arising from the control exerted by the scientist over the assignment of treatments. It is the scientist's control over his material that is the most important distinction between surveys and experiments; in surveys there is control over sampling of units while in experiments there is control over the assignment of treatments to units. From this difference in control flows the difference in the approaches to the design and, more importantly, the analysis of surveys and experiments. Experiments can be carried out within surveys. Brewer et al. (1977) give examples of experimental designs at the pilot stage of a survey for testing questionnaire layouts, diary