At the Nexus of Observational and Experimental Research: Theory, Specification, and Analysis of Experiments with Heterogeneous Treatment Effects

Abstract

Experimentalists are increasingly examining heterogeneous treatment effects, in which observed individual-level characteristics are hypothesized to moderate an experimental treatment effect. Such work places researchers at the nexus of experimental and observational approaches. In this paper, we discuss the theoretical and statistical issues that can arise in testing such hypotheses. We note that inclusion of an observed (as opposed to randomly-assigned) moderator introduces the possibility of confounds that are commonplace in observational data analysis but too-easily ignored in experimental data analysis. We simulate several different data generating processes that include heterogeneous treatment effects, and we discuss the implications of various statistical models. We aim to provide researchers who examine heterogeneous treatment effects with background and advice that enable them to identify where common issues may arise and to develop research designs and implement statistical tests that will mitigate them.