Learning About and From Variation in Program Impacts Using Multisite Trials

Abstract

The present paper, which is intended for a diverse audience of evaluation researchers, applied social scientists, and research funders, provides a broad overview of the conceptual and statistical issues involved in using multisite randomized trials to learn about and from variation in program effects across individuals, across policy-relevant and theoretically relevant subgroups of individuals, and across program sites. Learning about variation in program effects involves detecting and quantifying this variation. Learning from variation in program effects involves studying the factors which predict or explain it. The paper is divided into four sections, plus a brief final discussion. The first section introduces the concepts and issues involved. Section 2 focuses on detecting and quantifying variation in effects of program assignment, which are often referred to as effects of intent to treat (ITT). Section 3 extends the discussion to variation in effects of program participation, which are often referred to as a complier average causal effect (CACE) or a local average treatment effect (LATE). Section 4 considers moderators of program effects (individual-level or site-level factors that influence the sign and magnitude of these effects) and mediators of program effects (individual-level or site-level “mechanisms” by which these effects are produced).