Randomized controlled trials informing public policy: Lessons from project STAR and class size reduction

Abstract

Randomized controlled trials (RCTs) and related research strategies are increasingly seen as the preferred methodology for evaluating policy interventions, but a single-minded focus on identifying causal effects limits their capacity to support actual policy decisions. A detailed look at Project STAR illustrates this point and offers some possible correctives. Initiated by the Tennessee legislature to help decide whether to enact state-wide class size reduction (CSR), STAR compared the test scores of students randomly assigned to classes of different size. It addressed a well-formed research question, but focused narrowly on refining a single link in a long chain of evidence necessary to address the policy question at hand: whether CSR would be a good use of a large increase in education spending. It disregarded the limitations of test scores as indicators of education quality and ignored general equilibrium effects on teacher quality and salaries. Moreover, the emphasis it placed on estimating average CSR effects in a given setting diverted attention from the heterogeneity of these effects and the conditions that mediated their impact, limiting its external validity. These observations continue to be relevant for the design of policy-oriented research, and for the academic training of empirical economists.