Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative

Abstract

This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation’s Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).