A Multilevel Multisite Outcomes-by-Implementation Evaluation of an Early Childhood Literacy Model

Abstract

The evaluation of an intervention delivered across multiple treatment sites presents a unique opportunity for evaluators to gauge the manner and degree to which the “impact” of treatment varies across implementation conditions and different target populations. However, the availability of implementation data for each treatment site, while presenting the opportunity for more sophisticated impact assessment, also presents an analytic challenge. In the following, multilevel growth models were applied to data describing students and classroom sites to demonstrate how the multilevel modeling framework can be used to analyze data obtained from a multisite program implementation. Results of the investigation indicated that increased adherence to the program model was not associated with more positive recipient outcomes. Further examination of the null finding indicated that the highest and lowest rates of literacy growth observed in the study were concentrated in several low-implementing sites. Implications for multisite evaluation design and practice are discussed.