Opening Black Boxes of the Meta-Analyses: What Do the Underlying Studies Look like?

Abstract

Altogether the review of the research evidence left many unanswered questions and prompted us to turn to the individual program evaluations, on which the meta-analyses were based, before making more definite assessments. This is the focus of Chap. 6. We selected evaluations studies of 8 programs that had drawn our attention for various reasons: prominence of the programs in review studies (e.g. PATHS, and Tools of the Mind), the presence of more or less explicit program theories (e.g. Comer’s school development program, and Positive Action) and we also tried to document variation in low, medium or relatively high program effects. For each program at least one program evaluation was discussed, while for some programs several evaluation studies were referred to. In order to provide the reader with an impression of the contents of these programs, considerable space was given to descriptions of the program interventions. Intervention modes were most frequently curriculum documents, like lesson programs and scripted teaching approaches, specific teacher training for the program, and sometimes adapted modes of school organization, staff cooperation and parent involvement. Lack of explicit program theories was a weakness in the majority of the evaluation studies that were reviewed. Questions about the appropriateness of the outcome measures, as discussed in the previous chapter, were also manifest in these case studies. Many interventions are only assessed with no, or short-term follow up and program evaluations are quite heterogeneous in the choice of outcome measures and, in targeting specific student populations, there is a bias towards disadvantaged groups.