Evaluating an Evaluation Program: Unknowable Outcomes

Abstract

The Texas Education Agency (TEA) supports a number of programs to improve student outcomes within the state. To this end, the TEA has made agreements with a number of private entities to provide after-school enrichment programs. The programs should provide academic support, enrich the lives of students, prepare students for college and a career, and build family engagement in education. By providing the programs, TEA expected that students will improve in academic performance, grade promotion, attendance, behavior, and family engagement. The overarching goal is to ensure that all students graduate from high school and are ready for college and a career. The State, and its educational contractors, are interested in ensuring that the programs meet the needs of the students and satisfy program goals. The State, therefore, mandates a yearly evaluation of the after-school programs. This evaluation focuses on five areas: resource availability, implementation practices, program activities, student targeting, and improved student outcomes. At issue is how well the available metrics capture program implementation, student outcomes, and program impacts. The metrics for the evaluations are a matter of state law. An educational consultant was hired to develop standards and “measures,” and these were approved by the Texas legislature. The metrics, however, are insufficient for effectively evaluating the after-school programs. Measures of student outcomes and program impacts, developed with Rasch analysis methods, are sorely needed but simply not possible with the data available; such attempts at measurement are also not likely to succeed on a post hoc basis. This paper summarizes the evaluation system for the after-school programs and highlights the problems with assessing outcomes and program effectiveness. This paper concludes with recommendations for future changes to evaluation methods.