Clear Away the Smoke and Mirrors of Value-Added

Abstract

Accurately measuring performance is important to any organization, including schools. State longitudinal data systems with annual student assessments are expanding the possibilities for measuring school and teacher performance. This can increase the potential for evaluating educators in ways that are fair and improve performance and student outcomes. Unfortunately, despite almost two decades of the new that links student test scores to schools, our performance measurement and accountability systems are still broken. Some of the problems--narrowing the curriculum and teaching to the test--have received widespread attention. Another critical problem is less widely recognized. The Principle of as I will call it, is that we should hold people accountable for what they can control. But accountability systems violate this principle when they focus on student attainment, or student achievement at a point in time. The problem with attainment has to do with a basic fact of human development: Knowledge and skill accumulate over each person's life. What adults know and can do depends on what they experienced as students, which in turn depends on experiences before school, all the way back to birth (Heckman 2006). Kindergarteners have vastly different early childhood experiences and readiness for school. For example, black students start kindergarten with scores that are already 22 percentile points behind white kindergartners (Fryer and Levitt 2004). Schools have not caused these starting-gate inequalities, because most students haven't set foot in a classroom before kindergarten. The gaps are so large and persistent that even effective schools don't completely overcome them. Not only are the gate inequalities large, but nonschool factors continue to influence children as they progress through school. So, while the new focus on student achievement was a step forward in some ways, both state and federal policies misuse test scores in ways that violate the Fundamental Principle of Accountability. These misuses have consequences. Performance measures based on student test score attainment are partly responsible for pressuring schools to exclude students from testing (Figlio 2005) and pushing out good teachers (Clotfelter et al. 2004) who are frustrated by a system that punishes them no matter how well they perform. It's also well known that school systems, especially urban school districts, spin their wheels in an endless cycle of reform and changes in curriculum, instruction, and leadership (Hess 1998). A continual cycle of reform might make sense if those schools were continually failing, but low-performing schools often aren't failing by more reasonable definitions. As I will show below, many low-attainment schools are actually high-performing, and the reverse is also true. While we have focused on low-attainment schools, high-attainment schools present an equally large, but hidden, problem. In these schools, students enter with high scores, so their attainment remains high almost no matter what the schools do. These schools often largely ignore external accountability. Fortunately, we can take steps to help avoid confounding out-of-school influences with school or teacher performance. The Basics of Value-Added Accountability Performance can be reasonably defined, according to the Fundamental Principle of Accountability, as what each school and teacher contributes to student outcomes. But how can we measure that? One simple way to measure performance is by subtracting the initial level of student achievement test score from the end-of-year test score. This is illustrated in Figure 1, in which each arrow represents the achievement growth of students in two hypothetical schools: Rockefeller Elementary and Smith Elementary. The bracket in the lower left corner indicates the differences in initial achievement--the starting gate inequality--and shows that Rockefeller students begin in a stronger position than Smith students. …