Value-Added Assessment Finding: Poor Kids Get Poor Teachers

Abstract

THE SUMMER 2004 issue of Research Points, a four-page flier from the American Educational Research Association, opens with the fundamental case for value-added assessment: Today's accountability systems place the blame on schools for inadequate student academic achievement, which seems unfair to many people. They believe that family background and the socioeconomic mix of students in the classroom exert such a strong influence on student learning that teachers and schools can have only a limited effect. Important research from the 1960s appeared to bolster that view, but recent studies show clearly that a student can learn more from one teacher than from another and that teachers and schools matter. So the question now is not whether schools and teachers can make a difference, but how much they affect student learning. important research from the 1960s, of course, would have been Equality of Educational Opportunity, more commonly referred to simply as Coleman Report. inherent appeal of Value-Added Assessment (VAA) is obvious: instead of relying on a gross end-of-year test and the comparison of successive cohorts over time (the model currently in use for NCLB), VAA tracks the growth of individual students. Ideally, it permits a student's growth in any one year to be compared to that student's growth history in previous years. Given a student's achievement history, we can predict how much growth to expect in a given year. We can then examine the actual growth. That allows us to see any deviations, plus or minus, from the predicted growth and to try to account for the deviations in terms of the impact of family, community, or teachers. As I noted in the May Research column, proponents of the best-known value-added system, the Tennessee Value Added Assessment System (TVAAS), have claimed that its results are independent of the most common demographic variables. In the Spring 2004 issue of the Journal of Educational and Behavioral Statistics (JEBS), Dale Ballou of Vanderbilt University and William Sanders and Paul Wright of the SAS Institute compare results obtained by not controlling for such variables and by controlling for them. They find minimal differences between the controlled and uncontrolled results. This finding alone is not likely to satisfy most researchers. As Research Points observes, Identifying more or less effective teachers using value-added measures is subject to great statistical uncertainty, and research offers very little guidance in determining how much weight to give value-added assessments. Consequently, many researchers are skeptical about relying on value-added estimates in high-stakes personnel decisions. Of course, no matter how sophisticated the statistical modeling of a value-added system is, the old maxim garbage-in, garbage-out applies. TVAAS was originally developed using off-the-shelf items from the CTBS. I'm certain that all Kappan readers have their own horror stories about improper methods used to raise test scores. Beyond that, though, there is the finding reported by Daniel Koretz, Robert Linn, Lorrie Shepard, and David Frisbie in 1991 that test scores don't seem to generalize much beyond the specific test in question. This may be because The Real World Is More Complicated Than We Would Like, to borrow the title of an article by Michigan State University's Mark Reckase, which also appears in the spring 2004 issue of JEBS. Reckase points out, among other things, that the various models of VAA use a simple difference score from year to year and that this presumes similar test content from year to year. But, asks Reckase, what if a teacher emphasizes pre-algebra skills in a year when the test shifts to coordinate geometry? differences in test scores for the two years will be hard to interpret. It might be that the teacher should have shifted her instruction to align with the test, or it might be that her emphasis was appropriate given her students' level of achievement. …