“Beware of Geeks Bearing Formulas”

Abstract

Listen

Translate article

In fall 2007, Ellen Foote, principal of Intermediate School 289, got an unexpected surprise when she opened her mail. The New York City Department of Education had graded its on an A-F scale for first time, using a method it would later call the most sophisticated accountability system in country (Gootman 2008). What made New York City's approach so different from No Child Left Behind-style accountability was that primary factor dictating a school's grade was students' year-to-year on standardized tests. To further emphasize to apples comparisons between schools, were compared not only with all in city, but also with demographically similar to their own. Yet, Foote's decorated school--the only federally recognized Blue Ribbon middle school in city and a regular on best schools list in a popular school guidebook--had been stamped with an unsavory D (Medina and Gebeloff 2008). One year later, Department of Education issued its school progress reports a second time. And again, New Yorkers scratched their heads trying to make sense of results. What does it mean, for example, when 77% of elementary and middle that received an F in 2007 jump to an or a B in 2008? Mayor Michael Bloomberg had a resolute answer to this question: Not a single school failed again. ... The fact of matter is it's working (Medina and Gebeloff 2008). But Ellen Foote, whose school had received an second time around, wasn't so sure. Her school was same, but her grade was very different. A school doesn't move from a D to an in one year unless there is a flaw in measurement or standardized test itself, she said. We have not done anything differently, certainly not in response to progress report (Medina and Gebeloff 2008). Even less enthusiastic were education measurement experts asked to comment on wild grade swings between 2007 and 2008. Wrote Daniel Koretz (2008b), an educational psychologist at Harvard Graduate School of Education, [The New York City results show] far more instability from one year to next than could credibly reflect true changes in performance. ... It does not make sense for parents to choose schools, or for policy makers to praise or berate schools, for a rating that is so strongly influenced by error. Walt Haney, an education professor at Boston College, was similarly perplexed: These [grades] are showing dramatic changes that can have nothing to do with what is actually happening (Medina and Gebeloff 2008). Here was a district that set out to combat central problem with federal accountability system: over-reliance on simple passing rates and conflation of school and out-of-school influences on students' test performance. In short, New York City's system of measuring was intended to provide a better measure of school performance. Why, then, was this new system producing such dramatically different results from one year to next? The finer points of accountability systems arouse as much enthusiasm as an opportunity to watch plastic flowers grow. But as with all things, details are key to understanding New York City's curious grade vacillations, as well as promise and perils of growth models--systems designed to estimate progress individual students make over time--for school accountability. But there is no panacea here. While models improve on accountability systems in many ways, readers must also understand that a poorly designed is no better than poorly designed proficiency we have now. WHAT ARE GROWTH MODELS? By now, you've probably heard terms like growth model and value-added tossed around in faculty meetings or in education policy debates. In practice, these terms are used interchangeably, but it's worth distinguishing between two. …