Tips for Readers of Research: Handle Pass Rates with Care

Abstract

THE ACHIEVEMENT gap receives a lot of attention these days. It is, after all, alleged raison d'etre of No Child Left Behind. It is usually defined as difference between test scores of white students on one hand and those of black and Hispanic students on other. (Asian students almost always get ignored for some reason.) Unfortunately, educators and journalists alike have fallen into trap of reporting changes in pass rates on tests as if they were measures of achievement gap. They are not. At best, passing rates are poor proxies for real measures. The Massachusetts Department of Education; Michael Cohen and Matt Gandal of Achieve, Inc.; David Herszenhorn and Michael Winerip of New York Times; Mayor Michael Bloomberg of New York City; Chancellor Joel Klein of New York City Public Schools; and Achievement Alliance, a consortium of Education Trust, National Council of La Raza, Citizens' Commission on Civil Rights, Just for Kids, Inc., and Business Roundtable, have all reported passing rates as if they were scores. They are not. Consider data from Massachusetts Department of Education, reproduced in Cohen and Gandal's Do Graduation Tests Measure Up? Figure 1 shows passing rates for class of 2003 on Massachusetts Comprehensive Assessment System. On initial administration, when students were 10th-graders, blacks and Hispanics passed at rates far lower than whites: blacks, 37%; Hispanics, 29%; whites, 77%. By sixth retest, passing rates had become much more similar. Cohen and Gandal take these data as showing that the achievement gap closed dramatically. Probably not. Consider following hypothetical data. Hypothetical Data Score Needed to Pass = 60 Pass Average Rate Score Gap Black students, 2004 60% 62 White students, 2004 100% 78 16 Black students, 2005 70% 68 White students, 2005 100% 92 24 Between 2004 and 2005, black students improved both their passing rate (going from 60% to 70%) and their score (from 62 to 78). So far so good. But white students, while not improving their passing rate, improved their score even more, so much that black/white gap in scores (the proper metric) actually increased. There is another problem with conclusions in Massachusetts report. The 10th-graders and 12th-graders are not comparable. Somewhere between freshman and senior years, 18,000 students disappeared from Massachusetts' rolls. Some no doubt moved to other states. But others would have been retained in grade or would have dropped out, perhaps because of failing test. In either case, these low-scoring students are no longer part of class of 2003. Naturally, state officials, who like to brag about students rising to challenge of tests, don't like it when researchers call attention to such dishonest presentation of data, but when Anne Wheelock conducted appropriate analysis of data for Massachusetts for class of 2004, she concluded that state had little reason to brag. She found these passing rates: Class of 2004 Pass Rate All students 74%, not 96% White students 80%, not 98% Black students 59%, not 88% Hispanic students 54%, not 85% Asian students 89%, not 95% Note that gap for final pass rate for class of 2004 is nearly as large as 10th-grade gap was for class of 2003. This is not a trivial case of researchers making statistical mountains out of practical molehills. At least one of journalists cited above knows this. When I presented my hypothetical data to Times' Herszenhorn, he replied, Of course, that's exactly way it works.... We have done some analysis of mean scale scores, enough to know that strong gains in suburbs were masked by looking mostly at who crosses line. …