The Proficiency Illusion

Abstract

ONE OF the most controversial provisions in No Child Left Behind is that it lets each state define Given the states' primary role in education and our tradition of local control, some find this approach appropriate. For others, it means that there will be a babel of definitions and a total lack of comparability across state lines. Now comes The Proficiency Illusion, an interesting variation the theme from John Cronin, Michael Dahlin, Deborah Adkins, and Gage Kingsbury of the Northwest Evaluation Association (NWEA). This report, sponsored by the Thomas B. Fordham Foundation, with a foreword by Chester Finn and Michael Petrilli, doesn't cure the worst problem of defining because it uses cut scores that result only in reports of percentage proficient. But it does provide an interesting new perspective how the definitions of vary around the nation. Some have wanted to use NAEP as the unifying metric, the gold standard against which to measure state standards. But NAEP isn't satisfactory for this purpose, because it is a national test not specifically oriented to any state's curriculum or standards. As long as states have different curricula, there will be varying degrees of mismatch between NAEP and state tests. Over the years, NWEA has developed Measures of Academic Progress (MAP) tests. These tests are used by a number of states, and the item pool consists of 5,200 items. But using item response theory, NWEA places all MAP items a common scale. Thus we can ask how well Colorado does its own state test and how well it does MAP tests, and we can ask the same about Wisconsin. Then we can compare the performance of the two states, even though they have different tests and the MAP tests for them use some different items. Those different MAP items are still a common scale. Sufficient numbers of students take MAP tests to permit such comparisons for 26 states. The comparison of most interest is in the location of the cut score. For each state, NWEA looked at the proportion of students attaining the state test. It then asked, What score the MAP would produce the same proportion of students? If, say, 75% of the students in a state were the state test, the question would be What score the MAP test would declare 75% of the students proficient? The MAP percentile score that yielded the appropriate of would be reported for NCLB purposes. The results show that state test cut scores vary enormously. Figure 1 indicates that in Texas a student scoring at the 12th percentile the MAP tests would be labeled proficient. (I ignore Colorado's lower score because that state has a tougher standard for internal use than for NCLB purposes.) By contrast, for a student to be called in California, he or she would have to score at the 61st percentile the MAP. This is where it gets tricky and Finn and Petrilli present some misleading exposition. Here they are discussing the of a fourth-grader in Michigan, which has one of the lowest cut scores in reading, versus the in South Carolina, Massachusetts, and California, which have the highest: This fourth-grader [Susie Smith] lives in suburban Detroit, and her parents get word that she has passed Michigan's state test. She's proficient in reading and math. Mr. and Mrs. Smith understandably take this as good news; their daughter must be on grade level and track to do well in later grades of school, maybe even go to college ... [What they don't know is] that proficiency Michigan's state tests has little meaning outside the Wolverine State's borders; if Susie lived in California or Massachusetts or South Carolina, she would have missed the proficiency cut-off by a mile. Maybe, maybe not. Here are the study's authors writing the same issue: A student who can navigate the California or Massachusetts reading requirements has clearly achieved a much different of competence than has one who just meets the Colorado or Wisconsin standard. …