The barking dog that bites: Test score volatility and school rankings in Punjab, Pakistan

Abstract

This paper is the first to explore the implications of test score volatility for school accountability policies in a lower-middle-income country. Using two new datasets from the province of Punjab, Pakistan, we find that rankings based on school-level averages of students’ test scores fluctuate considerably from one year to the next due to factors unrelated to school quality. First, most variation in test scores is within, not between schools, allowing noise to play a large part in determining school rankings. Second, variation in school test scores is partially attributable to fluctuations in the characteristics of student cohorts. Third, an even larger share is explained by one-time shocks. Fourth, these problems are more pronounced in small schools. As a result, rankings that rely on test score levels or changes often fail to identify top-performing or rapidly-improving schools, since they are influenced by factors beyond schools’ control.