Validity of Majority-Minority Performance Gaps Measurements on PISA Tests

Abstract

Every three years, with the publication of the PISA survey results, a ritual of comparisons is beginning: Growing Gaps, Major Drop and Huge Gaps in Ability are some of the popular headlines describing the performance gaps between population groups within and between countries. The stated goal of the PISA survey is to provide comparable results between countries and populations and across cohorts. In this paper, we elucidate the extent to which heterogeneous engagement measured by response time and performance endurance during the test affects the measurement of performance gaps between minority and majority ethnicities in Israel, Spain and the United States. We find, for example, that white-Black and white-Hispanic gaps increase by more than 20% from the first to the second test chapter due to the better endurance of white students. This increase in gaps was seen in eight of the eleven minority groups. We recommend policy makers and researchers to be careful when using low-stakes tests for assessing performance gaps between ethnicity groups.