Now, whose schools are really better (or weaker) than Germany's? A multiple testing approach

Abstract

Using PIRLS (Progress in International Reading Literacy Study) data, we investigate which countries' schools can be classified as significantly better or weaker than Germany's as regards the reading literacy of primary school children. The ‘standard’ approach is to conduct separate tests for each country relative to the reference country (Germany) and to reject the null of equally good schools for all those countries whose p-value satisfies p i ⩽ 0.05. We demonstrate that this approach ignores the multiple testing nature of the problem and thus overstates differences between schooling systems by producing unwarranted rejections of the null. We employ various multiple testing techniques to remedy this problem. The results suggest that the ‘standard’ approach may overstate the number of significantly different countries by up to 30%.