The danger of inadequate conceptualisation in PISA for education policy

Abstract

Due to the broad acceptance of the Programme for International Student Assessment (PISA) and other comparative studies as instruments of policymaking, its accuracy is essential. This article attempts to demonstrate omissions in the conceptualisation, and consequently in calculation and interpretation, of one of the central points of PISA 2006 and 2009. The authors point to the danger of such omissions for possible evidence-based education policymaking. With regard to the reproduction of social inequalities through education, we focus on variance in student performance between schools and within schools on the science scale (PISA 2006) and in the mother tongue (PISA 2009). The thesis of the PISA study is that there are countries (Finland, Iceland and Norway) where between only 14 and 29% of the average OECD variance (33%) is attributed to between-school variance – which is good – and there are countries (Germany, Bulgaria, Austria, Hungary, Belgium, Slovenia, the Netherlands, etc.), where the variance between schools is at least ‘one-and-a-half times that of the OECD’ – which is bad. For Slovenia, we demonstrate a significantly different share of variance explained by between-school variance of the same or similar cohort if we move the point of research by only nine months. Our main argument is that such a difference in the share of variance is not the result of a substantial change in the results (grooving difference in the performance) but the result of formal organisational change – the transition of the cohort to upper secondary education. The difference in calculations also radically changes policy implications, which is crucial. While PISA data suggest the necessity of policy measures to reduce between-school variance, our calculations suggest the need for reconsideration of policy measures aimed at the reduction of within-school variance, gender differences, and differences in performance between native and immigrant students.