Which Schools and Pupils Respond to Educational Achievement Surveys?: A Focus on the English Programme for International Student Assessment Sample

Gabriele B Durrant,Sylke V Schnepf

doi:10.1111/rssa.12337

Abstract

SummaryNon-response is a major problem facing research in the social sciences including in education surveys. Hence, research is needed to understand non-response patterns better as well as non-response as a social phenomenon. Findings may contribute to improvements in the future designs of such surveys. Using logistic and multilevel logistic modelling we examine correlates of non-response at the school and pupil level in the important educational achievement survey the ‘Programme for International Student Assessment’ (PISA) for England. The analysis exploits unusually rich auxiliary information on all schools and pupils sampled for PISA, whether responding or not, from two large-scale administrative sources on pupils’ socio-economic background and results in national public examinations. This information correlates highly with the PISA target variable. Findings show that characteristics that are associated with non-response differ between the school and pupil levels. Our results also indicate that schools matter in explaining pupil level response, which is often ignored in non-response analysis. Our findings have important implications for future education surveys. For example, if replacement schools are used to improve response, our results suggest that it may be more important to match initial and replacement schools on the socio-economic composition of their pupils than on any of the factors that are currently used.

Highlights

Results of international educational achievement surveys make it possible to compare countries’ success in educational outcomes and have impacted widely on education policy debates in most of the countries covered by these surveys
Modelling school response Given that the Programme for International Student Assessment (PISA) target variable is achievement, we first investigate if school response is associated with pupil ability within schools
After an initial slight decrease of the response probability for schools with on average low achieving pupils, it increases to its maximum around the median Key Stage 4’ (KS4) point score of 36 and decreases thereafter

Summary

Introduction

Results of international educational achievement surveys make it possible to compare countries’ success in educational outcomes and have impacted widely on education policy debates in most of the countries covered by these surveys. Attention to the survey results has not been matched by thorough reviews of the data quality of such surveys (Araujo et al, 2017), in particular with respect to non-response. It is of interest to better understand non-response, both nonresponse patterns and correlates of non-response. A better understanding will benefit future survey designs, in particular when variables about the survey data collection process (so-called paradata, Couper, 1998) are available. The sparse analysis of non-response in education surveys stands in contrast to household surveys, for which considerable research on the nature and correlates of non-response, including paradata, is available (for a summary see Groves, 2006; and Durrant and Steele, 2009; Kreuter, 2013; Durrant et al, 2015).

Objectives

Results

Conclusion