‘If you have a question that doesn’t work, then it’s clearly going to upset candidates’: what gives rise to errors in examination papers?

Abstract

ABSTRACT This study investigated an under-researched source of measurement error in high-stakes examinations, namely mistakes in examination papers (e.g. multiple-choice questions containing more than one correct answer). Such mistakes (henceforth errors) are important, as they can undermine students’ performance in the examination, while also eroding public confidence in the examination system. In light of this, this study sought to understand what gives rise to such errors. To this end, it drew upon in-depth interviews with 36 assessment professionals with knowledge and experience of the examination paper construction process. Through illuminating the demands of the paper construction process and the characteristics of the people who are involved in it, this investigation identified a number of ‘active’ and ‘latent’ human failures that can lead to errors in examination papers. The article argues that, to minimise the risk of error in examination papers, efforts to ‘adapt’ people to the demands of the paper construction process should be carefully combined with efforts to adapt the paper construction process to people requirements. Apart from contributing new knowledge to the field, the study’s findings have also practical implications: examination providers can use them to optimise their paper construction processes to enhance the quality of their assessment instruments.