On reducing errors in assessment instruments

Abstract

ABSTRACT Errors that occasionally manifest in examination papers and other educational assessment instruments can threaten reliability and validity. For example, a multiple choice question could have two correct response options, or a geography question containing an inaccurate map could be unanswerable. In this paper we explore this oft-neglected element of fairness. We adopt the theoretical position that assessment instrument errors stem from different types of human failure, which in turn stem from system-level failure. We then explore how the term ‘error’ is used in conceptually different ways, potentially engendering confusion in academic discourse. We analysed error incident records from an international assessment organisation to develop a taxonomy of assessment instrument error types. Our taxonomy covers errors that manifest in assessments in all school subjects. We used it to develop an approach to process analysis for any system of checks applied during the later stages of instrument construction. This facilitates subsequent evaluation of the psychological causes of failure. The taxonomy and process analysis approach have applications in analysing types and rates of errors in different educational assessment instruments, in systemically comparing the demands and efficacies of alternative construction processes, and as structural and communication tools in qualitative research on error culture.