What is a holistic judgement, anyway?

Abstract

ABSTRACT Holistic judgement is an appealing approach for many assessment contexts due to its perceived simplicity and efficiency. However, it has been somewhat under-explored conceptually. Drawing on examples from assessment contexts including vocational performance, comparative judgement and the use of holistic judgement in teacher assessment for high stakes grading, we explicate a three-part definition of what holistic judgement constitutes. Holistic judgements result in singular outputs, derive from a comprehensive consideration of relevant constructs and acknowledge that the elements considered within them interconnect. However, holistic judgements may be made using considerably different processes by different judges without contravening this definition, and the ways in which different elements are weighted may vary. We then explore some factors, specific to assessment contexts, that might make holistic judgements more challenging, including materials being very different from one another, non-uniform candidate performance across materials and the presence of construct-irrelevant material. We set this assessment-specific discussion in the context of literatures on decision-making in psychology, medicine and other contexts. We conclude with some recommendations for when holistic judgement should and should not be used, and how holistic judgements can be made less cognitively challenging through the use of appropriate guidance and feedback to judges.