An exploration of the examination script features that most influence expert judgements in three methods of evaluating script quality

Abstract

Some methods of determining grade boundaries within examinations, such as awarding, paired comparisons, and rank ordering, entail expert judgements of script quality. We aimed to identify the features of examinees’ scripts that most influence judgements in the three methods. For contrasting examinations in biology and English, a Latin square design enabled each of three matched groups of 10 experienced examiners to use each method to determine grade boundaries in an experimental setting. Additionally, every script was rated separately for nine potentially influential features by at least two examiners working independently. All three methods generated plausible grade boundary marks. The most influential features were inferred from analyses of the relationships between script feature ratings and grading judgements. Although the three methods yielded similarities and variations in the most influential features, no influential features were deemed likely to compromise the methods’ validities.