Why are some GCSE examination questions harder to mark accurately than others? Using Kelly's Repertory Grid technique to identify relevant question features

Abstract

It has long been established that marking accuracy in public examinations varies considerably among subjects and markers. This is unsurprising, given the diverse cognitive strategies that the marking process can entail, but what makes some questions harder to mark accurately than others? Are there distinct but subtle features of questions and their mark schemes that can affect accuracy? Such features could potentially contribute to a broad rationale for designating questions to markers according to personal expertise. The aim of this study was to identify question features that can distinguish those questions that are marked highly accurately from those that are marked less accurately. The study comprised an exploration of maths and physics questions from past GCSE examinations, which were marked in an experimental setting by groups of markers and yielded differing marking accuracies. The questions also varied in their difficulty for GCSE candidates, and in the cognitive strategies needed to mark them. Kelly's Repertory Grid technique and semi‐structured interview schedules were used in meetings with highly experienced principal examiners, who had led the experimental marking of the questions. The data generated comprised ratings for each question on a number of question features (constructs). The ratings were analysed together with the marking accuracy data, enabling an investigation of possible relationships between each question feature and (i) marking accuracy, (ii) question difficulty for the candidate, and (iii) apparent cognitive marking strategy usage. For both subjects, marking accuracy was found to be related to various subject‐specific question features, some of which were also related to question difficulty (for the candidate) and/or apparent marking strategy complexity. For both maths and physics, several other subject‐specific question features were found to be unrelated to accuracy. Overall, the findings have potential implications for the management of markers and for question design.