Investigating Examiners' Thinking

Abstract

Since examination results affect many people’s life chances, it is essential that examination marking is accurate. Accuracy can be maximised by reducing the demands of the marking process and by increasing examiners’ personal expertise. The aim of our study was to deepen understanding of how examiners’ thinking is influenced by marking task demands. To do this, we identified question features that trigger or demand the use of cognitive marking strategies entailing reflective judgements. Kelly’s Repertory Grid (KRG) technique was used with highly experienced examiners to identify the most influential question features in a past international biology examination paper for 16year-olds. The examination questions were of known marking accuracy and marking strategy complexity. KRG technique requires participants to make comparisons among triads of elements, which may be people, objects, or concepts. Constructs are thereby elicited, which are arguably the highly individualistic dimensions in which the participant perceives and interprets the world, and indicators of the terms he or she uses to make judgements. In this study, the examiners generated grids of question features (constructs) on which the biology questions differed from one another. Subsequently, they rated each examination question for each feature using self-generated 5-point scales. Semi-structured interviews were also conducted. Analysis of the examiners’ question feature ratings and interview data revealed five features that are highly likely to trigger or demand the use of cognitive marking strategies entailing reflective thought processes. For example, questions requiring examinees to use their own words or formulae require complex cognitive marking strategies. A further five features may contribute less directly to marking strategy usage.