Investigating a judgemental rank‐ordering method for maintaining standards in UK examinations

Abstract

A new judgemental method of equating raw scores on two tests, based on rank‐ordering scripts from both tests, has been developed by Bramley. The rank‐ordering method has potential application as a judgemental standard‐maintaining mechanism, because given a mark on one test (e.g. the A grade boundary mark), the equivalent mark (i.e. at the same judgemental standard) on the other test can be determined. If the two tests come from different years then the standard from the earlier year can be applied to the later year. The current standard maintaining method used by Awarding Bodies in England, Wales and Northern Ireland, is the ‘awarding meeting’. Here expert judgement takes place within the context of a variety of statistical information, including score distributions and hence likely pass rates. The rank‐ordering method, in contrast, involves harnessing expert judgement independently of any statistical information. The aim of this study was to investigate the extent to which the outcome of an awarding meeting could be cross‐validated using a rank‐ordering exercise. Furthermore, the study aimed to discover whether rank‐ordering produces similar results when the activity is conducted by post compared with a face‐to‐face meeting. The results showed that the outcomes of the postal exercise were closely replicated by the meeting exercise, indicating that the method is a reliable technique for capturing expert judgement. In terms of cross‐validation of the rank‐order outcomes with awarding data, there was some concurrence and some disparity at key grade boundaries. However, because the awarding meeting uses more information and different procedures from a rank‐ordering exercise, the outcomes should not be expected to be the same. The potential advantages and disadvantages of replacing the ‘top‐down, bottom‐up’ judgemental part of an awarding meeting with a rank‐ordering exercise are discussed.