Combination, Aggregation and Reconciliation: evidential and consequential bases

Abstract

abstract In educational assessment, it is widely assumed that (possibly weighted) addition of scores achieved on individual items is the most appropriate method of deriving a single domain score. However, the inferences that can be drawn from such combined scores are often quite limited, because strength in one area can compensate for weakness in others. For this reason, during the development of assessments for the National Curriculum of England and Wales, a variety of innovative methods of combination have been proposed. A classification of combination schemes is presented, which analyses combination schemes in terms of both the inferences that are possible from the combined scores, and the possible consequences of the use of such schemes. In particular, it is shown that some combination schemes (termed manipulable) create incentives for teachers to treat different students in different ways (although whether teachers then do so is a much more complex issue). An index of manipulability is proposed and estimates of the index are presented for eight combination schemes.