Using calibrated exemplars in the teacher-assessment of writing: an empirical study

Abstract

Background: Many in education argue for the importance of incorporating teacher judgements in the assessment and reporting of student performance. Advocates of such an approach are cognisant, though, that obtaining a satisfactory level of consistency in teacher judgements poses a challenge. Purpose: This study investigates the extent to which the use of a two-stage method of assessment involving calibrated exemplars provides judgements from teachers that are consistent. Teachers were not given extensive training and moderation. We chose the assessment of early writing as a context to investigate the method as it is fundamental to students’ progress in schooling. Sample: Stage 1: Eleven teachers of four- to seven-year olds (kindergarten to year 2) were invited to collect their students’ performances. Sixty performances that represented the range of ability were selected from approximately 300 performances. Fifteen teachers from 12 schools made pairwise comparisons of performances. Stage 2: Fourteen teachers representing six schools plus the co-ordinator of the study participated in this stage of the exercise. Convenience sampling of teachers was employed. Design and method: Stage 1: The method of pairwise comparison was used to calibrate the performances of students by developing a performance scale. These performances were then used as exemplars, which are referred to here as calibrated exemplars. Stage 2: Teachers assessed student performances simply by judging which calibrated exemplar a performance was most alike. In a separate exercise, two experienced markers assessed another set of 118 writing performances using both (1) a criterion-based rubric and (2) the calibrated exemplars. Results: The two-staged process showed a level of consistency in teacher judgement-making. In addition, judgements made by experienced markers with the calibrated exemplars correlated well with judgements made using the criterion-based rubric. Conclusions: The findings suggest that using calibrated exemplars has potential as a method of teacher assessment in contexts where extensive training and moderation is not possible or desirable. Further research is needed to establish whether the findings generalise to the classroom context and whether consistency could be demonstrated on a large scale in this and other curriculum areas. Research is also needed to investigate whether the calibrated exemplars can be supported with qualitative information for use in formative assessment.