Using Inter-Rater Discourse to Trace the Origins of Disagreement: Towards Collective Reflective Practice in L2 Assessment

Abstract

This article explores how the analysis of inter-rater discourse can be used to support collective reflective practice in second language (L2) assessment. To demonstrate, a focused case of the discourse between two experienced language teachers as they negotiate assessment decisions on L2 written texts is presented. Of particular interest was the discourse surrounding the raters’ most divergent assessment decisions, which in this case were those relating to Task Achievement. Thematic analysis indicated that rater discourse predominantly focused on explicit objective factors, primarily the L2 texts and the rating scale; however, rater discourse also focused on more subjective, rater-centred factors. The discourse surrounding these rater-centred factors was often central to the identification and resolution of rating disagreements. The paper argues that the subjective dimension of language assessment needs to be more directly and systematically reflected upon in language teaching contexts and that analysis of rater discourse, especially discourse focused on points of disagreement between raters, provides a valuable mechanism to facilitate this.