Rater variation in pragmatic assessment: The impact of the linguistic background on peer-assessment and self-assessment

Abstract

Abstract The present study investigates variability among raters from different linguistic backgrounds, who evaluated the pragmatic performance of English language learners with varying native languages (L1s) by using both self- and peer-assessments. To this end, written discourse completion task (WDCT) samples of requesting speech acts from 10 participants were collected. Thereafter, the participants were asked to assess their peers’ WDCTs before assessing their own samples using the same rating scale. The raters were further asked to provide an explanation for their rating decisions. Findings indicate that there may indeed be a link between a rater’s language background and their scoring patterns, although the results regarding peer- and self-assessment are mixed. There are both similarities and differences in the participants’ use of pragmatic norms and social rules in evaluating appropriateness.