Effects of marking method and rater experience on ESL essay scores and rater performance

Abstract

This study examined the effects of marking method and rater experience on ESL (English as a Second Language) essay test scores and rater performance. Each of 31 novice and 29 experienced raters rated a sample of ESL essays both holistically and analytically. Essay scores were analysed using a multi-faceted Rasch model to compare test-takers’ writing ability estimates, inter-rater agreement, and rater severity and self-consistency across marking methods (holistic vs. analytic) and rater groups (novice vs. experienced). Raters tended to be less severe with analytic marking. Holistic marking led to a higher level of inter-rater agreement, while analytic marking led to higher self-consistency. The two rater groups differed in terms of severity, inter-rater agreement and self-consistency. Furthermore, the two marking methods performed differently across the two rater groups. Overall, the findings suggest that the two marking methods might be useful for different assessment purposes, contexts and raters. Implications for ESL writing assessment practice and research are discussed.