Effects of raters, scoring rubric, and tasks on English writing assessment for Korean high school students

Abstract

The current research was designed to provide implications for future development of English writing assessment by investigating characteristics of factors such as raters, scoring methods, and tasks affecting variability in the resulting scores of writing items of the National English Ability Test (NEAT). In addition, the interplay between raters and other factors was investigated to pinpoint potential problems confronting the NEAT. Korean high school students made use of a concordancer in English writing activities for about one month. Pre- and post-tests were administered to the students, and their performance was analyzed by means of the multi-faceted Rasch measurement. The results showed that two Korean raters and one native English speaker rater performed with fair consistency, and yet the native rater without rater training showed significant biases unlike the Korean raters who received intensive rater training and was certified as raters. As for the scoring method, the students had the most difficulty with language use, and task completion and content seemed to have overlapping characteristics in the construct definition. In addition, the writing task should be reconsidered for sound validity in terms of type.