Accuracy in the scoring of writing: Studies of reliability and validity using a New Zealand writing assessment system

Abstract

Accuracy in the scoring of writing is critical if standardized tasks are to be used in a national assessment scheme. Three approaches to establishing accuracy (i.e., consensus, consistency, and measurement) exist and commonly large-scale assessment programs of primary school writing demonstrate adjacent agreement consensus rates of between 80% and 100%, and consistency and measurement coefficients ranging between .70 and .80, .60 and .80, respectively. A New Zealand educational assessment project has developed a set of writing assessment rubrics that contain curriculum based rating scales for six purposes of writing each with its own bank of writing prompts for use by classroom teachers. Standardization of the rating scales and prompts was conducted with representative samples of students, aged 10–13. This article describes two studies that established the validity of the scoring system for use in New Zealand classrooms. Adjacent agreement consensus fell between 70% and 90%, while consistency and measurement correlations fell in the range .70–.80 in both studies. This consistency with international standards was sufficiently robust to provide confidence in the underlying norms provided by the asTTle assessment tool. Relatively low levels of training were required by teachers to reach this degree of accuracy. The accuracy of scoring gives government and teachers confidence in the validity of the project's rating scales and suggests that classroom teachers will be able to generate accurate scores upon which instructional decisions can be based.