Reliability of raters: The effects of rating errors on the speech rating process

Abstract

The purposes of the present investigation were to determine the sources and amount of error variance associated with rating speeches and other communicative performance. Using students’ everyday ratings of their classmates’ speeches, interrater reliability and interrater agreement were computed by utilizing the completed rating scale. Leniency error, halo error, and trait error scores were computed by utilizing the six trait scores comprising the rating scale. Raters were both unreliable and a source of considerable error. The leniency error, and the halo error accounted for the majority of the total error variance. The six trait errors accounted for the remaining error. Raters within class were consistent but there was no consistency among raters in different classes. Finally, untrained raters committed more of all eight types of errors and more leniency errors than trained raters.