Individualized feedback to raters in language assessment: Impacts on rater effects

Abstract

This study examined the impacts of individualized feedback on rater effects in performance-based language assessment. The raters were 93 native Chinese speakers without previous rating experience, and they were randomly assigned to one of the three treatment groups. The three groups of raters differed in the way that they received individualized feedback in a given period: (a) a control group receiving no feedback, (b) a single-feedback group receiving the feedback once, and (c) a double-feedback group receiving the feedback twice. The results indicated that individualized feedback significantly reduced rater severity/leniency and rater misfit. Specifically, raters from both the double-feedback and single-feedback groups exhibited lower levels of rater severity/leniency, compared with those from the control group. Raters from the double-feedback group showed lower levels of rater misfit than those from the single-feedback group. With regard to the retention of reduction in rater effects, individualized feedback was found to assist raters from the double-feedback group to maintain the reduction in rater severity/leniency. It also helped raters from the single-feedback group to maintain the reduction in rater misfit. These findings may shed light on the applications of individualized feedback in the designs of face-to-face and online rater training programs.