Abstract
Abstract Rater variability has long been a concern in performance assessment. Among the various factors leading to variability, raters’ thinking processes may play an important role in shaping the rating outcomes. However, few studies have attempted to link raters’ cognition with rating accuracy. Thus, this study aims to explore how raters’ use of cognitive and meta-cognitive strategies might influence rating accuracy in the context of College English Test Band 4 essay scoring in China. Thirteen CET4 raters were classified into ACCURATE and LESS ACCURATE groups based on their rating accuracy, which was operationalised as the match between the ratings obtained from the target rater and those obtained from an expert panel measured by the Many-Facet Rasch Model analysis. Concurrent think-aloud was utilised to elicit the cognitive and meta-cognitive strategies the raters employed during the rating process. The thinking processes of the two rater groups were compared both quantitatively and qualitatively. The results suggest that more accurate raters are better at integrating information from target essays and are more self-conscious about their rating accuracy. Findings are expected to explain to some extent why some raters are effective in decision-making and generate more accurate ratings, which will have important implications on rater training.
Published Version
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call