Integrated writing scores based on holistic and multi-trait scales: A generalizability analysis

Abstract

Abstract Scoring reading-to-write (RTW) tasks is known to be more challenging than independent tasks given that raters should attend to features of source use, in addition to other writing skills. Holistic scales have been traditionally used with this task type; however, analytic scales have recently received increasing attention. While research has looked into score generalizability of RTW tasks, few studies have addressed the impact of rating scales on RTW score reliability. For this purpose, the current study compares score reliability from both holistic and multi-trait rating scales. Following a generalizability theory approach, five raters scored 60 essays written by EFL university students using both holistic and multi-trait scoring methods. The results indicated that scores obtained based on the multi-trait rubric were found to be more reliable than those obtained from a holistic rubric. The results also showed that source use was the most reliable feature among all dimensions of the multi-trait rubric. The study results provide sufficient evidence for the use of multi-trait rating scales in the context of scoring RTW tasks. This outcome could encourage more writing instructors to use these scales in their assessment practices given the detailed information they provide about writers’ performance on RTW tasks.