The role of automated writing evaluation holistic scores in the ESL classroom

Abstract

The advancement of automated writing evaluation (AWE) has made it possible to complement instructor input with immediate scoring and qualitative feedback to inform students' writing development. However, little is known about the usefulness of AWE holistic scores for classroom purposes. In our longitudinal mixed-methods study, we utilized Criterion®, an AWE service developed by Educational Testing Service (ETS), to investigate the use of holistic scores in three university-level English as second language (ESL) writing courses. We investigated the correlation between AWE scores and instructors' numeric grades and analytic ratings on two major course assignments and analyzed AWE score use from the perspectives of three ESL writing instructors and 67 students. Quantitative analyses yielded a series of low to moderate positive correlations between Criterion scores and both the instructors' grades and analytic ratings, which raised issues in integrating the scores for pedagogical purposes; survey and interview data showed that instructors used Criterion scores strategically to gauge students' writing needs. The students also made efforts to revise their papers in order to get a higher score. These findings added to previous work on the usefulness of AWE programs and provided justification for use or non-use of automatically generated scores for classroom-based formative assessment.