Validity of automated essay scores for elementary-age English language learners: Evidence of bias?

Abstract

Given increased prevalence of automated writing evaluation (AWE) systems in classroom settings, more research is needed to explore the potential for bias in automated scores with respect to English language learners (ELLs). Thus, this research study investigated and compared the predictive validity of automated and human scoring methods for elementary-age English ELLs on a writing test designed for ELLs and a state writing test designed for the general population. This study focused on the MI Write AWE system and sampled 2829 students comprising ELLs and non-ELLs in Grades 3–5. Results of multilevel regression analyses and simple slopes estimation indicated that, for ELLs, the automated MI Write score had similar predictive validity to the human score for both writing tests. However, automated and human scores for ELLs were less closely related to the state writing test score than scores for non-ELL students. Findings suggest that MI Write’s automated scoring was not uniquely biased relative to human scoring but does reproduce the same biases evident with human scoring. Implications and directions for future research are discussed.