Universal screening with automated essay scoring: Evaluating classification accuracy in grades 3 and 4

Abstract

The adoption of the Common Core State Standards and its associated assessments has placed increased focus on writing performance. Consequently, weak writers may be at risk of failing Common Core English language arts (ELA) assessments. Thus, the current study sampled a diverse group of third and fourth grade students (n=100 and 130, respectively) who were administered Fall and Spring writing screeners using the procedures of a Direct Assessment of Writing (DAW). Results were used to predict whether students did or did not attain grade-level standards as evaluated by the summative Smarter Balanced ELA assessment. Writing screeners were scored using the Project Essay Grade (PEG) automated essay scoring system. ROC curve analysis and logistic regression were used to evaluate screening models. Area under the ROC curve (AUC) values for grade 3 were in the acceptable range (Fall=0.74, Spring=0.75). AUCs approached or fell within the excellent range for grade 4 (Fall=0.79, Spring=0.83). Sensitivity-based and d-based cutpoints were selected and measures of diagnostic accuracy, including sensitivity and specificity, are reported. Results indicate that automatically-scored DAW has promise for universal screening for writing risk.