Examining the Writing Processes in Scenario‐Based Assessment Using Regression Trees

Abstract

Scenario‐based writing assessment has two salient characteristics by design: a lead‐in/essay scaffolding structure and a unified scenario/topic throughout. In this study, we examine whether the scenario‐based assessment design would impact students' essay scores compared to its alternative conditions, which intentionally broke the scaffolding effect and/or topic effect. Furthermore, with the availability of rich keystroke log data, we used tree‐based methods to investigate which writing process features had greater influence in predicting students' essay scores under different assessment conditions. The results revealed significantly lower essay scores when the unified topic effect was removed and no significant essay score change when neither scaffolding nor topic effects existed compared to the original scenario‐based assessment condition. The findings also suggested that different assessment conditions call on different configurations of writing process features. The rate of typo correction in the writing process was consistently among the highest ranking features that predicted the essay scores well, regardless of the presence of scaffolding and/or topic effects. The broken topic effect might affect the extent of word finding and retrieval as well as editing across multiple words by the students during writing. Topics for future study and limitations are also discussed.