The impact of essay organization and overall quality on the holistic scoring of EFL writing: Perspectives from classroom english teachers and national writing raters

Abstract

This three-phase mixed-methods study aimed to examine if essay organization and overall quality interferes with the holistic scoring of EFL essays in both classroom and national large-scale standardized writing assessment contexts in Chinese higher education. Phase One involved semi-structured interviews with 16 college English teachers across China in terms of their perceptions of the impact of essay organization and overall quality on their holistic scoring of EFL essays. Phase Two included 20 national writing raters’ holistic scoring of 30 EFL essays of low and high overall qualities for the quantitative investigation of score variability and reliability within the framework of generalizability theory. Phase Three involved follow-up writing rater interviews for the qualitative examination of their consideration and treatment of essay organization and overall quality in scoring EFL essays in the national large-scale writing assessment context. Phases One and Three qualitative results indicated that most classroom teachers and national writing raters considered language, content, and organization integratively while scoring high overall quality essays but mainly considered language or content while scoring low overall quality essays. Phase Two quantitative results showed that in the large-scale writing assessment the holistic score variability decreases but its reliability increases as the EFL essay overall quality increases from low to high. Important implications for EFL writing assessment in both contexts are discussed.