Predicting EFL writing ability from levels of mental representation measured by Coh-Metrix: A structural equation modeling study

Abstract

• Coh-Metrix indices were mapped onto the construction–integration (CI) model of comprehension. • Structural equation modeling (SEM) was used to measure the levels of mental representation in CI. • Many-facet Rasch measurement was employed to measure EFL essay quality. • SEM was used to predict essay quality using the CI's levels of mental representation. • Interdependence of the levels of mental representation and essay quality was confirmed. This study aims to invoke a theoretical model to link the linguistic features of text complexity, as measured by Coh-Metrix, and text quality, as measured by human raters. One hundred and sixty three Chinese EFL learners wrote sample expository and persuasive essays that were marked by four trained raters using a writing scale comprising Word Choice, Ideas, Organization, Voice, Conventions, and Sentence Fluency traits. The psychometric reliability of the writing scores was investigated using many-facet Rasch measurement. Based on the construction–integration (CI) model of comprehension, three levels of mental representation were delineated for the essays: the surface level (lexicon and syntax), the textbase, and the situation model. Multiple proxies for each level were created using Coh-Metrix, a computational tool measuring various textual features. Using structural equation modeling (SEM), the interactions between the three levels of representation, text quality, and tasks were investigated. The SEM with the optimal fit comprised 23 observed Coh-Metrix variables measuring various latent variables. The results show that tasks affected the situation model and several surface level latent variables. Multiple interactions were identified between writing quality and levels of representation, such as the Syntactic Complexity latent variable predicting the situation model and the situation model latent variable predicting Conventions and Organization. Implications for writing assessment research are discussed.