Using multiple texts in an integrated writing assessment: Source text use as a predictor of score

Abstract

Interest in integrated tasks is increasing in second language writing, accompanied by a concern for appropriate interpretation of performances and scores from these tasks. Integrated writing adds an element not found in traditional independent writing: the use of source text material. This study investigates how source text use appears in performances on an integrated writing task, and how it differs across score levels and task topics. Educational Testing Service (ETS) supplied 480 performances on the writing section of the Internet-based Test of English as a Foreign Language (TOEFL iBT) to explore these questions. The integrated TOEFL task involves a comparative summary of listening and reading texts that present differing views on a topic. In this study, multiple regression analysis was used to consider three areas of source text use: (1) the importance of source text ideas that writers included in their summary, (2) the use of ideas from a reading source text and from a listening text, and (3) the borrowing of exact wording from the source texts (verbatim source use). These three areas were analyzed across nine score levels and indicated that score and source use are related. Overall, these features of source text use explained over 50% of the variance in scores on the reading–listening–writing task. The use of the listening text and the inclusion of important ideas from source texts explained the most variance, while use of the reading text and verbatim source use were less predictive. The latter two held a negative correlation with score, indicating that the lower scoring essays had more of these features. These findings support the claim that integrated writing assessment elicits academic writing processes, which is reflected by score. High-scoring writers selected important ideas from the source texts and used the listening text as the task prompt instructed. Low scoring writers depended heavily on the reading texts for content and direct copying of words and phrases. These findings support the validity of interpreting integrated task scores as a measure of academic writing but provide a nuanced look at the contribution of certain source use features.