Investigating test-takers' cognitive processes while completing an integrated reading-to-write task:evidence from eye-tracking, stimulated recall and questionnaire

Abstract

The TBEM-8 (Test for Business English Majors, band 8) is a newly-developed, nationwide test of business English proficiency administered to business English undergraduates in China at the end of their final year. One notable feature of the test is that it includes a reading-to-write task in which test-takers read texts in English and Chinese and then use this information to write an essay on a business-related topic. Although the test has been operational for several years, there is currently little validity evidence to support claims about the cognitive processing which takes place during this reading-to-write task. This presents a threat to the quality of inferences drawn from test scores. The present research examined test-takers’ cognitive processes while completing the TBEM-8 reading-to-write task, aiming to gain further insights into cognitive processing on this integrated task type. Two separate studies were conducted. In Study I, 16 participants completed this task while their eye movements were tracked by a Tobii TX300 eye-tracker. These eye traces then formed the stimuli for a stimulated recall session to elicit cognitive processes; in Study II, another 172 participants responded to a reading-to-write process questionnaire after completing the task. This questionnaire was developed by Chan (2013) and adapted for the TBEM-8 reading-to-write task. A pilot study was also conducted to finalise the main study questionnaire, in which 40 items were grouped to reflect the cognitive processes that writers are hypothesised to undergo. The results showed that test-takers engaged in a wide range of cognitive processes specified in Shaw and Weir’s (2007) model of writing and Spivey’s (1990, 1997, 2001) discourse synthesis model during task completion, thus justifying the current use of it in the TBEM-8 test. Text interpretation and selecting were the two most frequently reported processes according to participants’ stimulated recalls, and macro-planning and translating were the two least reported processes. A high level of agreement was found in participants’ responses to the reading-to-write process questionnaire, with more than 70 percent of participants choosing either “agree” or “strongly agree” in 28 items, and only four items achieving an agreement rate below 60 percent. The correlation analysis between the use of cognitive processes/eye-tracking measures and test-takers’ performance on the TBEM-8 reading-to-write task yielded no statistically significant results(at the 0.05 or 0.01 levels), except for a moderate positive correlation (ρ=.499, p=.049) between the participants’ max visit duration on Source 5 (key concepts and expressions) and their reading-to-write performance, and one (ρ=.432, p=.098) between the counts of text interpretation-2 process (reading source materials) and the task performance if the p-value was set to 0.1. This study demonstrated the usefulness of combining eye-tracking, stimulated recall and questionnaire methods for generating insights into the complexity of cognitive processing on an integrated reading-to-write task. Findings from the analysis of all sources of data were triangulated and discussed, providing a solid basis for the conclusions drawn about test-takers’ cognitive processing during task completion. Also, a model of reading-to-write process was proposed to illustrate how different categories of cognitive processes examined in this study interact with each other for successful task completion.