Abstract
The current study aims to explore the cognitive validity of the iBT TOEFL reading test by investigating test takers’ eye movements on individual items. It is assumed that successful test takers would adopt the intended reading processes, the same types and levels of cognitive processes that they would use for real-world reading tasks. Forty-seven Chinese ESL students participated in the study, in which they took the TOEFL reading practice testlet on a computer, completed comprehension subskill tasks, and had stimulated recall interviews. Results showed that test takers tend to rely heavily on careful reading skills, while expeditious reading skills were rarely activated. The scope of reading was often restricted to the local level; learners hardly read more than a paragraph to answer questions. In some factual question items, successful readers were more efficient in reading and locating key information, whereas such group differences were not found in other items. Lastly, the gaze plots suggest that learners’ eye movements manifest various interactions between comprehension subskills, primarily subject to bottom-up linguistic processing. The limitations and implications of learners’ eye-tracking data for test validation will be further discussed.
Highlights
IntroductionThe concept of validity has been expanded and diversified for recent decades, resulting in variations (e.g., content validity, criterion validity, construct validity, cognitive validity, consequential validity); of fundamental interest to language testers would be whether a test measures what it is supposed to measure
The concept of validity has been expanded and diversified for recent decades, resulting in variations; of fundamental interest to language testers would be whether a test measures what it is supposed to measure
The results of the Kruskal-Wallis test showed that both high and low scoring groups spread out their attention over the six paragraphs (χ2 = .728, p = .981 for the high scoring group and χ2 = 4.461, p = .485 for the low scoring group); the first or the last paragraph did not necessarily receive more attention from the readers and the reading patterns did not differ across groups
Summary
The concept of validity has been expanded and diversified for recent decades, resulting in variations (e.g., content validity, criterion validity, construct validity, cognitive validity, consequential validity); of fundamental interest to language testers would be whether a test measures what it is supposed to measure. The target construct, or the psychological reality defined by theoretical models, is often the object of discussion; a valid test should be able to tap the construct to be measured and test scores adequately reflect on the construct that test takers possess, or language ability. To evaluate the construct validity of a language test, test validation has relied heavily on correlation-based statistical analysis. Experimental evidence was examined through differential-groups and intervention studies (Brown, 2005). Such statistical approaches based on test outcomes have gained much criticism because numbers do not convey conceptual information and test scores do not explain how test takers derived their answers (Weir, 2005). Without knowing test takers’ thinking processes, any inferences and actions derived from test scores cannot be justified
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
