Cognitive Evidence for Construct Validity of the IELTS Reading Comprehension Module: Content Analysis, Test Taking Processes, and Experts’ Accounts

Abstract

Despite the growing demand for the International English Language Test System (IELTS) Academic as a high-stakes language proficiency test, and its significant impact on the lives of test takers, only a limited amount of validity research has been published. Among the four modules of the test, the Reading Comprehension Module (RCM) has been the least researched, with few studies examining specific dimensions of the reading construct operationalized by the test (e.g., Bax, 2013; Weir, Hawkey, Green, & Devi, 2009). Following Messick's (1989) advice that construct is central in considerations of validity and informed by Khalifa and Weir's (2009) Multi-Componential Reading Comprehension Model, this qualitative case study investigated the IELTS RCM construct. It drew evidence from three phases: 1) content analysis of an RCM sample test; 2) verbal reports of the processes used during RCM performance, elicited from three groups of test takers (N= 21) with different language backgrounds (i.e., first/L1 English; second/L2, English as a Foreign Language (EFL) learners in Iran; across levels of language proficiency); and, 3) verbal reports and interviews with (N= 10) testing experts, who judged the skills, knowledge sources, processes, and strategies (SKPSs) (Gorin, 2006), that were tapped by each RCM task. Results of the content analysis showed gaps in representing features and practices of academic reading at both levels of texts and tasks. Coding (Saldana, 2009) of the test takers' verbal accounts indicated the test construct differed across language backgrounds and levels of language proficiency. For L2 test takers, test performance was basically task-based, consisting of careful reading at sentence and inter-sentential levels rather than meaningful comprehension, and raising questions about the test as a measure of the academic reading construct. The experts' accounts reinforced findings from the test takers' reports and suggest that the RCM tasks tap into micro, lexico-grammatical features rather than macro, textual comprehension. Higher order inferential global comprehension reading skills were disproportionately underrepresented. Implications for different stakeholders are discussed. The study concludes that more research is essential in order to justify the use of IELTS RCM as an English language proficiency measure.