A validity argument for score meaning of a computer-based ESL academic collocational ability test based on a corpus-driven approach to test design

Abstract

Among the language capacities that L2 learners of English develop as they increase their proficiency is the ability to use appropriate collocations in the relevant registers of language use. Assessment of this capacity may therefore provide an efficient means of distinguishing among examinees with various levels of language proficiency within a particular register, although research has not yet attempted to operationalize such a measure. The purpose of this research is to explore the use of a measure of developmental, registerspecific collocational knowledge as a means of making relevant distinctions among examinees in a minimal amount of time. Ultimately, such a test could be used in conjunction with other information for norm-referenced decisions such as placement, with a convincing argument for such uses. The first step, however, is to develop the interpretive argument and validity argument for score meaning of such a measure. This validation study followed a mixed-methods embedded and sequential explanatory design consisting of quantitative and qualitative data collection (Creswell & Plano Clark, 2007). Quantitative data were collected from performance on the test of collocational ability, a productive vocabulary size test, and a reading test by 206 Chinesespeaking learners of English at three levels of English proficiency. Qualitative data were collected for a sample of the participants during and after the quantitative data collection and analyzed in order to identify evidence to explain initial quantitative results from the collocational ability test including test-taking processes. Qualitative data included screen capturing during test administration (n = 10), post-test interviews (n = 6) and a test reflection survey (n = 206). The test development, piloting, data collection, and analysis provide backing for the assumptions underlying inferences in the interpretive argument.