Did they take the same test? Examinee language proficiency and the structure of language tests

Abstract

This study investigated the relationship between examinee proficiency and the structure of the Test of English as a Foreign Language (TOEFL) and the Speaking proficiency in English Assessment Kit (SPEAK). Specifically, using multi-group structural equation modeling, this study tested two competing hypotheses about the relationship: whether or not the dimensions of language ability become more or less differentiated as a function of increasing examinee proficiency. Among four alternative models (a second-order model, a correlated specific factor model, a single general factor model, and a totally divisible model), a second-order factor model was established as a baseline model for three ability groups and then estimated simultaneously with cross-group equality imposed. The results showed that the tests were partially equivalent, thereby supporting neither of the hypotheses. Finally, the multiple indicators and multiple causes modeling analysis suggested that the language backgrounds of test-takers were related to measurement non-invariance.