Implications For Language Testing Research Articles

Who succeeds and who fails? Exploring the role of background variables in explaining the outcomes of L2 language tests

This study explores whether and to what extent the background information supplied by 10,155 immigrants who took an official language test in Norwegian affected their chances of passing one, two, or all three parts of the test. The background information included in the analysis was prior education, region (location of their home country), language (first language [L1] background, knowledge of English), second language (hours of second language [L2] instruction, L2 use), L1 community (years of residence, contact with L1 speakers), age, and gender. An ordered logistic regression analysis revealed that eight of the hypothesised explanatory variables significantly impacted the dependent variable (test result). Several of the significant variables relate to pre-immigration conditions, such as educational opportunities earlier in life. The findings have implications for language testing and also, to some extent, for the understanding of variation in learning outcomes.

Social attitudes, intelligibility and comprehensibility: The role of the listener in the perception of non-native speech

This study investigates the role of the listener in the perception of non-native speakers and their speech. Our goal is to examine the impact of listener characteristics on their attitudes towards non-native speakers and their speech. In addition, we aim to explore the relationship between listeners’ attitudes and the intelligibility, comprehensibility and perceived foreign accentedness of non-native speech. 126 participants living in Flanders, Belgium, completed a questionnaire probing their attitudes towards non-native speakers of Dutch. An analysis of background variables of these listeners revealed that their age, educational level, extent of contact with non-native speakers and especially political preference could predict their responses to questionnaire items. In a subsequent session, participants performed (a) a speaker/speech evaluation task, (b) a transcription task measuring intelligibility, and (c) a comprehensibility and accentedness rating task. This latter session required participants to transcribe and evaluate speech samples of Dutch produced by speakers of Mandarin Chinese. We found a significant correlation between comprehensibility and accentedness and a number of attitudinal dimensions, such as the perceived status of, and solidarity with, the speaker. The study has implications for language testing, as it demonstrates the impact of listeners’ social attitudes on the assessment of non-native speech.

Interpreting the Validity of a High-Stakes Test in Light of the Argument-Based Framework: Implications for Test Improvement

The validity of large-scale assessments may be compromised, partly due to their content inappropriateness or construct underrepresentation. Few validity studies have focused on such assessments within an argument-based framework. This study analyzed the domain description and evaluation inference of the Ph.D. Entrance Exam of ELT (PEEE) sat by Ph.D. examinees (n = 999) in 2014 in Iran. To track evidence for domain definition, the test content was scrutinized by applied linguistics experts (n = 12). As for evaluation inference, the reliability and differential item functioning (DIF) of the test were examined. Results indicated that the test is biased because (1) the test tasks are not fully represented in the Ph.D. course objectives, (2) the test is best reliable for high-ability test-takers (IRT analysis), and (3) 4 items are flagged for nonnegligible DIF (logistic regression [LR] analysis). Implications for language testing and assessment are discussed and some possible suggestions are offered.

What Counts in Second Language Oral Communication Ability? The Perspective of Linguistic Laypersons

AbstractApplied linguists have developed complex theories of the ability to communicate in a second language (L2). However, the perspectives on L2 communication ability of speakers who are not trained language professionals have been incorporated neither into theories of communication ability nor in the criteria for assessing performance on general-purpose oral proficiency tests. This potentially weakens the validity of such tests because the ultimate arbiters of L2 speakers’ oral performance are not trained language professionals. This study investigates the perspectives of these linguistic laypersons on L2 communication ability. Twenty-three native and non-native English-speaking linguistic laypersons judged L2 speakers’ oral performances and verbalized the reasons for their judgments. The results showed that the participants focus not only on the linguistic aspects of the speaker’s output but also on features that applied linguists have less paid attention to. Even where speaker’s linguistic errors were acknowledged, message conveyance and comprehensibility of the message contributed to their judgment. The study has implications for language testing and the development of tests reflecting the construct of English as a lingua franca.