Language Testing Research Articles

A scoping review of research on second language test preparation

Test preparation has garnered considerable attention in second language (L2) education due to the significant implications that successful performance on a language test may have for academic advancement, future career opportunities, and immigration prospects. Meanwhile, an overemphasis on test preparation has been criticized for encouraging the cultivation of construct-irrelevant test-taking strategies at the expense of developing general language proficiency. To systematically explore how test preparation has been investigated in the literature, we conducted a scoping review of 66 studies on L2 test preparation. Specifically, this study examined the key characteristics of publications on test preparation, the main themes explored, the study and participant characteristics, as well as the essential aspects of their research methodologies. The results of this review revealed various trends in the literature on L2 test preparation, such as the exclusive focus on English as the target language, the lack of diversity in stakeholders as participants, the dominance of international language tests, and the paucity of experimental studies that utilize advanced statistical techniques. In addition to interpreting the results of our analysis, we discuss the implications of this scoping review and outline several directions for future research on test preparation.

Slogans as a policy distractor: a case of ‘washback’ discourse in English language testing reforms in Japan

ABSTRACT This paper examines recent reforms in English-language testing in Japan using a policy distraction framework. We identify the term ‘washback (effect)’ and other related discourses as major distractors and investigate how ‘washback’ discourses have functioned as political slogans or catchphrases in policy deliberation processes and how they have diverted attention and resources from more essential issues. By analyzing advisory panel minutes and other policy documents, we demonstrate how policy distraction operates. Some committee members initially introduced ‘washback’ discourse in a deliberation meeting, citing studies on language testing. However, this discourse quickly became a political slogan, transforming into a dubious rationale for advocating the use of commercial four-skills English tests in university entrance exams. This ‘washback’ discourse led to policy distraction and the overlooking of more significant issues, such as class size reduction and the improvement of teachers’ working conditions. Additionally, our analysis reveals underlying factors triggering this distraction, including Japanese ideological views on English education and budgetary austerity in education. We discuss the political and pedagogical implications of these findings, particularly regarding the identification of political distractions, their potential threat to teacher agency, and strategies for addressing and correcting these distractions to facilitate social change.

The Impact of Intertextuality in Movies on Language Learning

This paper examines the effectiveness of using intertextuality in second language learning through movies. Two types of materials were prepared: first, the American animated movie <i>Shrek 2</i> (Adamson et al., 2004) was the primary material, and second, idioms from YouTube shorts were used as points of comparison for regular expressions found in <i>Shrek 2</i>. This case study involved five college students majoring in non-English subjects. Among them, one was at an intermediate level in English and four were at a low-intermediate level. After three months of participation, students underwent five tests: three language tests and two open-ended-question tests. The first three assessed recognition, salience, and memory. The results of the recognition and salience tests indicated that participants achieved perfect scores in regular expressions, thanks to intertextuality. The results of the memory test showed that idioms were remembered significantly better than regular expressions. However, this test suggested that intertextuality played a larger role in semantic contexts than in syntactic ones. In the first open-ended-question test, the participants successfully described both an original character and a new character in <i>Shrek 2</i>. The final question revealed that most participants appreciated two aspects of intertextuality: creative work and long-term memory gains.

Review of selected research in applied linguistics published in Australia (2015–2022)

Abstract This article provides a review of research in applied linguistics published in Australia in the period 2015–2022. Primarily, it is based on articles from Australian publications as material from other sources is more widely available to an international audience. The research has been published in such journals as the Australian Journal of Language and Literacy, Australian Review of Applied Linguistics (ARAL), BABEL, English in Australia, Papers in Language Testing and Assessment and TESOL in Context. Five key areas of research are discussed: First Nations peoples and their multilingualism, language and migration, language testing and assessment, language curricula and pedagogy, and teacher development, and their identity and pedagogical beliefs.

Innovation Discrete Approach Based on Arabic Language Test Assessment

Innovation Discrete Approach Based on Arabic Language Test Assessment

Twenty Years of Language Assessment Quarterly: An Interview with LAQ Founder Antony Kunnan

ABSTRACT Antony Kunnan founded the journal Language Assessment Quarterly in 2004. In this interview, Antony explains his rationale for starting the journal, what the journal has accomplished, how it differs from other language testing journals, and reflects on the evolution and future of the field of language assessment.

P.135 Automated awake brain mapping with eloquentaid: a novel tool for low-resource settings

Background: Intraoperative testing for awake craniotomies requires a multidisciplinary team which may not be available in low-resource settings. We explored the creation of an AI tool for automated testing. Methods: We developed a NodeJS application, EloquentAid (https://www.eloquentaid.com/), for language testing automation. The workflow was as follows: users select an image-based naming task and verbally identify the image in English. Then, the application transcribes the response using OpenAI’s Whisper transcription service. Finally, the application evaluates response correctness. Feedback is provided through auditory and color signals. To assess its reliability, we tested the EloquentAid versus a human rater using a 57-item test based on the Boston Naming Test. Participants were neurosurgery and neurology residents from the Philippines. Qualitative surveys were obtained post-test. Results: A total of 798 observations were recorded (N=14). Human-application agreement was 60.52%. Cohen’s kappa was 0.31 (fair agreement). There were no false positive identifications by EloquentAid. Noun-type was felt to affect human error (i.e. “knocker,” “yolk,” “trellis”). Accent and pronunciation were felt to affect EloquentAid errors. Conclusions: EloquentAid is a promising tool to facilitate intraoperative testing and brain mapping using AI for speech recognition and response evaluation. Preliminary data shows fair human-app agreements. Improvements in test items and pronunciation recognition may be made.

Validation of a Cantonese Version of the Amsterdam-Nijmegen Everyday Language Test (CANELT): A functional approach.

The current study aimed to validate the Cantonese version of the Amsterdam-Nijmegen Everyday Language Test (CANELT), a functional communication assessment tool for Cantonese speakers with aphasia. A quantitative scoring method was adopted to examine the pragmatics and informativeness of the production of people with aphasia (PWA). CANELT was translated from its English version with cultural adaptations. The performance on the 20-item CANELT collected from 56 PWA and 100 neurologically healthy Cantonese-speaking controls aged 30 to 79 years was orthographically transcribed. Scoring was based on the completeness of the main concepts produced in the preamble and subsequent elaborations, defined as Opening (O) and New Information (NI). Measures examining the validity and reliability were conducted. An age effect was found in neurologically healthy controls, and therefore z scores were used for subsequent comparisons between neurologically healthy controls and PWA. The test showed strong evidence for known-group validity in both O [χ2 (2) = 95.2, p < .001] and NI [χ2 (2) = 100.4, p < .001]. A moderate to strong correlation was found between CANELT and standardized aphasia assessment tools, suggesting satisfactory concurrent validity. Reliability measures were excellent in terms of internal consistency (Cronbach's α of .95 for both 'O' and 'NI'), test-retest reliability (ICC = .96; p < .001), intra-rater reliability (ICC = 1.00; p < .001), and inter-rater reliability for O (ICC = .99; p < .001) and NI (ICC = .99; p < .001). Sensitivity and specificity for O are 97% and 76.8%, respectively, while for NI, a sensitivity of 95% and specificity of 91.1% were obtained. Measures on validity and reliability yielded promising results, suggesting CANELT as a useful and reliable functional communication assessment for PWA. Its application in managing PWA and potential areas for development are discussed.

A Rasch-based validation of the University of Tehran English Proficiency Test (UTEPT)

Recently, psychometricians and researchers have voiced their concern over the exploration of language test items in light of Messick’s validation framework. Validity has been central to test development and use; however, it has not received due attention in language tests having grave consequences for test takers. The present study sought to examine the validity of the University of Tehran English language proficiency test (UTEPT) using 6 aspects of Messick’s validation framework (i.e., content, structural, external, generalizability, substantive, and consequential). To examine each validity dimension, the Rasch model was considered. To this end, the data of 394 test takers who had taken the test in 2023 were cleaned and analyzed for the assumptions of the Rasch model and then for the 6 aforementioned validity aspects. The results indicated that the structural (i.e., unidimensionality), generalizability (i.e., measurement invariance), and substantive aspects held. However, the evidence for the content, external, and consequential aspects of validity was less definitive. The implications of the findings for the validity of the test, for the realm of language testing, and for item developers and item writers will be discussed.

Accelerating handwritten number classifier with pipelined Multiply-Accumulate units in neural networks

This research project explores the development and assessment of a Handwritten Number Classifier based on Neural Networks, employing the widely used MNIST dataset. The primary objective of this study is to enhance computational efficiency, and to achieve this, it concentrates on integrating and contrasting non-pipelined and pipelined structures within the Multiply-Accumulate (MAC) unit. The initial phase involved conducting MATLAB simulations, which yielded promising results regarding the accuracy of weight calculations. Subsequently, Hardware Description Language (HDL) testing was carried out to further validate the classifiers performance. In the HDL testing phase, the classifier incorporating the pipelined MAC unit demonstrated a substantial 42.9% enhancement in processing speed when compared to its non-pipelined counterpart. These results highlight the potential advantages of employing pipeline processing in neural network architectures, emphasizing its effectiveness in achieving faster and more efficient image classification, particularly when dealing with extensive datasets. In conclusion, this research project not only presents valuable insights into improving the efficiency of neural network-based image classifiers but also lays the groundwork for potential future endeavors. These future directions may include adapting the classifier to handle more complex datasets and addressing emerging challenges.

Decoding IELTS writing performance: Grammatical Clauses mastery by Universitas Ahmad Dahlan students

Indonesian learners of English produced ungrammaticalities in their writing for the International English Language Testing System (IELTS) and faced complicated writing score descriptors for self-evaluations and improvements. These problems were observed among students at Universitas Ahmad Dahlan during their preparatory course for IELT. To address this issue, this study focused on identifying some rules in grammar, the grammatical and ungrammatical clauses based on subject-verb (S + V) structures in their writing performances and measuring the proportions of the two clause types in accordance with the performance’s score variation. Both qualitative and quantitative methods were employed. To do so, the study conducted archival research by retrieving data from the teacher's course documents. The archived handwritten writing performances were transcribed manually into digital texts. It employed syntactic analysis to classify the clauses as either grammatical or ungrammatical. It converted the numbers of each clause type per performance into percentages for comparison, and grouped them according to the different scores as the variable. The results showed that the participants produced predominantly simple grammatical clauses to minimize failures when generating more complex structures. To achieve the minimum score of 6 in the writing section, the proportion of grammatically correct and incorrect clauses should be above 47% and below 53%, respectively. These findings offer valuable insights into predicting scores, especially concerning the dimension of 'grammatical range and accuracy in the writing assessment, which benefit both learners and educators.

Transcription Decisions of Conjoined Independent Clauses Are Equitable Across Dialects but Impact Measurement Outcomes.

Transcription of conjoined independent clauses within language samples varies across professionals. Some transcribe these clauses as two separate utterances, whereas others conjoin them within a single utterance. As an inquiry into equitable practice, we examined rates of conjoined independent clauses produced by children and the impact of separating these clauses within utterances on measures of mean length of utterance (MLU) by a child's English dialect, clinical status, and age. The data were archival and included 246 language samples from children classified by their dialect (African American English or Southern White English) and clinical status (developmental language disorder [DLD] or typically developing [TD]), with those in the TD group further classified by their age (4 years [TD4] or 6 years [TD6]). Rates of conjoined independent clauses and the impact of these clauses on MLU varied by clinical status (DLD < TD) and age (TD4 < TD6), but not by dialect. Correlations between the rate of conjoined clauses, MLU, and language test scores were also similar across the two dialects. Transcription decisions regarding conjoined independent clauses within language samples lead to equitable measurement outcomes across dialects of English. Nevertheless, transcribing conjoined independent clauses as two separate utterances reduces one's ability to detect syntactic differences between children with and without DLD and document syntactic growth as children age. https://doi.org/10.23641/asha.25822675.

Incentivising Textbooks for Self-Study: Experimental Evidence from the Democratic Republic of the Congo

Abstract We designed and randomly evaluated the impact of textbooks for a self-study scheme in eastern areas of the Democratic Republic of the Congo targeting student achievement in primary schools. Students in treatment schools were 7 percentage points more likely to pass the national exam, and those who passed obtained higher scores. We also evidence higher scores on a French language test. The effects are primarily driven by student interest in textbooks, frequency of doing homework and motivation to go to school and continue education. Student achievement can thus be improved by intensified and diversified use of existing learning materials in poor and fragile settings.

The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests

BackgroundSituational Judgment Tests (SJTs) are commonly used in medical school admissions. However, it has been consistently found that native speakers tend to score higher on SJTs than non-native speakers, which can be particularly problematic in the admission context due to the potential risk of limited fairness. Besides type of SJT, awareness of time limit may play a role in subgroup differences in the context of cognitive load theory. This study examined the influence of SJT type and awareness of time limit against the background of language proficiency in a quasi high-stakes setting.MethodsParticipants (N = 875), applicants and students in healthcare-related study programs, completed an online study that involved two SJTs: one with a text-based stimulus and response format (HAM-SJT) and another with a video-animated stimulus and media-supported response format (Social Shapes Test, SST). They were randomly assigned to a test condition in which they were either informed about a time limit or not. In a multilevel model analysis, we examined the main effects and interactions of the predictors (test type, language proficiency and awareness of time limit) on test performance (overall, response percentage).ResultsThere were significant main effects on overall test performance for language proficiency in favor of native speakers and for awareness of time limit in favor of being aware of the time limit. Furthermore, an interaction between language proficiency and test type was found, indicating that subgroup differences are smaller for the animated SJT than for the text-based SJT. No interaction effects on overall test performance were found that included awareness of time limit.ConclusionA SJT with video-animated stimuli and a media-supported response format can reduce subgroup differences in overall test performance between native and non-native speakers in a quasi high-stakes setting. Awareness of time limit is equally important for high and low performance, regardless of language proficiency or test type.

Who spoke that language? Assessing early face-language associations in monolingual and bilingual infants.

In bilingual communities, knowing the language each speaker uses may support language separation and, later, guide language use in a context-appropriate manner. Previous research has shown that infants begin to form primary associations between the face and the language used by a speaker around the age of 3 months. However, there is still a limited understanding of how robust these associations are and whether they are influenced by the linguistic background of the infant. To answer these questions, this study explores monolingual and bilingual infants' ability to form face-language associations throughout the first year of life. A group of 4-, 6-, and 10-month-old Spanish and/or Catalan monolingual and bilingual infants were tested in an eye-tracking preferential-looking paradigm (N = 156). After the infants were familiarized with videos of a Catalan and a Spanish speaker, they were tested in two types of test trials with different task demands. First, a Silent test trial assessed primary face-language associations by measuring infants' visual preference for the speakers based on the language they had previously used. Then, two Language test trials assessed more robust face-language associations by measuring infants' ability to match the face of each speaker with their corresponding language. When measuring primary face-language associations, both monolingual and bilingual infants exhibited language-based preferences according to their specific exposure to the languages. Interestingly, this preference varied with age, with a transition from an initial familiarity preference to a novelty preference in older infants. Four-month-old infants showed a preference for the speaker who used their native/dominant language, while 10-month-old infants preferred the speaker who used their non-native/non-dominant language. When measuring more robust face-language associations, infants did not demonstrate signs of consistently matching the faces of the speakers with the language they had previously used, regardless of age or linguistic background. Overall, the results indicate that while both monolingual and bilingual infants before the first year of life can form primary face-language associations, these associations remain fragile as infants seemed unable to maintain them when tested in a more demanding task.

Application of language tests in university education. Evaluation of a formative process.

La aplicación de pruebas lingüísticas en contextos reales es una competencia fundamental que los estudiantes de los grados en Logopedia y en Educación Primaria deben aprender. El objetivo de esta investigación consiste en comprender y evaluar el proyecto formativo universitario llevado a cabo para que los estudiantes aprendan a aplicar pruebas lingüísticas. El proyecto se divide en tres fases: los estudiantes graban la aplicación de las pruebas lingüísticas, los profesores enseñan el proceso de evaluación, y otros estudiantes analizan y corrigen las pruebas grabadas. Un grupo de 200 participantes completa un cuestionario que incluye preguntas sobre las dificultades y desafíos encontrados y la idoneidad de la práctica. Se valora por separado la enseñanza de pruebas estandarizadas/baterías y la enseñanza de protocolos/registros. Los resultados sugieren que las actividades propuestas son apropiadas. Aunque no se observan diferencias significativas entre los grupos en cuatro variables, surgen disparidades en la comprensión de los vídeos, la toma de notas en las hojas de registro, la elaboración de conclusiones y la redacción de informes. Lo que significa que esta investigación es útil para aquellos docentes que estén interesados en llevar a cabo proyectos formativos con aplicación práctica en el aula y poder así generalizar el proceso formativo a otras universidades.

Language Errors among Iranian EFL Students with High and Low Levels of Writing Anxiety

Writing is known as one of the major important language skills and it is the most difficult language skills for most EFL and ESL learners. In addition, anxiety is one of the factors that many language learners face with it in the writing process. This paper examines language errors among Iranian students of English with high and low levels of writing anxiety. For this purpose, 250 English language learners from universities of the south of Iran were participated in this research. This research used two tools to collect data: the Second Language Writing Anxiety Questionnaire (SLWAI), which was a pre-test containing 22 items and three subscales of physical anxiety, avoidance behavior, and cognition based on a Likert point scale. In addition, a language test in the form of essay writing was used to recognize their different categories and main errors in the writing performance. The results of this research showed that there is a negative relationship between students' anxiety and the number of errors in their writing performance. This study discusses the implications of this finding for English language learners and teachers.

Montessori primary schools’ effectiveness: a quasi-experimental study on schooling outcomes

ABSTRACT Montessori is a highly individualized pedagogy that prioritizes following each child’s individual interests over standardized test scores. This quasi-experimental study compares groups of Montessori students (treatment groups, N = 535–710) with control groups of non-Montessori students, matched through the simple matching method, which controls for the effect of background variables on schooling outcomes. The tested hypothesis is that the treatment groups achieve mean scores that are not significantly lower and have a significantly higher standard deviation on standardized tests of mathematics and language. The study encompassed students in Grade 2, 5, 8, and 10 across two separate cohorts. Results indicate that Montessori students attain either comparable or significantly higher scores than non-Montessori students, depending on grade and cohort, with greater score variability observed, particularly at Grade 8 and in mathematics. This finding implies the necessity for future investigations into the effect of Montessori teachers’ instructional practices.

Perceived need for a teacher education course on assessment literacy development: insights from EAP instructors

Classroom assessment, as a crucial element of the teaching journey, plays a significant part in enhancing student achievement, student learning, and teacher instruction. As such, teachers should possess a good grasp of assessment literacy. This study aimed to explore the EAP teachers’ perceptions regarding the demand for in-service training in various areas of language assessment, in particular, examining how a language testing and assessment (LTA) teacher education course would impact the EAP instructors’ Language Assessment Literacy (LAL). In this qualitative study, pursuing a descriptive narrative design, 60 EAP instructors were selected through purposive sampling from those who have been working in state universities in Tehran. From these participants, 30 EAP teachers were asked to write narratives regarding their needs for training on LAL, and the other 30 participants were involved in an online semi-structured interview to determine how such a course could promote or inhibit their perceptions and practices of LA and LAL. Using MAXQDA to do content and thematic analysis, the research findings revealed that EAP instructors highly perceived a need for further training especially a practice-based one. In particular, the instructors thought of teacher education courses as contributing to their theoretical and practical knowledge of LAL. Hence, teacher education courses with a specific focus on LAL are highly required, changing EAP teachers’ perspectives on LTA, from being taught solely in theory to being taught practically in real classroom environments.

Decolonizing English language testing

AbstractTesting practices and the construct of English both serve separately and interactionally to promote activities of modernity and coloniality. Tests categorize and rank learning and knowledge in discrete, static ways. The construct of the English language through standardization and other processes upholds linguistic purism ideologies. Such concerns in both areas lead to the guiding question: What and whose purposes are being served with testing in English? In engaging with this question, the author posits that decolonizing approaches to testing and the construct of English are needed. In this article, she presents assessment practices developed within Indigenous learning contexts that can inform general approaches to assessment and language with implications for English language testing. In doing so, the article offers actions toward imaginative and creative reconfigurations of English testing according to epistemologies of communities that have confronted colonialist activities.