Field Of Language Testing Research Articles

Developing an MLA-test for young learners – insights from measurement theory and language testing

This article presents the process of constructing and validating a test of metalinguistic awareness (MLA) for young school children (age 8–10). The test was developed between 2021 and 2023 as part of the MetaLearn research project, financed by The Research Council of Norway. The research team defines MLA as using metalinguistic knowledge at a point in real time. To develop the test, the research team relied heavily on general measurement theory (psychometrics) and insights from the field of language testing and assessment. This aids in the process of developing a practical and user-friendly MLA-test which yields valid scores and reliably measures the state and the development of MLA. This innovative approach to MLA testing offers explicitness about the theoretical construct of the test and at the same time contributes to controlling the difficulty and discriminatory power of each test item. We argue that the test development process presented here is useful when developing tests of MLA and similar concepts, as it facilitates comparison of results as well as replicability. This development process and its detailed description also make it possible to adapt the MetaLearn MLA-test to other contexts.

Writing assessment literacy and its impact on the learning of writing: A netnography focusing on Duolingo English Test examinees

Language assessment literacy has emerged as an important area of research within the field of language testing and assessment, garnering increasing scholarly attention. However, the existing literature on language assessment literacy primarily focuses on teachers and administrators, while students, who sit at the heart of any assessment, are somewhat neglected. Consequently, our understanding of student language assessment literacy and its impact on learning remains limited. Moreover, previous language assessment literacy research has been predominantly situated in classroom assessment contexts, and relatively little scholarly attention has been directed to large-scale testing contexts. To address these gaps, this study investigated the role of language assessment literacy in students’ learning for the writing section of the Duolingo English Test (DET) through a netnographic approach. Twenty-three online videos posted by test takers were analyzed with reference to the existing conceptualizations to examine learners’ language assessment literacy of the DET writing section. Highlighting learners’ voices, we propose a new model relating writing assessment literacy to learning which has the potential to develop the learner-centered approach in language assessment literacy research. It elucidates the internal relationships among different dimensions of students’ language assessment literacy and their impacts on the learning of writing. We, therefore, discussed the findings of this study to argue for the importance of the transparency of assessment and the opportunities to learn provided by large-scale assessments, and to call for teachers' attention to students' language assessment literacy and understanding of the writing construct.

Expanding Kane's argument-based validity framework: What can validation practices in language assessment offer health professions education?

One central consideration in health professions education (HPE) is to ensure we are making sound and justifiable decisions based on the assessment instruments we use on health professionals. To achieve this goal, HPE assessment researchers have drawn on Kane's argument-based framework to ascertain the validity of their assessment tools. However, the original four-inference model proposed by Kane - frequently used in HPE validation research - has its limitations in terms of what each inference entails and what claims and sources of backing are housed in each inference. The under-specification in the four-inference model has led to inconsistent practices in HPE validation research, posing challenges for (i) researchers who want to evaluate the validity of different HPE assessment tools and/or (ii) researchers who are new to test validation and need to establish a coherent understanding of argument-based validation. To address these identified concerns, this article introduces the expanded seven-inference argument-based validation framework that is established practice in the field of language testing and assessment (LTA). We explicate (i) why LTA researchers experienced the need to further specify the original four Kanean inferences; (ii) how LTA validation research defines each of their seven inferences and (iii) what claims, assumptions and sources of backing are associated with each inference. Sampling six representative validation studies in HPE, we demonstrate why an expanded model and a shared disciplinary validation framework can facilitate the examination of the validity evidence in diverse HPE validation contexts. We invite HPE validation researchers to experiment with the seven-inference argument-based framework from LTA to evaluate its usefulness to HPE. We also call for greater interdisciplinary dialogue between HPE and LTA since both disciplines share many fundamental concerns about language use, communication skills, assessment practices and validity in assessment instruments.

Language assessment literacy

Numerous references to ‘new’ literacies have been added to the discourse of various academic and public domains, resulting in a multiplication of literacies. Among them is the term ‘language assessment literacy’ (LAL), which has been used as a subset of Assessment Literacy (AL) (Gan & Lam, 2022) in the field of language testing and assessment and has not been uncontested. LAL refers to the skills, knowledge, methods, techniques and principles needed by various stakeholders in language assessment to design and carry out effective assessment tasks and to make informed decisions based on assessment data (e.g., Fulcher, 2012*; Inbar-Lourie, 2008*[1]; 2013; Taylor, 2009*, 2013*).

Test score comparison tables: How well are they serving test users?

While several test concordance tables have been published, the research underpinning such tables has rarely been examined in detail. This study aimed to survey the publically available studies or documentation underpinning the test concordance tables of the providers of four major international language tests, all accepted by the Australian Department of Home Affairs for Australian visa purposes. To evaluate the concordance studies, we first identified the good practice principles in concordance research through a review of both the relevant literature and leading professional standards in the field of educational measurement and language assessment. Next, we reviewed the concordance studies against the identified good practice principles. Our findings revealed that the information supplied by test providers varied, with some making the full research papers available, whereas others providing little information about their underpinning research. None of the concordance studies fulfilled all the good practice principles. Based on the findings of this study, we offer recommendations for future concordance research in the field of language testing as well as suggestions for practice.

The Application of ChatGPT in Language Test Design – The What and How

ChatGPT, an artificial intelligence-based chatbot, has recently gone viral among users thanks to its ability to create human-like interactive experiences. It is now a trend to make use of this artificial intelligence (AI) tool in different fields of language education, including language testing and assessment. With a concern related to the potential and challenges of applying ChatGPT in designing language tests, this study has investigated the current use of the chatbot among language teachers at different schools and institutions. Participants are expected to share their experiences utilizing the chatbot to assess their learners. Data from a survey and in-depth interviews will reveal initial findings to answer questions related to the popularity of this chatbot among language teachers and its application in language test design. Based on the result discussion, the presentation concludes with some implications for language teachers to make the most of their future use of ChatGPT in the field of language testing.

Volume Introduction: Special Issue in Honour of James Dean Brown’s Five-Decade Contribution to Language Testing and Assessment

It is not every day, year, or even decade that a journal dedicates an entire issue to honour the contributions of an individual scholar in the field. The purpose of this Special Issue of Language Teaching Research Quarterly is to celebrate the work of James Dean Brown, a scholar who has made an enormous impact on the field of language testing and assessment. Following his retirement from the field, this Special Issue was created in JD’s honour and is dedicated to him and his many contributions to the field.

English Teachers' Perceptions of a Language Assessment Literacy Course

Language assessment literacy has gained recent attention in the field of language testing, particularly on teachers’ profile. However, the literature on LAL is limited regarding teachers’ perceptions of language assessment courses. In this paper, we used a case study method to characterize the perceptions of eighteen English language teachers into three components of an online assessment course: contents, activities, and impact on their professional development. For data collection, we used a questionnaire and a focus group interview. Findings indicate that the teachers perceived course contents as organized, relevant, and useful; they also considered test analysis and collaborative tasks as valuable. Regarding professional development, the teachers explained that test-analysis tasks made them aware of their mistakes in assessment. Finally, the teachers suggested that the course raised their awareness of what language assessment is and does. Based on these results, we provide recommendations for LAL courses elsewhere.

English Teachers' Perceptions of a Language Assessment Literacy Course

Language assessment literacy has gained recent attention in the field of language testing, particularly on teachers’ profile. However, the literature on LAL is limited regarding teachers’ perceptions of language assessment courses. In this paper, we used a case study method to characterize the perceptions of eighteen English language teachers into three components of an online assessment course: contents, activities, and impact on their professional development. For data collection, we used a questionnaire and a focus group interview. Findings indicate that the teachers perceived course contents as organized, relevant, and useful; they also considered test analysis and collaborative tasks as valuable. Regarding professional development, the teachers explained that test-analysis tasks made them aware of their mistakes in assessment. Finally, the teachers suggested that the course raised their awareness of what language assessment is and does. Based on these results, we provide recommendations for LAL courses elsewhere.

Application of Grounded Theory Methodology Using CEFR in the Field of Language Testing

The use of grounded theory methodology (GTM) is rare in the field of language testing as indicated by the literature survey. However, this study used GTM in a study on first year undergraduate students writing skills in Fiji. The study used the Common European Framework for Reference (CEFR) to evaluate student writing skills successfully at a university in Fiji. The CEFR is one of the most comprehensive frameworks for language evaluation that has been widely used by language testing organisations mostly in Western countries. This paper takes a methodological position rather than looking at empirical data from the research to engage novice users of GTM to apply it to the field of linguistics, specifically to language testing research. It provides step-by-step guide and a discussion of relevant literature that attest to the use of GTM in linguistics. The methodological contributions and the unique data set of the study will advance scholarly and social policy conversations on this topic. The study makes an original contribution to the body of knowledge on how grounded theory research methodologies can be applied to a longitudinal language testing research context. At present, language testing in higher education relies on data from conventional formative and normative assessments. Approaches such as grounded theory and longitudinal research design have rarely been used. The paper will benefit novice researchers such as Masters and PhD scholars as well as policy makers in applying the methodology to their studies.

Reflections on the past and future of language testing and assessment: An emerging scholar’s perspective

In its 40th year, Language Testing journal has served as the flagship journal for scholars, researchers, and practitioners in the field of language testing and assessment. This viewpoint piece, written from the perspective of an emerging scholar, discusses two possible future trends based on evidence going back to the very first issue of this journal. First, this paper outlines past efforts to describe and define the construct of second language communication, noting that much work has yet to be done for a more complete description in terms of interactional competence and nonverbal behavior. The second trend highlights the growing movement in applied linguistics toward research transparency through Open Science practices, including replication studies, the sharing of data and materials, and preregistration. This paper outlines work to date in Language Testing that encourages open practices and emphasizes the importance of these practices in assessment research.

Validation of the Vocabulary Size Test

The Vocabulary Size Test (VST) is one of the most commonly used assessment tools for measuring English vocabulary size in the field of language testing. Despite its common usage, only a limited number of validity and reliability studies have been carried out with regard to the VST. Besides, they were mostly predicated on the Rasch model. This validation study has attempted to reveal evidence for construct validity for the VST, and to this end, item response theory (IRT) analyses were performed based on the three-parameter logistic model (3PLM). The assumptions of IRT were investigated via factor analysis (unidimensionality) and Yen’s Q3 statistic (local independence). Detailed differential item functioning (DIF) analyses were conducted with Mantel-Haenszel, Lord's chi-square test, and Logistic regression methods to add evidence based on internal structure and to check fairness as a lack of measurement bias. The validation results with IRT showed that the 3PLM fitted the data better than the one- and the two-parameter logistic models. DIF results indicated that 10 items exhibited large DIF (seven favoring males and three favoring females). The results further showed that the guessing effect was not negligible for the VST.

Cheating on Testing: Cramming on Test Preparation

Professor Glenn Fulcher’s research has influenced language testing and assessment in two ways: in proposing a new understanding of data-driven speaking test design, and in drawing attention to the core identity of language testers. The former has had a powerful impact on practical speaking testing and assessment. The latter has asked a very essential question of what language testers do, exploring philosophical, historical, and social ideas from a pragmatic view of language testing and assessment. In particular, Fulcher (2015)’s chapters on meritocracy and values in language tests explain why we have been using tests throughout history to discern vital roles of language tests and yet why people attempt to cheat on them, in relation to ‘three Ms’ ideas including meritocracy, material success, and motivation. Within this view, this paper aims to review TOEFL cramming practices on test preparation, using Fulcher (2015)’s notion of cheating. Korean test takers study TOEFL cramming skills at test preparation companies to acquire admission to schools, called Hagwons in Korean. However, there is little research regarding cramming practices in terms of cheating. Collected data from my doctoral study (Kim, 2016) were reviewed. It was found that cramming skills as a form of cheating were explicitly taught in class, including use of actual test questions, memorisation of sample answers, and the use of a testing environment. The study revealed reasons to lead the cramming skills: (1) practical benefits of test scores, (2) speaking and writing sections and (3) investments of test preparation. The implications of the study are to raise awareness of the importance of links between cheating and cramming in the field of language testing and teaching.

Promoting Change in Language Teaching and Assessment at Policy and Practice Levels: An Interview with Hossein Farhady

ABSTRACT This interview highlights Professor Hossein Farhady’s academic life and his sustained contribution to research in the fields of English Language teaching, language testing and ESP over the past decades. Through a number of questions, the interview asks Professor Farhady about his pioneering role in the field of language testing in Iran and his highly valued contribution to issues related to English language teaching and material design. The interview takes the shape of a narrative about Professor Farhady’s past from early days of schooling in a village near Makou in Iran to his PhD studies at UCLA in California and his post-PhD research career in different countries around the globe. The interview summarises his accomplishment in leading research projects, disseminating research outputs, and training new teachers, researchers and language test and material designers. It also provides an interesting portrait of his vision for the field which is based on his experience of working in various higher education contexts over the past decades. Most importantly, the interview demonstrates Professor Farhady’s commitment to research and his dedication to translating research findings to action and bringing change about in policy and professional practice. Above all, the interview is a recognition of his lifetime achievements and an acknowledgement of his dedication, diligence and devotion to research in applied linguistics.

Language assessment literacy and test validation: Highlighting the role of language teachers

Τheories of validity and validation have always formed an essential part of academic work in the field of language testing and have constituted an area of much debate and contestation. On the other hand, both theorising and evidence-gathering processes have been the privilege of few professions in the field. Against the background of more recent developments such as a closer alignment of language teaching and assessment, increased attention to individual learning needs and accessibility of language testing and assessment, expanded conceptualisations of validity seem to be required, considering diverse stakeholders in the validation process. The paper will discuss several frameworks of validation, and their potential for including important stakeholder groups such as language teachers in language test validation will be explored with a view to their enhancing language assessment literacy (LAL). It will be argued that teachers’ roles as professional agents can enhance the quality of the validation argument and their LAL at the same time.

Grade-Related Differential Item Functioning in General English Proficiency Test-Kids Listening

Differential Item Functioning (DIF) analysis is always an indispensable methodology for detecting item and test bias in the arena of language testing. This study investigated grade-related DIF in the General English Proficiency Test-Kids (GEPT-Kids) listening section. Quantitative data were test scores collected from 791 test takers (Grade 5 = 398; Grade 6 = 393) from eight Chinese-speaking cities, and qualitative data were expert judgments collected from two primary school English teachers in Guangdong province. Two R packages “difR” and “difNLR” were used to perform five types of DIF analysis (two-parameter item response theory [2PL IRT] based Lord’s chi-square and Raju’s area tests, Mantel-Haenszel [MH], logistic regression [LR], and nonlinear regression [NLR] DIF methods) on the test scores, which altogether identified 16 DIF items. ShinyItemAnalysis package was employed to draw item characteristic curves (ICCs) for the 16 items in RStudio, which presented four different types of DIF effect. Besides, two experts identified reasons or sources for the DIF effect of four items. The study, therefore, may shed some light on the sustainable development of test fairness in the field of language testing: methodologically, a mixed-methods sequential explanatory design was adopted to guide further test fairness research using flexible methods to achieve research purposes; practically, the result indicates that DIF analysis does not necessarily imply bias. Instead, it only serves as an alarm that calls test developers’ attention to further examine the appropriateness of test items.

Language Assessment Literacy: Insights for Educating English Language Teachers through Assessment

At some point, language teachers need to be engaged in language assessment in their profession. Because language assessment is such a primary task for teachers, the field of language testing is encouraging research around the knowledge, skills, and principles that are foundational for sound assessment. In this paper, I provide a definition of Language Assessment Literacy (LAL), especially when it comes to teachers, by reviewing existing models. I then discuss ongoing issues in this area and end the paper by offering language teacher educators suggestions for fostering LAL among pre- and in-service teachers. In the article, I argue that, if more LAL initiatives take place, we are collectively raising the status and nature of language assessment and its impact on teachers’ professional development.

Facilitating Positive Washback through Authentic Assessment in EFL Assessment

Washback has become a main concern for researchers in the field of language testing proven by numerous studies which have been done. The degree, nature, and scope of washback depend on the kinds of tests employed and the contexts where the tests are administered. This present article is aimed at discussing how authentic assessment brings positive washback in EFL assessment. It is only a literature review, not an empirical research article. This article limits and focuses the discussion on the washback at micro level which means the test effects on teaching and learning at classroom level. It addresses how authentic assessment leads to positive washback in EFL assessment. It also discusses how each form of authentic assessment can uniquely generates positive washback in EFL assessment. In conclusion, authentic assessment brings positive washback in EFL classes. It links classroom instructions to students’ real-world experiences. It is suggested for EFL teachers to be literate of how to facilitate positive washback in EFL assessment and how to employ authentic assessment in assessment practices. Some kinds of further researches are also encouraged.

Language Test Misuse

ABSTRACT During the past two decades, an increasing number of European countries have introduced language requirements for residency, citizenship, and sometimes even for entry to the country and family reunification. As a result, democratic rights as well as basic human rights have come to depend upon an individual’s ability to obtain a certain score on a language test and the consequences of failing may be detrimental. In the field of language testing, this use of language tests is often referred to as test misuse, yet the term lacks a precise definition in the literature. In this paper we propose a definition of test misuse in relation to language tests for migration purposes and focus particular attention on low-literate adult migrants for whom the requirements pose a considerable barrier. The main purpose of this paper is to address the question why language tests are being misused in migration policies, exploring linguistic, political as well as test theoretical explanations. We suggest that a more central role of test misuse in validity theory is essential in order to remedy its lack of research focus in our field.

Advancing the International Recognition of the Locally-Produced GEPT: An Interview with Jessica Wu

ABSTRACT Jessica Wu is Research and Development Program Director at the Language Training and Testing Center (LTTC) in Taipei, Taiwan. Jessica has been deeply involved in the development and validation of the General English Proficiency Test (GEPT) in addition to a number of other foreign language testing programs. Her research focuses on speaking assessment and the impact of standardized language testing. Jessica is a founding member of the Asian Association for Language Assessment (AALA), for which she has served as both co-president (2016 to 2017) and president (2018 to 2019). She currently serves as an advisor for the development and administration of L1 tests (i.e., Hakka, Southern Min, and Indigenous language proficiency certifications) in Taiwan. Jessica has authored a number of publications, including articles and book chapters in the field of language testing, and has presented her work at conferences around the world. Most recently, she co-edited English Language Proficiency Testing in Asia: A New Paradigm Bridging Global and Local Contexts with Cyril Weir and I-Wen Su. Her contributions to language testing are significant and immeasurable, particularly with regard to Asia. The following interview with Jessica, which was conducted via an online meeting platform due to the COVID-19 pandemic, took place in January 2021.