Language Testing Research Research Articles

Open Science and the next generation of language testing research

Open Science and the next generation of language testing research

Sharing, collaborating, and building trust: How Open Science advances language testing

The Open Science movement is taking hold around the world, and language testers are taking part. In this viewpoint, I discuss how sharing, collaborating, and building trust, guided by Open Science principles, benefit the language testing field. To help more language testers join in, I present a standard definition of Open Science and describe four ways language testing researchers can immediately partake. Overall, I share my views on how Open Science is an accelerating process that improves language testing as a scientific and humanistic field.

Examining Fijian first-year university students’ English proficiency on CEFR levels

English is the main lingua franca among the various ethnolinguistic groups in Fiji. This results in a sociocultural problem when students enter universities from different high school backgrounds. To this end, this study analysed the level of difference in writing abilities at the beginning of the first year of study and at the end of the first-year program of undergraduate students. The goal was to generate new insights that could inform ongoing efforts towards ameliorating academic English language proficiency in Fiji. The study took place at a university in Fiji where 120 students were sampled at the beginning of the first year and at the end of their first year of university program. This study used the Common European Framework of Reference for Languages (CEFR) to evaluate student writing skills and for the development of curricula to meet students’ needs. The study employed quantitative research design to a longitudinal language testing research context. The CEFR was successful in gauging the student’s preparedness for university study and work life. Out of the 120 students, it was observed that from the 62 students who were at the A1 level at the end beginning of the year progressed to levels B1 and B2. Out of 62, 21 moved to B1 where 41 moved to B2. Further, 49 students from A2 level also progressed as 44 of these students had B2 and another 5 made substantial improvement by moving up to C1 level. The findings of this study can be used in improving support systems to enhance the smooth transition of multilingual students from high school through university and into the workforce. The analyses have the potential to provide insights into how Fijian undergraduate students are performing compared to students in other countries and how the EFL curriculum might be adapted to students’ needs.

James Dean Brown’s 50 Years of Work in Second Language Studies: A Systematic Review

Panahi and Mohebbi review James Dean Brown’s 50-years of research in language testing, curriculum development and research statistics with reference to an impressionistic framework for analysis containing two components with their subcomponents: Annotations (i.e., briefing and implications) and main concepts and themes (i.e., testing and teaching terminology, research design, research instruments, data analysis, and domains). The review was carried out in two phases: In Phase I, we (Ali Panahi and Hassan Mohebbi) reviewed Brown’s all works and extracted approximately 1100 main concepts and themes leading to 28 main entries for testing and teaching terminology. The issues he has examined are much more extensive; the first 10 topics and themes most widely investigated, in the order of frequency, are language testing and assessment, research and statistics, curriculum development and language programs, cloze tests, CRTs and NRTs, TESOL, ESL, applied linguistics and language testing, placement, standardized and proficiency tests, connected speech and reduced forms, pragmatics tests and issues, and reliability and validity. In Phase II, JD Brown provides a discussion and his personal reflections on the systematic review.

Shiken Across 23 Years: James Dean Brown’s Statistical Advice

J.D. Brown (JD) served as the author of an instructional column titled "Statistics Corner" in the Shiken: JALTTesting & Evaluation SIG Newsletter for an extensive period spanning over two decades, from 1997 to 2019.This publication was under the auspices of the Testing and Evaluation Special Interest Group, a subdivision ofthe Japanese Association of Language Teaching. The audience for the columns was diverse, and includedlanguage teachers, graduate students, and language practitioners encountering real-world issues with languageassessment. Throughout his tenure, JD addressed over 40 reader-submitted inquiries related to testing andquantitative research. The subjects explored in these columns can be broadly classified into two thematic areas:Second Language Testing and Second Language Research. This present article aims to provide an exhaustiveexamination of the diverse range of topics covered within each of these thematic categories and to trace theevolution of these subjects over the course of JD’s twenty-year involvement with the newsletter. Its goal is tohelp situate JD’s abiding concern with connecting theory and practice.

Linking second language speaking task performance and language testing

Abstract This written version of a plenary to the Language Testing Research Colloquium relates research into second language speaking task performance to language testing. A brief review of models of communicative competence and of speaking is provided. Then, two major areas within testing are discussed: the concepts of difficulty and ability for use. The next section covers research into spoken task-based performance, covering effects from task conditions, and from task characteristics. In addition, the measurement of such performance is described and briefly compared with performance rating in testing. Then, the final section relates the task research findings to language testing. A framework for testing spoken performance is outlined, and the general claim made that effective sampling through tests, in order to generalise to real-world performance, can usefully draw on findings from second language task research, as well as the distinction between Conceptualiser and Formulator processes.

Language, educational inequalities and epistemic access: crafting alternative pathways for Fiji

PurposeThe goal of this article is two-fold. The first is to contribute new insights to inform education policies for addressing the underlying educational inequalities and injustices that are caused by lack of epistemic access in the context of Fiji higher education. The second is to explore how the Grounded Theory Methodology can be applied to longitudinal language testing research that seeks to reverse epistemic injustices and educational inequalities in Fiji and other comparable multilingual countries.Design/methodology/approachTo explore how the Grounded Theory Methodology can be applied to longitudinal language testing research that seeks to reverse epistemic injustices and educational inequalities in Fiji and other comparable multilingual countries. The study was conducted at a university in Fiji where 120 students were sampled at the beginning of the first year and at the end of their first year of university programme. The same cohort was tracked throughout the project, out of which 30 students were interviewed at the end of the first year.FindingsThe four indicators include: (1) lack of teaching and learning resources, (2) language barriers, (3) problems with the medium of instruction and (4) shortage of experienced teachers.Originality/valueAlthough widely acknowledged in previous studies from elsewhere, the indicators of educational inequalities identified in this study are worth reporting on due to the unique socio-cultural and linguistic context of Fiji.

English-Language Proficiency Requirements for Migration to Canada, Australia, the United Kingdom, and the United States, and the Implications for Language Testing Research

Language proficiency testing has long played a role as both facilitator and barrier in global migration. This paper outlines how Canada and other traditional migrant destination countries use English-language proficiency testing for immigration, asylum and resettlement, and citizenship. It identifies commonalities between these three related areas of migration and the covert values that are often embedded in the required test scores. It identifies different approaches for conducting research under the umbrella of Shohamy’s (2001) Critical Language Testing and concludes with language testing research and advocacy suggestions specific to the Canadian context.

Application of Grounded Theory Methodology Using CEFR in the Field of Language Testing

The use of grounded theory methodology (GTM) is rare in the field of language testing as indicated by the literature survey. However, this study used GTM in a study on first year undergraduate students writing skills in Fiji. The study used the Common European Framework for Reference (CEFR) to evaluate student writing skills successfully at a university in Fiji. The CEFR is one of the most comprehensive frameworks for language evaluation that has been widely used by language testing organisations mostly in Western countries. This paper takes a methodological position rather than looking at empirical data from the research to engage novice users of GTM to apply it to the field of linguistics, specifically to language testing research. It provides step-by-step guide and a discussion of relevant literature that attest to the use of GTM in linguistics. The methodological contributions and the unique data set of the study will advance scholarly and social policy conversations on this topic. The study makes an original contribution to the body of knowledge on how grounded theory research methodologies can be applied to a longitudinal language testing research context. At present, language testing in higher education relies on data from conventional formative and normative assessments. Approaches such as grounded theory and longitudinal research design have rarely been used. The paper will benefit novice researchers such as Masters and PhD scholars as well as policy makers in applying the methodology to their studies.

Computer-Aided Dynamic Evolution Analysis of Roof Decoration Art in Northern Jiangsu Based on Low-Frequency Remote Sensing Image Technology

In the application of remote sensing image fusion based on wavelet packet, the wavelet packet frequency shifting algorithm used to adjust the frequency sequence of wavelet packet decomposition is introduced to solve the frequency aliasing phenomenon caused by interval sampling. The roofs of ancient residential buildings in northern Jiangsu have a unique art. Features: Discusses the roof decoration art of residential buildings in northern Jiangsu, aiming to provide some references for modern architecture. It shows that the roofs of residential buildings are deeply influenced by the land culture and marine culture, and have both normative, delicate and exaggerated factors. . The roof ridge of the folk houses embodies the uniqueness and rich connotation of Nantong culture, and we need to actively explore the beneficial elements in it. By examining the overall trend of international computer-assisted language testing research, high-yielding countries, authors, institutions and high-cited journals, research hotspots and high-innovation literature.

Promoting Positive Washback of the Language Exam with the Assistance of Teachers

This paper aims to show how the impact of high-stakes language exams (washback in testing literature) has been conceptualized in language testing research and to reveal the findings of some selected studies to represent the possible negative and positive washback effects that appear to impact on language teaching and learning. Apart from representing the models of washback research and previous research studies, I also describe certain factors that can facilitate positive washback on teaching and thus learning. This review is not intended to be exhaustive, but merely to illustrate the possible positive washback of the language exam and to make suggestions for improving the quality of teaching methodology and test preparation methodology.

Future challenges and opportunities in language testing and assessment: Basic questions and principles at the forefront

In this invited Viewpoint on the occasion of the 40th anniversary of the journal Language Testing, I argue that at the core of future challenges and opportunities for the field—both in scholarly and operational respects—remain basic questions and principles in language testing and assessment. Despite the high levels of sophistication of issues looked into, and methodological and operational solutions found, outstanding concerns still amount to: what are we testing, how are we testing, and why are we testing? Guided by these questions, I call for more thorough and adequate language use domain definitions (and a suitable broadening of research and testing methodologies to determine these), more comprehensive operationalizations of these domain definitions (especially in the context of technology in language testing), and deeper considerations of test purposes/uses and of their connections with domain definitions. To achieve this, I maintain that the field needs to continue investing in the topics of validation, ethics, and language assessment literacy, and engaging with broader fields of enquiry such as (applied) linguistics. I also encourage a more synthetic look at the existing knowledge base in order to build on this, and further diversification of voices in language testing and assessment research and practice.

Anxiety as a Psychological Barrier to Students’ Speaking Fluency: IELTS as a Case Study

This study investigates the correlation between Foreign language anxiety and speaking fluency among IELTS test-takers at an X English Language centre in Morocco. The main objective of this paper was to examine whether there is a link between Foreign Language Anxiety (the independent variable) and English-Speaking fluency (the dependent variable). To verify whether there is an association between the two variables under investigation, to test the hypothesis, and answer the research question, two tests were used as research instruments. Students were asked to fill in a Foreign language anxiety questionnaire to estimate the frequency of anxiety and take an IELTS Speaking mock exam to measure the smoothness of their oral delivery. The data was collected in the X language centre during the academic year (2020-2021). The participants were 30 male and female students divided into three classes. In order to analyse the scores and frequency of responses that students obtained in the two tests, a Pearson correlation was used as a statistical procedure. The study results indicate a nonsignificant correlation between Anxiety and Fluency in the context under study. Therefore, the study concluded that other factors, such as fear of negative evaluation, are intervening and causing a decline in fluency among IELTS test takers, broadening the spectrum of research within language testing research.

Critical Language Testing, Multilingualism and Social Justice

AbstractThe paper reports on trends in language testing taking place over the years and aim at critical perspectives of testing and promoting inclusion, equity and justice. It begins with critical theories by Messick, Foucault and Bourdieu, leading to critical language testing (CLT) which focused on consequences and uses of tests. Given the power of tests and their detrimental effects questions were raised regarding the impact of language tests on individuals and society. Based on theories of sociolinguistics and applied linguistics it was realized that bi‐multilingual students, especially immigrants in the first years of migration and beyond continue to use their first languages as resources for academic functioning, referring to it as their ‘full language repertoire’. These languages are needed especially when processing academic school tests, presented in the new language which students have not yet acquired. The paper reports on studies where immigrant students are tested in multilingual tests and reach significantly high scores compared to those tested only in the new language. In experiments, using a variety of tools, students showed positive attitude to the procedure as they felt recognition and respect. Multilingual tests need to be practiced in schools to avoid language rights violations. Multimodal assessment methods are also proposed as additional ways for expanding the underpinnings of language construct. Lastly, it is recommended that language testing researchers and practitioners conduct research to identify cases of violate language rights and suggest new assessment models to overcome it, leading to increased social justice.

L2 English vocabulary breadth and knowledge of derivational morphology: One or two constructs?

Derivational morphology (DM) and how it can be assessed have been investigated relatively rarely in language learning and testing research. The goal of this study is to add to the understanding of the nature of DM knowledge, exploring whether and how it is separable from vocabulary breadth. Eight L2 (second or foreign language) English DM knowledge measures and three measures of the size of the English vocabulary were administered to 120 learners. We conducted two confirmatory factor analyses, one with one underlying factor and the other treating vocabulary breadth and DM as separate. As neither model had a satisfactory fit without introducing a residual covariance to the two-factor model, we conducted an exploratory factor analysis, which suggested two separate DM factors in addition to vocabulary breadth. Regardless, the analysis demonstrated that the DM knowledge was separate from learners’ vocabulary breadth. However, learners’ vocabulary breadth factor still explained a substantial amount of variance in learners’ performance on DM measures. We discuss theoretical implications and implications for L2 assessment.

Glenn Fulcher’s Thirty-Five Years of Contribution to Language Testing and Assessment: A Systematic Review

The present systematic review examines Glenn Fulcher’s contributions, works, philosophy, and research in language testing and assessment. The data includes his published articles, book chapters, books and interviews (except the one in this special issue) relevant to language testing and assessment from 1987 to March 2022. This study is conducted in two stages: From the sources, Ali Panahi and Hassan Mohebbi derived 127 commonly used main themes, 43 statistical and instrumental concepts, and 14 domains to create a framework for the analysis. We discovered that his research interests were wide-ranging. However, there was a focus on assessing speaking, rating scale design, validity, language assessment literacy and pedagogy, and the broader understanding of the role of assessment from a philosophical and societal perspective. Our analysis provides an overall understanding of the main themes, key concepts and major implications of Glenn Fulcher’s work. In the second stage of the study, Glenn presents his personal discussion and reflection of this systematic review.

Representation of Students with Special Educational Needs in Language Testing Research

The presence of students identified as English Language Learners (ELL) and Special Education Students (SES) in mainstream settings is growing. However, representation of these populations within language testing research remains unknown. This is important because language testing scores are increasingly used to evaluate student learning, teacher quality, and school quality. This meta-synthesis examined the representation of these two exceptional populations within the mainstream language testing literature published in two language testing journals. This meta-synthesis utilized inductive analysis and descriptive statistics. Findings indicated highly skewed representations of ELL and SES within the research corpus of the reviewed journals. Further analysis of participant demographics, research settings, test formats and test modalities are reported and discussed. Results highlight critical needs for the fields of exceptional education regarding the language testing of ELL and SES populations. The paper closes with suggestions regarding such needed research.

A sequential approach to detecting differential rater functioning in sparse rater-mediated assessment networks

Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting DRF may be limited in sparse rating designs, where it is not possible for every rater to score every student. In these designs, there is limited information with which to detect DRF. Sparse designs can also exacerbate the impact of artificial DRF, which occurs when raters are inaccurately flagged for DRF due to statistical artifacts. In this study, a sequential method is adapted from previous research on differential item functioning (DIF) that allows researchers to detect DRF more accurately and distinguish between true and artificial DRF. Analyses of data from a rater-mediated writing assessment and a simulation study demonstrate that the sequential approach results in different conclusions about which raters exhibit DRF. Moreover, the simulation study results suggest that the sequential procedure results in improved accuracy in DRF detection across a variety of rating design conditions. Practical implications for language testing research are discussed.

Putting Low Alphabetic Print Literacy on the Language Testing Agenda

ABSTRACT To date, language testing research has devoted little attention to adult L2 learners with low levels of alphabetic print literacy (LESLLA), even though this population makes up for a substantial proportion of the candidature of language tests used for migration purposes. This special issue focuses on LESLLA learners, shows how literacy impacts test performance beyond what might be considered construct relevant, and how a focus on LESLLA learners invites a reconceptualization of commonly held notions of literacy, test misuse and even validation.

A Revisit of Zumbo’s Third Generation DIF: How Are We Doing in Language Testing?

ABSTRACT The purpose of this study is to review the status of differential item functioning (DIF) research in language testing, particularly as it relates to the investigation of sources (or causes) of DIF, which is a defining characteristic of the third generation DIF. This review included 110 DIF studies of language tests dated from 1985 to 2019. We found that DIF researchers did not address sources of DIF more frequently in recent years than in earlier years. Nevertheless, DIF research in language testing has expanded with new DIF analysis procedures, more grouping variables, and more diversified methods for investigating sources of DIF. In addition, in the early years of DIF research, methods to identify sources of DIF relied heavily on content analysis. This review showed that while more sophisticated statistical procedures have been adopted in recent years to address sources of DIF, understanding sources of DIF still remains a challenging task. We also discuss the pros and cons of existing methods to detect sources of DIF and implications for future investigations.