Language Test Development Research Articles

MAIN STAGES OF THE LANGUAGE TEST DEVELOPMENT PROCESS AND VALIDATION ANALYSIS METHODS: THE CASE OF GAZI TÖMER

Testing is the most widely used assessment tool in measuring the development process of students' languagecompetence. Therefore, the existence of valid and reliable language tests appropriate to student achievements in education is of great importance. Research on language test development in foreign language teaching in Kazakhstan is insufficient. The aim ofthis study is to practically demonstrate the basic stages of the test development process and how to conduct content validity analysis with the example of the proficiency test of foreign language (Turkish) prepared by Gazi TÖMER. The research method was based on a quantitative research design. The sample of the research for validation was consisted of 7 experts with experience in Gazi TÖMER. For reliability, there were a total of 66 students studying at Abai KazNPU. The first part of the study was examinedfor the content validity of the relevant test. For this purpose, based on the criteria determined by Webb for the evaluation of learning outcomes, the opinions received from field experts were evaluated using the Lawshe technique. The CVR value of the 45-question test was .96, and the CVI was “significant”. The second part of the study was about reliability. For this purpose, the difficulty level of each item was examined based on the data obtained from the pilot application of the test. Data were analyzedusing Excel and SPSS programs. Additionally, the KR-20 value was calculated and found to be .80, and the test was considered reliable. It is believed that the steps and results included in the study will guide the process of creating valid and reliable tests in various fields.

Evaluating Iranian L2 Teachers’ Assessment Literacy for L2 Pragmatics by Applying the CEFR’s Pragmatic Competence Model: Possible Sociocultural-Informed Solutions

Nearly all multidimensional models of communication competence have pragmatic competence at their core. Proper assessment of second language (L2) pragmatics makes many demands on L2 teachers, both in terms of understanding the construct and in language test development. Therefore, being assessment literate helps teachers in developing effective tests geared toward educational outcomes. Despite the importance of assessment literacy (AL) for pragmatics, the field is still under-researched. Mindful of this fact and the significance of pragmatics in L2 learning and assessment, this study adopted the Common European Framework of Reference (CEFR) model of pragmatic competence to examine Iranian L2 teachers’ AL for L2 pragmatics and linked the results to the Sociocultural Theory (SCT) for formulating theoryinformed answers to the problems in the teachers’ assessment of L2 pragmatics. Group interviews were run with 67 participants and then qualitative and quantitative content analyses were performed. The paper discusses the identified problems in the assessment of L2 pragmatics and offers recommendations for raising L2 teachers’ AL for L2 pragmatics.

The Opportunities and Burdens of Online Test Processes

English language testing has been challenged by the COVID-19 pandemic as it required the test developers to instantly develop and implement online language tests in their unique contexts. In this regard, this study aims to investigate EFL language test writers’ reflections on their planning, development, administration, reporting, and reflection processes of a test in the emergency remote teaching environment and display the potential opportunities and burdens these processes have brought about. The data were collected by utilizing semi-structured interviews with nine test writers who had testing experience in language preparatory classes at three state universities in Turkiye. The results indicated that the test writers were mainly challenged by ensuring the security of the exams, adapting question types to online platforms, and dealing with technical problems which resulted in additional workload. However, amidst these challenges, the study also identified notable opportunities, including the practical application of exam procedures, the digital transformation of exams with a variety of resources, item banking, ecological benefits, and fostering professional development. Overall, this compulsory experience during the pandemic indicated numerous conveniences to enhance language testing and contribute to the development of robust language assessment and testing in the future.

An AI Generated Test of Pragmatic Competence and Connected Speech

Language testing is witnessing increasing interest in the potential for AI to support test development and validation. To date, published research involving AI in language testing has typically been conducted in the context of high-stakes proficiency tests and the potential for this technology to support local language testing is under-researched. The current study seeks to address this gap by reporting on the piloting of an AI generated language test in the context of a university in the UK. The focus of the paper was selected to explore two key areas in the work of J.D. Brown, namely pragmatics and connected speech. In the study, international students with English as a second language completed an AI generated test of pragmatics created from a transcript of spontaneous interaction, and a test-taking strategy questionnaire. Results demonstrated that the test did not provide a sufficiently reliable measurement of test takers’ pragmatic competence, although reliability estimates did improve with the removal of problematic items. Implications for language test development involving AI are discussed.

The effect of task complexity and gender differences on EFL learners’ writing performance

The current study investigates the Effect of Task Complexity and Gender Differences on EFL Learners’ Writing Performance. Task complexity is the inherent cognitive demands of the tasks imposed on the learners by the structure of the tasks. It is investigated along the resource-dispersing variables of pre-task planning time, post-task editing time and prior knowledge. This study also tries to intertwine the concepts of task complexity and sociolinguistics (i.e., gender differences). The participants are 160 high school students (80 males and 80 females) studying at international schools in Riyadh City, Saudi Arabia. The writing topics were selected among the essay prompts similar to those of the IELTS test using the cause-and-effect essay task type. A series of T-Test and one-way ANOVAs were used for detecting the significant differences within the same gender and across genders, Cronbach's alpha (α) for the task reliability, Wilk’s Lambda for variable contributions, Partial Eta squared (ηp2) for the effect size, and the Observed Power (or post-hoc power) for the test statistical power. This study is thought to be laden with manifold implications for varied stakeholders within the realm of didactics, including language teachers, learners, syllabus designers, test developers, educational bodies and many other individuals in the ministries of education and higher education. Should this be done, it can contribute to improving EFL undergraduate students’ ability to a level of writing production that is more likely to be satisfactory. The current study found different effects of task complexity on EFL learners’ writing performance across gender. There were both similar and different impacts on CAF metrics between the writing tasks across genders. The results indicate a partial support to the predictions of Skehan and Foster (2001) and those of Robinson (2005) where there is a partial increase in the CAF scores along the resource-dispersing variables. Conversely, findings show no support to the Gender Similarities Hypothesis (GSH) by J. S. Hyde (2005). This study also explored the effect percentage of task complexity along the resource dispersing variables planning time, editing time and prior knowledge on CAF. Besides, the researcher sorted the significant CAF metrics within same gender and across genders according to their occurrence and percentage in the performance of the current student participants.It is recommended that pedagogical tasks should be sequenced for learners on the basis of increasing in their cognitive complexity to help promote learners' performance in terms of accuracy, fluency, and complexity. In addition, educators should motivate male students to engage more and perform better in the writing classes; syllabus designers should introduce new teaching methodologies that help bridge the gap in gender differences between boys and girls.

Assessing the dimensionality of Chinese as a second language reading: a confirmatory factor analysis approach

Abstract Second language (L2) reading is a complex construct, and there is no consensus about its dimensionality. The investigation of the dimensionality of L2 reading ability is of great significance since it can provide important implications for teaching and testing. Understanding of the factor structure, subskills, and sub-knowledge of L2 reading is the starting point for language teachers and test developers to plan a syllabus, describe students’ reading proficiency, and develop reading assessments. This study investigated the factor structure of Chinese L2 reading, which has been less commonly studied compared to alphabetic languages. Three hypotheses derived from the literature review have been tested: 1) L2 reading is a unitary skill; 2) L2 reading is bi-divisible; 3) L2 reading is tri-divisible. A series of confirmatory factor analyses shows that the correlated three-factor model is the most appropriate for explaining the factor structure. L2 Chinese reading comprehension can be conceptualized as consisting of three inter-correlated components: lower-level decoding, interim-level textbase construction, and higher-level situation-model building. This study also explores subcomponents of each factor, highlighting Chinese-specific linguistic features and corresponding cognitive processes. Implications for future research, teaching, and testing are provided.

Developer involvement and COI disclosure in high-stakes English proficiency test validation research: A systematic review

Language test developers play a critical role in gathering evidence to support test score interpretations and real-world uses, but independent research is necessary to put claims of validity to the test (Kane, 2013). This study systematically reviews the prevalence of developer involvement in high-stakes English proficiency test research published in five peer-reviewed language testing journals from 2016 to 2021, focusing on aspects of validity addressed, research methodology, and disclosure of relevant conflicts of interest (COI). Nearly half of the 181 included studies featured some degree of developer involvement, in terms of authorship or funding, and developer-involved studies had some notable methodological advantages, including larger sample sizes and access to official test materials/data. COI reporting was generally poor, with only 4-7% of studies with potential conflicts making adequate disclosures. We discuss implications of these findings in terms of evaluating tests and make recommendations for greater transparency in COI reporting and validation research.

DESIGNING ENGLISH LANGUAGE TESTING USING A WEB-BASED MONITORING PLATFORM

As the learning activities were switched to online because of the COVID-19 pandemic, changes in the way of learning also affects the facilities used both in the learning and the test activities. This situation requires language test developers to think more creatively to make the test online and to provide safe environment for the test without compromising the validity of the test. This paper aims to describe the steps of choosing the appropriate application used for the online language test, trials of the test, and the user manual made for the test. So, it requires new guidelines and new way of proctoring the online language tests. There are 4 steps that accommodate the needs, 1) Technology Selections, 2) User Manual Making, 3) Trials, and 4) Question Making. The process of designing a language assessment following the steps sends a platform suitable for the needs. Because of the restrictions from the COVID-19 , the platform can provide a valid and safe environment for the language testing because it was conducted based on web monitoring. Then, the platform can provide webcam monitoring and variations of questions in language assessment, such as audio for listening tests. User Manuals help the test taker in understanding on how to access the platform, and the trials made it more comprehensive. From the questionnaiers, it can be concluded that the platform help to solve the problems and the test can be conducted without problematic issues.

Decision making in large-scale language testing: Intersections of policy, practice and research

In this article, I delve into the intricacies of strategic decision making when a large-scale language test is reformed. Adopting a case study approach, I examine the activities involved in the reform of the College English Test (CET), an English language test for university students administered nationwide in the Chinese mainland (Zhang, 2022; Zheng & Cheng, 2008). In its 30+-year history, the CET has undergone several revisions and reforms (see Jin, 2010; 2020 for an overview). To illustrate how the major policies on test revision and reform have been formulated and implemented in this particular context, I present a brief overview of some major decisions that have been made over the decades. Through these cases, I hope to demonstrate the intricate interactions among policy intentions, professional requirements, ethical considerations, practical constraints, as well as the role of language testing professionals when strategic decisions are made to ensure a sustainable and healthy development of large-scale and high-stakes language tests.

Benchmarking performances of L2 spoken Lithuanian produced by young learners

The Common European Framework of Reference (CEFR), widely recognized as one of the most important documents in foreign language teaching, defines the levels of communicative language proficiency. The common points of reference are primarily meant to assist foreign language teachers and test developers striving to adhere to a unified understanding of CEFR levels. Yet it is not uncommon among practitioners to claim a particular level of a task, text or sample of learner performance drawing merely on their individual experience and intuition, with only limited knowledge of the original descriptor scales. The main purpose of this paper is to describe general procedures for relating foreign language tests and learner performances to the CEFR as they are prescribed by the Association of Language Testers in Europe (ALTE). The five stages of the linking process are familiarisation, specification, standardisation training and benchmarking, standard setting and validation. While the full implementaion of the linking process is a complex undertaking mostly applied in the context of high stakes examinations, certain stages of the linking process could be implemented by individual language schools and programmes. Benchmarking is one of such stages. It involves identification of illustrative learner performances for different proficiency levels. This is a stage which also has its relevance in daily teaching practice and could be easily incorporated in routine assessment procedures thus increasing foreign language teachers' awareness of CEFR levels. Therefore the discussion of the formal linking procedures in this paper is followed by a demonstration of an exercise in benchmarking. It is meant to acquaint the Lithuanian readers with the complexity of the linking process and encourage a more principled approach to level allocations. The exercise in alignment with CEFR level decribed here involves L2 spoken Lithuanian produced by five foreign learners of young age. Sample performances were taken from a corpus of L2 Lithuanian compiled within the XXX project. Since the subjects are children of 11–12 years, another important challenge in the alignment with the CEFR is children's maturity and the necessary adaptation of CEFR descriptors for young learners. Qualitative analysis of five learner samples offers a number of insights into practicalities of linking procedures and discusses ways to ensure reliability and objectivity.

Underrepresentation of the construct in Spanish standardized foreign language exams: a computer assisted exam proposal

Standardized tests are a reality in a globalized world and, consequently, in the Spanish system. Language test developers have already claimed that the current model of assessment has little to do with real language use, as factors such as the shift from paper to computer, information search and other uses of language or multimodal communication are not being considered (García Laborda & Fernández Álvarez, 2021). This theoretical paper proposes (1) a novel construct for testing that incorporates new functionalities which have been shown able to test appropriately language performance in academic settings; and (2) the proposal and design of an online multimodal exam that adapts to the new needs for language assessment in the University Entrance Examination. It begins with an analysis of current research to show the need for a change in the construct, and concludes by proposing a model for this high-stakes exam, based on a new construct of language proficiency and multimodal interaction for computer-based exams. The paper concludes that this model can change the way we understand technology-based exams and could be applicable to most current exams.

ISTORIJA JEZIČKOG TESTIRANjA

The paper presents the history of the development of language test- ing in the light of various, once prevailing theories about the nature, acquisition and use of language. The initial chapter is dedicated to the first, traditional or pre-scientific phase in the development of language testing, which is related to the end of the 19th century and the then dominant grammar-translation method. What follows is a presentation of the period of discrete point testing, resulting from a structuralist interpretation of language, as well as the attempt to overcome the shortcomings of this type of testing. The final chapters deal with the theory of communicative competence, which has led to an essential and final turning point in relation to traditional methods. We also paid attention to the new challenges that the language testing process faced in the conditions of a pandemic and a complete transition to online teaching, and the answers to the questions that were recognized as problematic could be given by some future research.

Artificial Intelligence-Generated and Human Expert-Designed Vocabulary Tests: A Comparative Study

Artificial intelligence (AI) technologies have the potential to reduce the workload for the second language (L2) teachers and test developers. We propose two AI distractor-generating methods for creating Chinese vocabulary items: semantic similarity and visual similarity. Semantic similarity refers to antonyms and synonyms, while visual similarity refers to the phenomenon that two phrases share one or more characters in common. This study explores the construct validity of the two types of selected-response vocabulary tests (AI-generated items and human expert-designed items) and compares their item difficulty and item discrimination. Both quantitative and qualitative data were collected. Seventy-eight students from Beijing Language and Culture University were asked to respond to AI-generated and human expert-designed items respectively. Students’ scores were analyzed using the two-parameter item response theory (2PL-IRT) model. Thirteen students were then invited to report their test taking strategies in the think-aloud section. The findings from the students’ item responses revealed that the human expert-designed items were easier but had more discriminating power than the AI-generated items. The results of think-aloud data indicated that the AI-generated items and expert-designed items might assess different constructs, in which the former elicited test takers’ bottom-up test-taking strategies while the latter seemed more likely to trigger test takers’ rote memorization ability.

Modern principles of developing the subtest “Reading and Use of Language” of the TORFL-II

The article is devoted to the topical problem of modern principles of developing tests of Russian as a foreign language (RFL), taking into account the world experience. The purpose of the article is to summarize the modern principles of language test development and to show how they are implemented in the new tests of Russian as a foreign language. The materials of the article include the research papers of Russian and foreign authors in the sphere of methodology over the past 20 years, as well as modern formats of testing in foreign languages. At the first stage of the study, general scientific methods of generalization, systematization and structuring were used. At the second phase, a new format of the RFL test Reading and Use of Language (B2) was modelled, combining language and communication competence testing. At the third stage, an experiment was conducted to test the new format. 48 foreign master students studying the program Russian Language and Russian Culture in the Aspect of Russian as a Foreign Language in Saint Petersburg State University took part in the experiment. It was concluded that the modern language test, in accordance with the basic cognitive and communicative principle of learning and control, should be based on the following principles: testing skills in different types of speech activities mainly on the text material; interdependence between the type of the task and the speech genre of the text being created/used in the task; basing on a linguistic and didactic description of the communicative competence level; integrative approach; using different types of test tasks within one subtest; the principle of increasing complexity of tasks; taking into account the complexity of each task in its assessment; task feasibility according to students educational level; taking into account the values of the multicultural world; taking into account international experience; basing on reliability and validity criteria of test tasks. These principles implemented in the new TORFL-II subtest format Reading and Use of Language are presented in the article. The implementation of modern test principles should ensure that all speech control facilities are systematically allocated to the appropriate level and parameters for their assessment. The above-mentioned principles of test creation and the example of their implementation can be taken as the basis of a full-fledged system of control and measurement materials based on linguistic and didactical descriptions of each level.

Developing and Validating a Computerized Adaptive Testing System for Measuring the English Proficiency of Taiwanese EFL University Students

ABSTRACT Prompted by Taiwanese university students’ increasing demand for English proficiency assessment, the absence of a test designed specifically for this demographic subgroup, and the lack of a localized and freely-accessible proficiency measure, this project set out to develop and validate a computerized adaptive English proficiency testing (E-CAT) system for Taiwanese EFL university students. Drawing on the guidelines posited by L2 testing specialists, we devised and followed a six-stage procedure to develop this E-CAT system: determining the test purpose, defining the construct, developing the test items, designing the administration processes, performing the field testing, and constructing the E-CAT system. Upon its completion, we performed two validation studies on the simulated data and as such offered the backing for the generalization inference and the explanation inference, which combined to lend support for the validity argument for the E-CAT score-based interpretations and uses. This project highlighted the importance of test localization, foregrounded the utility of computerized adaptive testing and item response theory in language test development and validation, and generated a localized, free-of-charge English proficiency test for Taiwanese university students to satisfy the graduation benchmark requirement and/or to demonstrate their English proficiency levels when job hunting.

Students' Grammatical Errors in Essay Writing: A Pedagogical Grammar Reflection

Students of English as a Foreign Language (EFL) are expected to master the fundamental of grammar so they can produce good essays. However, despite having learnt English at secondary or university level, students tend to make many grammatical errors in their writing. This study presents the grammatical errors made by college EFL students in their essays and the pedagogical implications viewed from those errors. This is a descriptive research with 30 second-year students who enrolled Essay Writing class as participants. Thirty written essays produced by the students were analyzed for the grammatical errors. The findings revealed that there were 368 grammatical errors found in the students’ essays. The most common one was in verb use (48%). Besides, the errors were also found in nouns (12%), prepositions (8%), determiners (8%), pronouns (8%), adverbials (6), adjectives (5%), and conjunctions (5%). If teachers do not assist students to comprehend the concept of parts of speech, and essential and nonessential clauses, these students will continue to make errors in their more advanced writing. The findings may have useful pedagogical implications for English language teachers, syllabus designers, and test developers. Understanding students’ difficulties and providing appropriate grammar instructions are the keys to teach grammar.

Assessing the validity of an IAU General English Achievement Test through hybridizing differential item functioning and differential distractor functioning

The current study sought to examine the validity of a General English Achievement Test (GEAT), administered to university students in the fall semester of 2018–2019 academic year, by hybridizing differential information (DIF) and differential distractor function (DDF) analytical models. Using a purposive sampling method, from the target population of undergraduate students studying in different disciplines at Islamic Azad University (IAU), (Isfahan branch), a sample of 835 students taking GEAT were selected. The 60-item multiple-choice test comprised four sub-sections; namely, vocabulary, grammar, cloze test, and reading comprehension. The students’ test scores served as the targeted data and the validity of the test was examined through the application of Cochran-Mantel-Haenszel (CMH) and multinomial log-linear regression models for detecting DIF and DDF, respectively. To account for the assumption of uni-dimensionality, the test sub-sections were analyzed independently. Furthermore, the assumption of local independence was checked based on correlational analysis and no extreme values were observed. The results of the study identified five moderate-level DIF items and one DDF item signaling an adverse effect on test fairness due to the existing biased items. Notably, these findings may have important implications for both language policymakers and test developers.

Complementary evidence in the early-stage validation of language tests: Classical Test Theory and Rasch analyses

Test validation may more aptly be conceived of as the process of designing language tests responsibly. While a good test gains in reputation as it is administered over time, the early stages of its validation are perhaps the most critical. There is now general agreement that the validation process should be reported in the form of an argument that brings together multiple sets of evidence to justify the design and implementation of the measurement instrument, the language test. The format of such integration is, however, still contestable ground. Referring to an example of language test design and development, this paper seeks to demonstrate how a framework for responsible test design may be employed to achieve such an integrated argument, as well as how two of the methodological tools most frequently employed to muster empirical evidence for validating test design, namely classical test theory (CTT) and Rasch analyses, complement each other in designing tests responsibly. While most language tests designed in South Africa have used CTT, the employment of Rasch analyses has been more limited. A secondary aim of the paper is therefore to provide applied linguists who work in the subfield of language testing with an example of how the latter kind of analysis can complement the former. In all, however, these disparate approaches must be integrated into the theoretical justification for the development of language tests, in order to satisfy a number of conditions for their responsible design.

The underlying action-oriented and task-based approach of the CEFR and its implementation in language testing and assessment at university

Abstract The underlying methodological approach of the CEFR is defined as being “action-oriented” and task-based (Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press: 9), although it explicitly leaves room for a variety of approaches, since the CEFR is a framework document. The action-oriented approach is, however, not coherently developed throughout the publication as it does not explain its application in assessment (Chapter 9). Chapter 9 presents a general introduction to the principles of language assessment but does not explain how the assessment can be contextualised so that the test-taker acts as a “social agent”. Although related documents on testing and assessment such as the Manual for Relating Language Examinations to the CEFR (Council of Europe 2009) and the Manual for Language Test Development and Examining (Council of Europe 2011) provide tools on assessment practices, they do not explain how to conceptualise an action-oriented task-based assessment. The CEFR Companion Volume (Council of Europe 2020) also stresses the underlying action-oriented approach of the CEFR in teaching and learning, but as it focuses explicitly on teaching and learning, it too does not cover the area of testing and assessment. For university language teaching, learning and assessment, it is necessary to develop a coherent, holistic methodology that applies the action-oriented and task-based approach to language teaching, learning and assessment, and applies a constructive alignment, which has a particular focus on the learner as a social agent. Initially, this requires the development of course content and the related assessment tasks to be based on a thorough needs analysis for a specific language teaching programme. In this way, university language centres can rethink their approach to teaching and assessment and implement changes both in their teaching and in their assessment and testing in order to offer a more coherent programme to their learners, which makes better use of the CEFR Framework. This paper summarises the principles of task-based language testing and explains in which contexts this approach might be particularly fruitful. It also presents the author’s experience gained in implementing the approach at institutional and national levels, and the challenges involved in managing this change.

The role of informal digital learning of English and a high-stakes English test on perceptions of English as an international language

This study investigated the relationship among informal digital learning of English (IDLE) practice, a high-stakes English test, English productive skills, and perceptions of English as an international language (EIL). Eighty-nine English as a foreign language (EFL) undergraduate students across three South Korean cities participated in the study. The participants submitted their scores in the Test of English for International Communication (TOEIC, South Korea’s most popular high-stakes English test), took English speaking and productive vocabulary-level tests, and completed surveys that measured the frequency of their IDLE activities and EIL perceptions. Results of the hierarchical multiple regression analyses showed that IDLE practice and TOEIC scores were significant predictors of EIL perceptions. The structural equation modelling analysis further revealed that IDLE practice partially mediated the relationship between TOEIC scores and EIL perceptions. This indicates that students with higher TOEIC scores tended to practise IDLE activities more frequently, which enabled them to experience diverse accents and users of English and, in turn, help increase their EIL perceptions. It also suggests that more proficient EFL speakers may not necessarily become competent EIL users. These findings are discussed with consideration of South Korea’s socio-educational contexts, followed by pedagogical implications for English language educators and test developers.