Field Of Language Testing Research Articles

A Human Face to Technology? – Digital Exams are Coming

Technological development has reached a point where computerised foreign language testing is possible as well as desirable. It was not caused by the pandemic, which only speeded up development as all the elements of digital, then online testing were already in place. Technological developments are doubly interesting because they also appear to militate against communicative language testing, the prevailing orthodoxy in the field for the past forty years. At the same time, the field of language testing also faces other challenges, such as a language tests becoming less of a social act, stakeholders functioning in changing roles and last but not least, measurement itself is facing a challenge from business logic.

Incorporating Communicative Competence in Assessment and English Language Teaching in Multilingual Settings

There has been a high level of agreement among scholars that communicative competence should be integrated within language learning and assessments. The study attempts to unravel the issues of how communicative competence can be assessed and measured in multilingual environments and how communicative language testing can be promoted. Using the content analysis approach as the qualitative method, it begins with the historical review of communicative competence that started at the beginning of 1970s to the most current concept involving intercultural communicative competence. Then, some practical models of communicative competence that can be used to propose a measurement of communicative competence are presented. Later, this article argues that there is an upsurge need to shift the paradigm of language testing and language assessment towards communicative competence. Moreover, the nature of language testing should not only concern linguistic or knowledge competence but also recognize the different varieties of English. This study implies that, in the field of language testing, language test designers should encompass the concept of communicative competence in the test construct to include real-life language use, and by extension, to increase test validity. As for teachers, a reform integrating communicative competence in classroom language assessment has become essential within the scope of language teaching.

A Comparative Study of Test Takers’ Performance on Computer-Based Test and Paper-Based Test Across Different CEFR Levels

Computer-based test (CBT) and paper-based test (PBT) are two test modes to the test takers that have been widely adopted in the field of language testing or assessment over the last few decades. Due to the rapid development of science and technology, it is a trend for universities and educational institutions striving rather hard to deliver the test on a computer. Therefore, research on the comparison between these two test modes has attracted much attention to investigate whether the PBT could be completely replaced. At the same time, task difficulty is always a key element to reflect test takers’ performances. Numerous studies have laid a solid foundation and guidance about the comparative study of test takers’ performance on CBT and PBT, but there still remains a scarcity from the perspective of task difficulties with different Common European Framework of Reference for Languages (CEFR) task levels in particular.
 This study, therefore, compared the test takers’ performance on both CBT and PBT across tasks with different CEFR levels. A total of 289 principal recommended high school test takers from Macau took the pilot Test of Academic English (TAE) at a local university. The results indicated that there was a difference between test takers’ performance on different test modes across different CEFR levels, but only CEFR A2 level showed a statistically difference between CBT and PBT. And since science and technology are continuously developing, it is essential for the university to consider switching the test mode from PBT to CBT.

The Construct of Multilingualism in Language Testing

ABSTRACTThis special issue brings together research in multilingualism within applied linguistics to delineate conceptualizations of multilingualism as constructs amenable for classroom and standardized testing systems operations. Specifically, for language testing and assessment, there have been multiple calls for the field to embrace multilingual approaches not only to reflect the full (linguistic) humanity of multilingual peoples but also to contest decades to centuries of marginalization and discrimination against multilingual practices outside monolingual standards. By bringing together and expanding orientations toward multilingualism in content and language assessments, this special issue is positioned to aid with significant shifts in the field of language testing. Separately and together, these papers work to engage with the construct of multilingualism in assessment in ways that point to the complexities and issues that arise. Embracing complexities and the realities they represent for test users serves to create methods and approaches in the field, which is critical for scholarship to keep pace with changes in society and to align with the knowledge base in applied linguistics.

Moving the field of vocabulary assessment forward: The need for more rigorous test development and validation

AbstractRecently, a large number of vocabulary tests have been made available to language teachers, testers, and researchers. Unfortunately, most of them have been launched with inadequate validation evidence. The field of language testing has become increasingly more rigorous in the area of test validation, but developers of vocabulary tests have generally not given validation sufficient attention in the past. This paper argues for more rigorous and systematic procedures for test development, starting from a more precise specification of the test's purpose, intended testees and educational context, the particular aspects of vocabulary knowledge which are being measured, and the way in which the test scores should be interpreted. It also calls for greater assessment literacy among vocabulary test developers, and greater support for the end users of the tests, for instance, with the provision of detailed users' manuals. Overall, the authors present what they feel are the minimum requirements for vocabulary test development and validation. They argue that the field should self-police itself more rigorously to ensure that these requirements are met or exceeded, and made explicit for those using vocabulary tests.

Test-taker perception of and test performance on computer-delivered speaking tests: the mediational role of test-taking motivation

BackgroundResearch on the test-taker perception of assessments has been conducted under the assumption that negative test-taker perception may influence test performance by decreasing test-taking motivation. This assumption, however, has not been verified in the field of language testing. Building on expectancy-value theory, this study explored the relationships between test-taker perception, test-taking motivation, and test performance in the context of a computer-delivered speaking test.MethodsSixty-four Japanese university students took the TOEIC Speaking test and completed a questionnaire that included statements about their test perception, test-taking motivation, and self-perceived test performance. Five students participated in follow-up interviews.ResultsQuestionnaire results showed that students regarded the TOEIC Speaking test positively in terms of test validity but showed reservations about computer delivery, and that they felt sufficiently motivated during the test. Interview results revealed various reasons for their reservations about computer delivery and factors that distracted them during the test. According to correlation analysis, the effects of test-taker perception and test-taking motivation seemed to be minimal on test performance, and participants’ perception of computer delivery was directly related to test-taking effort, but their perception of test validity seemed to be related to test-taking effort only indirectly through the mediation of perceived test importance.ConclusionOur findings not only provide empirical evidence for the relationship between test-taker perception and test performance but also highlight the importance of considering test-taker reactions in developing tests.

A quantitative analysis of TOEFL iBT using an interpretive model of test validity

BackgroundValidity is a notable concept in language testing which has concerned many researchers and scholars in the field of language testing due to its importance in decision making process. Tests’ results always introduce consequences to test takers’ lives which emphasizes the need to ensure their validity. Detecting and delineating the potential sources that may threaten the validity of standardized tests of English proficiency is therefore of great importance.MethodsThis study was a quantitative investigation of test validity using an interpretive model. The main purpose of this study was to quantitatively assess and determine the relationship between a series of inferences in an interpretive argument-based framework that can potentially threaten the validity of the test.ResultsTo this end samples of TOEFL iBT were analyzed and the obtained results determined its validity expressed in terms of the chain of inferences in the interpretive model.ConclusionThe findings indicated that there is a direct relationship between the performance on the TOEFL iBT and the target domain of English language use, observed test scores are reflective of intended academic language abilities, observed test scores can be generalized to similar language tasks in different occasions and test forms, and expected test scores can be accounted for by underlying language abilities in an academic environment.

Assessment Orientations of State Primary EFL Teachers in Two Mediterranean Countries

Many researchers have highlighted the central role that assessment plays in second language (L2) classrooms and have expressed the need for research into classroom-based language assessment (CBLA), an area that is gradually coming into its own in the field of language testing and assessment (e.g., Hasselgreen, 2008; Leung, 2014; Hill & McNamara, 2012; Turner, 2012). Motivated by the prominence of CBLA in recent discussions, the present study set out to investigate the CBLA practices, knowledge and skills of Greek and Cypriot primary school EFL teachers. The data was collected through teacher interviews and classroom-based tests. The results showed that teachers employ a summative orientation towards evaluating their students’ performance and seem to have unclear ideas about the purposes and implementation of formative assessment, mainly due to lack of professional training in language assessment. The paper concludes with suggestions as to how EFL teachers’ CBLA literacy can be enhanced.

Appraising Pre-service EFL Teachers' Assessment in Language Testing Course Using Revised Bloom's Taxonomy

The teachers need to be conceived as a “change agent” and not as a mere transmitter of knowledge and culture. In developing countries like Iran, one of the most significant concerns in the field of teachers’ education is efficiency of pre-service programs. To this aim, the current descriptive-evaluative study intended to describe the state of pre-service teachers' assessment in the field of language testing by (a) examining the exam questions to find out whether they are aligned with curriculum objectives and syllabus (content validity, (b) exploring whether they take care of higher order cognitive processes and (c) finding what combinations of cognitive process levels and knowledge types in Revised Bloom's Taxonomy are prevalent in the questions. The results exhibited an unbalanced coverage of content in exams. Also the questions were found to be inadequate in terms of measuring complex cognitive skills (Analyze and Evaluate); Remember and Understand domains take up 91.6 % of all questions and no item was found for Create. Three combinations of cognitive process level and knowledge type was dominant in the data set: (1) Remember Factual Knowledge, (2) Understand Conceptual Knowledge, and (3) Apply Procedural Knowledge. These associations confirm the Anderson and Krathwohl's (2001) proposition.

An Assessment of IELTS Speaking Test

The present study focusses on assessing the speaking module of IELTS. The assessment discusses both positive aspects and weaknesses in IELTS speaking module. The researcher has also suggested some possible measures for the improvement in IELTS speaking module and increasing its validity and reliability. The researcher has analysed and assessed IELTS speaking module in the light of both theoretical and practical perspectives presented by experienced researchers in the field of language testing and evaluation. The researcher’s major concern in the assessment of IELTS speaking module is to do utmost effort to avoid the element of subjectivity and to present some logical and practical suggestions for improving IELTS speaking module. DOI: http://dx.doi.org/10.11591/ijere.v3i3.6047

A comparison of video- and audio-mediated listening tests with many-facet Rasch modeling and differential distractor functioning

The rise in the affordability of quality video production equipment has resulted in increased interest in video-mediated tests of foreign language listening comprehension. Although research on such tests has continued fairly steadily since the early 1980s, studies have relied on analyses of raw scores, despite the growing prevalence of item response theory in the field of language testing as a whole. The present study addresses this gap by comparing data from identical, counter-balanced multiple-choice listening test forms employing three text types (monologue, conversation, and lecture) administered to 164 university students of English in Japan. Data were analyzed via many-facet Rasch modeling to compare the difficulties of the audio and video formats; to investigate interactions between format and text-type, and format and proficiency level; and to identify specific items biased toward one or the other format. Finally, items displaying such differences were subjected to differential distractor functioning analyses. No interactions between format and text-type, or format and proficiency level, were observed. Four items were discovered displaying format-based differences in difficulty, two of which were found to correspond to possible acting anomalies in the videos. The author argues for further work focusing on item-level interactions with test format.

Strategies for testing statistical and practical significance in detecting DIF with logistic regression models

This study examines three controversial aspects in differential item functioning (DIF) detection by logistic regression (LR) models: first, the relative effectiveness of different analytical strategies for detecting DIF; second, the suitability of the Wald statistic for determining the statistical significance of the parameters of interest; and third, the degree of equivalence between the main DIF classification systems. Different strategies to tests–LR models, and different DIF classification systems, were compared using data obtained from the University of Tehran English Proficiency Test (UTEPT). The data obtained from 400 test takers who hold a master’s degree in science and engineering or humanities were investigated for DIF. The data were also analyzed with the Mantel–Haenszel procedure in order to have an appropriate comparison for detecting uniform DIF. The article provides some guidelines for DIF detection using LR models that can be useful for practitioners in the field of language testing and assessment.

Use Of Formative Assessment, Self- And Peer-Assessment In The Classrooms: Some Insights From Recent Language Testing And Assessment (LTA) Research

A considerable number of studies on formative teacher assessment and feedback, learner self- and peer-assessment have been carried out in the field of Language Testing and Assessment (LTA) research over the last two decades. These studies investigated the above mentioned concepts from different perspectives (impact of assessment on learning, attitudes towards assessment, comparison between teacher and learner assessment practices, types and quality of formative teacher feedback), in a number of different contexts (English as second language classrooms, mainstream classrooms, immersion classrooms, i.e. for mainstream classrooms, teaching and learning is done through the medium of a second or additional language) and in a number of different ways (experimental studies, observational studies, surveys, research review studies). This paper systematically reviews most recent research on formative teacher assessment and feedback, learner self- and peer-assessment, and reveals gaps which have not yet been addressed by research.

Guest Editorial to the special issue on language assessment literacy

In the editorial to the first issue of Language Testing Journal, which appeared in June 1984, the editors, Arthur Hughes and Don Porter, state that “This new journal has come into being as a forum devoted exclusively to the issues that concern those involved in, or simply interested in, the assessment of language ability in one form or another.” In addition to encouraging a wide writerand readership, they also assure that “the field covered will be a broad one”, encompassing diverse language teaching learning and testing situations (1984, p. i). This approach is indeed reflected in the range of topics covered in this very first issue: from validity and reliability concerns in language tests, to research on the test taker’s perspective and to testing English as a Second Language versus Mother Tongue and English as a Foreign Language. The contributions to the journal upon its inception mapped out at that point in time the basic areas of research in the language testing terrain, setting the stage for professional developments in the field that have been shaped to a great extent by the journal’s publications. It is thus apt that this current issue, which marks the 30th anniversary of the journal, attempts to examine via the lens of language assessment literacy the knowledge base that underlies and typifies language testing and assessment as a discipline of study. Discussion on the nature of assessment literacy, a concept introduced with reference to the knowledge assessors need to possess (Stiggins, 1991), reflects an ongoing debate on the nature of the professional knowledge in the field of language testing. The debate has emerged more forcibly as of late due to an increased demand for and use of assessment data by a more diverse group of stakeholders than before, many of whom are novices in the field and have limited or partial knowledge in language testing and assessment and its concerns. The discussion is conducted on inter-related themes: the first focuses on assessment literacy in the language domain, that is, whether there is an agreed-upon theoretical, practical, and experiential knowledge base required for fulfilling assessment and testing functions in language-related situations; the second relates to whether language testing professionals should act as gatekeepers and allow in only those who are considered assessment literate, or whether the profession should reach out to impart differential language assessment knowledge according to needs, to wide circles of applied linguists, teachers, parents, bureaucrats, and politicians. Another issue that is currently 480126 LTJ30310.1177/0265532213480126Language TestingEditorial 2013

学内開発プレイスメントテスト得点解釈と使用の妥当性の評価について Evaluating Validity for In-House Placement Test Score Interpretations and Uses

Validity theory has evolved dramatically in the past few decades. The most prominent theory in recent years is an argument-based validity framework, proposed by Kane (1992, 2004, 2006). To evaluate test score interpretations and uses based on Kane’s framework, test developers first need to provide interpretive arguments and then validity arguments by proving sound warrants for the following four inferences: (a) scoring from observation to an observed score, (b) generalization from the observed score to the universe score, (c) extrapolation from the universe score to a target score, and (d) decision from the target score to use. In the field of language testing, a number of studies have been conducted to investigate the validity of test score interpretations and uses, especially for the ones considered to be high-stakes such as the TOEFL (Chapelle, 2008; Chapelle, Enright, & Jamieson, 2010). However, not many studies have been conducted to validate in-house placement test score interpretations and uses, and no study has evaluated the validity of such low-stakes tests using Kane’s validity framework. Regardless of whether the tests are high or low stakes, test developers need to be responsible for validating their test score interpretations and uses in order to attest to the validity. This study uses Kane’s (2006) argument-based validity framework to evaluate the validity of in-house placement test score interpretations and uses. The research questions are as follows: (a) to what extent do examinees get placement items correct and high-scoring examinees get more placement items correct; (b) to what extent are placement items consistently sampled from a domain sufficient in number so as to reduce measurement error; (c) to what extent do the difficulty of placement items match the objectives of a reading course; and (d) to what extent do placement decisions made to place examinees in their proper level of the course have an impact on washback in the course? An in-house placement test made up of 40-item grammar, 40-item vocabulary, and 10-item reading sections was developed and administered to 428 first-year private-university students in April 2010. The item format adopted was all multiple-choice so the answer sheets could be easily scored with a reader. Based on their test scores, about 60 high-scoring students and 50 low-scoring students were placed into one of two advanced or one of two basic reading classes. The remaining students were placed into one of several intermediate classes. A 55-item grammar achievement test was administered twice (once as a pretest and then again as a posttest) to the two basic and two intermediate classes. In addition, a 51-item class evaluation survey was administered to investigate students’ participation in the reading classes and to gauge students’ satisfaction with the classes and study support. Warrants for a validity argument of score inference were based on the results of the item analysis. A warrant for a validity argument of generalization inference was based on the composite generalizability coefficient of .92. A warrant for a validity argument of extrapolation inference was based on FACETS analysis, showing that difficulty estimates of learning levels were in an expected difficulty order. A warrant for validity arguments of decision inference was based on the basic-level students’ score gain on an achievement test and their positive reactions to a class evaluation survey. All the validity arguments presented in this study support the validity of the placement test score interpretations and uses. However, to further improve the validity of the test score interpretations and uses, it is necessary to investigate washback effects of the placement test in the reading classes and to revise the test to make grammar, vocabulary, and reading sections with 30 items each.

C-test vs. Multiple-choice Cloze Test as Tests of Reading Comprehension in Iranian EFL Context: Learners' Perspective

Cloze tests have been widely used for measuring reading comprehension since their introducing to the testing world by Taylor in 1953. But in 1982, Klein-Braley criticized cloze procedure mostly for their deletion and scoring problems. They introduced their newly developed testing procedure, C-test, which was an evolved form of cloze tests without their deficiencies (Klein-Braley, 1982 cited in Baghaei, 2008). After that, the effectiveness of C-test and cloze test became the main interest of the scientists in the field of language testing. The present study aims to compare the results of multiple-choice cloze test with those of C-test as measures of reading comprehension. To this end, one traditional C-test and one fixed ratio (n=7) multiple-choice cloze test were prepared from reading passages with similar readability level. The subjects of the study were 27 female EFL advanced learners. The results of the study revealed that multiple-choice cloze is a better measure of reading comprehension. Through a retrospective study which was done at the end of the tests, the students' impressions and opinions about tests and their own performance were recorded and taken into consideration. The implications of the findings and suggestions for more studies are discussed within a foreign language testing context.

A Study of the Effect of Entrance Examination of the Universities (EEOU) on the EFL Teaching Methodology in Iran

Researches and educators’ recognition of positive and negative effects of tests on teaching and learning activities goes back at least four decades. Although educators frequently claim that they do not want to teach to a test, the reality is that every educator wants his/her students to be successful. Decision makers, teachers, and students equate this success in large part with high test scores, resulting in classroom instruction that is reflective of test practices and/or expectations. The effect that testing has on teaching and instruction is a phenomenon referred to as washback. However, this phenomenon has been examined empirically by only a few studies in the field of language testing. This qualitative study aimed to (i) investigate the existence and nature of the washback effect, and (ii) examine the role of EEOU context in promoting beneficial washback. The findings confirmed the existence of washback phenomenon in general and negative washback effects of EEOU context on language teaching and learning in particular.

A Synthesis of 15 Years of Research on DIF in Language Testing: Methodological Advances, Challenges, and Recommendations

This article reviews research on differential item functioning (DIF) in language testing conducted primarily between 1990 and 2005 with an eye toward providing methodological guidelines for developing, conducting, and disseminating research in this area. The article contains a synthesis of 27 studies with respect to five essential sets of characteristics: (a) features of the tests utilized, (b) features of the learner groups employed, (c) features of the DIF detection methods applied, (d) features of the reporting of DIF effects, and (e) features of the explanations for and consequences drawn from DIF results. Methodological recommendations for improving the conceptualization and the reporting of DIF research in the field of language testing is provided with an eye toward developing statistical meta-analyses that aid practitioners in interpreting the relevance of DIF results more precisely.

14. DISCOURSE APPROACHES TO WRITING ASSESSMENT

At present, mastery of English for educational and professional purposes contributes significantly to expansion of the role of English as a language of world communication. In this context, the teaching—and, consequently, the assessment—of EFL/ESL writing is receiving more attention than ever before. This chapter argues that, although this renewed interest in the evaluation of writing has resulted in advances in the field of language testing, it has by and large failed to incorporate insights gained from recent developments in text analysis. There is a considerable gap between current practices in writing assessment and criteria suggested by advances in knowledge of discourse structure. We illustrate this gap by contrasting current practices in the scoring of two major EFL/ESL writing tests with knowledge of text characteristics generated from recent developments in text analysis. The review concludes by making the case for bridging the gap and by proposing a model of writing assessment that incorporates both linguistic and discoursal features of texts.

MULTIPLE-CHOICE AND CLOZE TESTS: WHAT DO THEY REALLY MEASURE?

<p>The objective of this article is to show the results of a MA research carried out from july 1995 to august 1996. The aim was to find out what mental process(es) a testee goes through, while taking a multiple-choice and cloze test, in order to arrive at a correct answer. Cohen (1984:70-1) states that “there is a small but growing literature on how students go through the process of taking language tests.” Very little has been done in the field of language testing and even less has been done in second-language testing. The conclusions reached in this research is just a tiny light at the end of a long tunnel which can bring insights for further research in this area.</p>