Computer-based Language Tests Research Articles

Applying Computer-Based Language Test to Young Children

Introduction: This study aimed at exploring the feasibility of applying a computer-based language test to young children aged 2–4 years. Methods: Thirty-two Cantonese-speaking children, aged 2–4 years, were recruited from local kindergartens. All participants underwent an assessment using both the computer-based and paper-pencil versions of the Macau Cantonese Language Screening Scale for Preschool Children, following a crossover study design. A short break of 15–30 min was provided between the two assessments. The data were analysed at three levels: the overall test, subcategory, and individual item levels. At the overall test and subcategory levels, data were analysed using the paired samples t-test and ICC. At the item level, the percentage of agreement and Cohen’s kappa value were selected to assess the agreement of the two test formats. Results: Excellent agreement was found for the overall test level, and good agreement was observed for four of the five subcategories. At the individual item level, 28 of the 35 items showed more than 80% agreement, and 16 items showed substantial to almost perfect agreement. Conclusion: These results suggest that the two test formats give similar total scores and subcategory scores for children aged 2–4. For children older than 2 years and 6 months, the agreement for matching items is as high as 83.68% (1,318/1,575). The computer-based test is thus highly recommended for this group of children. For children younger than 2 years and 6 months, a modified computer-based test is suggested to accommodate their needs.

STUDENTS’ PERCEPTIONS OF COMPUTER-BASED LANGUAGE TEST (CBLT) AND VALIDATION OF CBLT-LISTENING QUESTIONNAIRE USING EXPLORATORY FACTOR ANALYSIS

Background and Purpose: One of the difficulties that emerged during online learning was the teaching and learning of listening skills which prompted the development of an online platform, Computer-based Language Test (CBLT), to assist educators conduct listening practices and assessment with students without experiencing geographical limitations. As this is a novel platform, this paper intends to examine students’ perceptions of listening skills and online learning after utilising this platform using the developed CBLT-Listening Questionnaire.
 
 Methodology: This cross-sectional quantitative study employed questionnaires to collect data from diploma students. Two-stage sampling was utilised whereby the first stage used stratified random sampling in selecting classes from twenty-three (23) branches of the university. Simple random sampling was practised in the second stage which yielded 410 responses. The data were subjected to descriptive and exploratory factor analyses (EFA) in SPSS.
 
 Findings: Descriptive analysis revealed that the respondents find listening a difficult skill to acquire yet view CBLT an excellent initiative for online distance learning. EFA revealed that the items in the Listening construct were significant and to be retained. However, the EFA results reported that one item from the Online Distance Learning construct needs to be deleted from the questionnaire.
 
 Contributions: The validated CBLT-S Listening Questionnaire serves as an instrument to investigate the impact of a novel online platform (CBLT) on students' perceptions on listening and online learning. The results from this study evidenced that online platforms that incorporate practices and assessment are viewed positively by students for pedagogy in language teaching.
 
 Keywords: Computer-based language test, exploratory factor analysis, listening skill, online listening test, CBLT-listening questionnaire.
 
 Cite as: Khairuddin, Z., Anuar, N., Azhari, M. A., Serip Mohamad, N. H., Mohd Daud, K., Ahmad, A. R., & Abd Rashid, M. H. (2023). Students’ perceptions of Computer-Based Language Test (CBLT) and validation of CBLT-listening questionnaire using exploratory factor analysis. Journal of Nusantara Studies, 8(2), 222-241. http://dx.doi.org/10.24200/jonus.vol8iss2pp222-241

Readiness towards Using Computer-Based Language Test (CBLT) among Malaysian Tertiary ESL Learners

The vast majority of ‘Gen Z’ are in the positive light about the future of technology in education. Due to the Covid-19 pandemic, there has been a shift in teaching, learning and assessing methods. To keep abreast with current technology, computer-based language test (CBLT) was used as one of the methods to assess students’ language proficiency during the shift. For this study, the CBLT was adopted and conducted among students who were taking an English proficiency course in a public university mainly located in Selangor, Malaysia. They were required to sit for the test as part of the course fulfilment. Thus, the purpose of the study is to investigate whether there are significant differences in the CBLT 1, CBLT 2 and CBLT 3 scores among Business Management (BM), Social Sciences and Humanities (SSH), and Science and Technology (ST) students, who were from different branches around Malaysia. The study employed a quantitative research design in which a total of 16079 students’ scores from three (3) different tests were recorded and analysed. The results of this analysis will provide insights on the conclusion that students from different programmes were able to adapt to the use of the CBLT in assessing their listening skills.

Modeling Rapid Guessing Behaviors in Computer-Based Testlet Items.

In traditional test models, test items are independent, and test-takers slowly and thoughtfully respond to each test item. However, some test items have a common stimulus (dependent test items in a testlet), and sometimes test-takers lack motivation, knowledge, or time (speededness), so they perform rapid guessing (RG). Ignoring the dependence in responses to testlet items can negatively bias standard errors of measurement, and ignoring RG by fitting a simpler item response theory (IRT) model can bias the results. Because computer-based testing captures response times on testlet responses, we propose a mixture testlet IRT model with item responses and response time to model RG behaviors in computer-based testlet items. Two simulation studies with Markov chain Monte Carlo estimation using the JAGS program showed (a) good recovery of the item and person parameters in this new model and (b) the harmful consequences of ignoring RG (biased parameter estimates: overestimated item difficulties, underestimated time intensities, underestimated respondent latent speed parameters, and overestimated precision of respondent latent estimates). The application of IRT models with and without RG to data from a computer-based language test showed parameter differences resembling those in the simulations.

Students’ Perceptions on the Implementation and Challenges of Computer-Based Language Test for Diploma Course Assessment

Online assessment has become prominent in the education system. However, the implementation of CBLT has posed different challenges faced by students in completing the assessment. Therefore, the present study aimed to find out students’ perceptions on the implementation of CBLT and its challenges, to determine the significant differences between the implementation of CBLT and faculty clusters. This study utilised a quantitative research approach. The questionnaire employed in this study was adopted and adapted from previous studies with 3159 students from a population of 4600 who enrolled for ELC121. The data was analysed using Statistical Package for the Social Sciences (SPSS v27) and were reported for mean, standard deviation, and ANOVA. The findings show that the students were positive towards the implementation of CBLT and there were significant differences of perception among faculty clusters. Thus, the future teaching and learning process should integrate the use of technology.

Computer-based Language Testing: Potential and Pitfalls

Computer-based Language Testing: Potential and Pitfalls

Computer-Based English Language Testing in China: Present and Future

ABSTRACTIn this special issue on high-stakes English language testing in China, the two articles on computer-based testing (Jin & Yan; He & Min) highlight a number of consistent, ongoing challenges and concerns in the development and implementation of the nationwide IB-CET (Internet Based College English Test) and institutional computer-adaptive English tests, respectively: conceptualizing the construct of computer-based language testing, ensuring fairness for test takers with differing levels of computer literacy, and achieving comparability between tests or tasks of different delivery modes. In this article, we provide an overview of the research studies on computer-based English language testing conducted by Chinese scholars and published in major Chinese academic journals in recent decades, aiming to identify the research topics, gaps, and agendas that could have implications beyond Chinese contexts in promoting better use of computer technologies in and for English language testing.

Computer Literacy and the Construct Validity of a High-Stakes Computer-Based Writing Assessment

ABSTRACTOne major threat to validity in high-stakes testing is construct-irrelevant variance. In this study we explored whether the transition from a paper-and-pencil to a computer-based test mode in a high-stakes test in China, the College English Test, has brought about variance irrelevant to the construct being assessed in this test. Analyses of the test scores, text complexity, language errors, and writing processes revealed that computer-based writing elicited cognitive processes among test takers that were similar to those of traditional paper-based writing; moreover, a high level of computer familiarity had a facilitative effect on test-takers’ performances. In considering these results we argue that, in the 21st century, the construct of a computer-based language test needs to be reconceptualized by drawing on Chalhoub-Deville’s view of local context-bound constructs: computer literacy should be viewed as an important contextual facet interacting with the construct measured in a computer-based language assessment.

Development and Validation of a Computer Adaptive EFL Test

ABSTRACTThe first aim of this study was to develop a computer adaptive EFL test (CALT) that assesses test takers’ listening and reading proficiency in English with dichotomous items and polytomous testlets. We reported in detail on the development of the CALT, including item banking, determination of suitable item response theory (IRT) models for item calibration, item selection procedure, ability estimation method and stopping rule. The CALT in this study, as part of the graduation requirement for non-English majors at a major Chinese university, is a high-stakes test. We think that validation of such a high-stakes test is important and necessary, so the second aim was to investigate the validity issues of the CALT by examining the equivalence of the factor structure of CALT and a computer-based language test (CBLT) within Bachman and Palmer’s assessment use argument (AUA) framework. It is our hope that the effort made in this study to investigate the comparability of CALT and CBLT to ensure that test score interpretations of CALT are meaningful and impartial will prove worthwhile.

Looking towards the Future of Language Assessment: Usability of Tablet PCs in Language Testing

This research addresses the change in how the Spanish University Entrance Examination can be delivered in the future. There is a wide acknowledgement that computer tests are very demanding for the delivering institutions which makes computer language testing difficult to implement. However, the use of tablet PCs can facilitate the delivery at even lower cost than the regular computer based language testing. 183 students in their last year of high school took a computer based language test which included reading, writing, listening and speaking. The research aspects were 1) whether they feel at ease with the tablet PC exam; 2) if they felt that visual aspects were accessible and 3) whether the interface organization was clear. The paper first has a brief description of the OPENPAU® platform, after it addresses the field study based on questionnaires and observations with students during the test delivery, finally the results indicate that this means of language test delivery could be adequate not only for the Spanish University Entrance Examination but for most standardized tests.

Designing a Voip Based Language Test

Assessing speaking is one of the most difficult tasks in computer based language testing. Many countries all over the world face the need to implement standardized language tests where speaking tasks are commonly included. However, a number of problems make them rather impractical such as the costs, the personnel involved, the length of time for interviews and many other factors. Additionally, reliability as compared to face-to-face tests is continually challenged by issues such as comfort with the interface, navigability and, among others and more important, the lack of visual interaction and the high anxiety created by interacting with a interlocutor with more than a limited interactional activity. This paper addresses a new approach to language testing by the use of VOIP devices. It also addresses its benefits and the way to implement it within the Spanish framework of nationally organized standardized tests.

An Overview of The Needs of Technology in Language Testing in Spain

Over the few years, computer based language testing has become prevailing worldwide. The number of institutions the use computers as the main means of delivery has increased dramatically. Many students face each day tests for well-known high-stakes decisions which imply the knowledge and ability to use technology to provide evidence of language mastery. However, this situation is not the same everywhere. While countries like the United States or the United Kingdom are spearheads of this tendency, others like Spain, lag way behind. This paper addresses the needs of use of computer based language testing in the new Spanish educational system. To do so, this paper introduces the current situation in Spain, then proceeds to do a SWOT analysis emphasizing the lacks and opportunities for language based assessments and concludes by stating the positive issues involving them.

A Proposal on the Validation Model of Equivalence between PBLT and CBLT

The validity of the computer-based language test is possibly affected by three factors: computer familiarity, audio-visual cognitive competence, and other discrepancies in construct. Therefore, validating the equivalence between the paper-and-pencil language test and the computer-based language test is a key step in the procedure of designing a computer-based language test. By taking the test on Essentials of English-Speaking Countries as the case study, this paper elucidates the three-step model of validating the equivalence of the two types of test: investigating computer familiarity, assessing the impact of audio-visual cognitive competence, and examining other discrepancies in construct. The model proposed by this paper can offer some methodological insights on the way to establishing the validation model of the equivalence between the paper-and-pencil language test and the computer-based language test.

Communicative Language Testing: Implications for Computer Based Language Testing in French for Specific Purposes

Current evolutions of language testing have led to integrating computers in FSP assessments both in oral and written communicative tasks. This paper deals with two main issues: learners’ expectations about the types of questions in FSP computer based assessments and the relation with their own experience. This paper describes the experience of 23 freshmen French students enrolled in the Universitat Politecnica de Valencia who took a computer based version of a FSP test. Their attitudes were observed and annotated. The learners expressed their difficulties in taking a FSP computer delivered test. The paper suggests that the dramatic change between general French and French for Specific Purposes may imply significant differences in the students to the new computer context due to the changes in vocabulary, register and language use as well as computer ergonomics. The paper concludes: firstly, that basic notions of certain specialized forms and vocabulary should be introduced in high school; secondly, that FSP should tend to be more communicative and that FSP teaching should be more reflective and communicative than it may be in colleges at the moment; thirdly, that if computer based tests may be a valuable tool in FSP testing (Garcia Laborda et al, 2010), item choice and prompts should be carefully considered. The analysis hereby presented is contextualized in computer based language testing and is a relevant part of the OPENPAU (MINECO FFI2011-22442) project.

Item transformation for computer assisted language testing: The adaptation of the Spanish University entrance examination

Since the Spanish Educational system is changing and promoting the use of online tests, it was necessary to study the transformation of test items in the “Spanish University Entrance Examination” (IB P.A.U.) to diminish the effect of test delivery changes (through its computerization) in order to affect the least the current model. The purpose of this study was to describe and suggest the properties of a new test item taxonomy for the Spanish University Entrance Examination. After a convenient study and piloting by using previous research in computer-based language testing, the researchers created a taxonomy of test items for the I.B. P.A.U.

Training senior teachers in compulsory computer based language tests

The IBT TOEFL has become the principal example of online high stakes language testing since 2005. Most instructors who do the preparation for IBT TOEFL face two main realities: first, students are eager and highly motivated to take the test because of the prospective implications; and, second, specific studies would be necessary to see if instructors may be familiarized with the use of computers in teaching and/or language testing for other high stakes tests. Now, many standardized tests are going online as well such as IELTS, BULATS and others (García Laborda, 2007). National Boards of Education are also considering this possibility for the national university entrance examination (PAU) (García Laborda & Magal Royo, 2007; García Laborda, 2006). However, in this case, teachers may not be willing or well prepared to face the challenge due to factors such as ICT familiarity. To date, many papers have explored the role of teacher training in their adaptation to the inclusion or implementation of ICT in the foreign language classroom (Zepp, 2005; Tan et al., 2003; Rehbein et al., 2003). However, it is necessary to consider the differences between the use of ICT for language testing (as opposed to general teaching) and the profile of senior high school foreign language teachers who may not be as flexible as their younger colleagues. This paper describes the trainees’ final attitudes of a six-month course held in Valencia (Spain). 26 teachers from private schools who teach many of the students who take the test every year participated in an in-service teacher training course in Valencia. Instructional methods principally included training in ICT strategy and use through face-to-face instruction and also autonomous learning, group work, synchronous and asynchronous communication, and computer based test design. The teachers’ responses to long discussions in a control session and to a questionnaire, along with the observations of a 20-hour course and the results of the familiarization with different tools show that for this type of training it is not only necessary to familiarize teachers with the testing tool but also with the process of creating online tests and with the testing process itself (Chapelle & Douglas, 2006; Stoynoff & Chapelle, 2005). Results from the platform use and the final questionnaire indicated that the majority of trainees reacted positively to the training and were eager to let the research team work with their students. However, they still felt they would not be able to use ICT for testing in their classes.

Interface architecture for testing in foreign language education

The implications of new learning environments have been far-reaching and pervasive (Plass, 1998), at least in the field of interface design both in traditional computer and mobile devices (Fallahkhair, Pemberton, & Griffiths, 2007). Given the current status of efficient models, educators need the unproven unification of interfaces and working areas, which may facilitate the students’ integration within the working areas of computer based language testing (CALL) (Cushion, 2006). Based on previous works on interface design by Fulcher (2003), Chapelle & Douglas (2006) and Garcia Laborda (2007) and interface validation by Garcia Laborda & Magal Royo (2007) and current developments (IATEFL or TOEFL), in this paper we concentrate our efforts on describing the design and implementation as well the design principles of the working area of the computer assisted language testing (CALT) platform called PAUER that has been sponsored by the Spanish Ministry of Education between 2007-2010. As most areas of practice in higher education shift online, the work of learners and teachers increasingly takes place within the domain of similar platforms. The digital learning facilities will certainly follow similar patterns and lay outs, thus this presentation will reflect and be valid for similar projects worldwide. Since this platform is mostly based on visually enhanced learning, this paper analyses and critiques the form of visuality in which the contents are presented. The paper ends up considering issues such as ways of transmitting the information, meanings of icons and visual presentation, and the type of pedagogy that has been used to design PLEVALEX and PAU-ER. (Contains 4 figures)

ASSESSING LANGUAGE USING COMPUTER TECHNOLOGY

In this article, we propose to follow up on the most recent ARAL survey article on trends in computer-based second language assessment (Jamieson, 2005) and review developments in the use of technology in the creation, delivery, and scoring of language tests. We will discuss the promise and threats associated with computer-based language testing, including the language construct in relation to computer-based delivery and response technologies; computer-based authoring options; current developments; scoring, feedback, and reporting systems; and validation issues.

TRENDS IN COMPUTER-BASED SECOND LANGUAGE ASSESSMENT

In the last 20 years, several authors have described the possible changes that computers may effect in language testing. SinceARAL's last review of general language testing trends (Clapham, 2000), authors in theCambridge Language Assessment Serieshave offered various visions of how computer technology could alter the testing of second language skills. This chapter reflects these perspectives as it charts the paths recently taken in the field. Initial steps were made in the conversion of existing item types and constructs already known from paper-and- pencil testing into formats suitable for computer delivery. This conversion was closely followed by the introduction of computer-adaptive tests, which aim to make more, and perhaps, better, use of computer capabilities to tailor tests more closely to individual abilities and interests. Movement toward greater use of computers in assessment has been coupled with an assumption that computer-based tests should be better than their traditional predecessors, and some related steps have been taken. Corpus linguistics has provided tools to create more authentic assessments; the quest for authenticity has also motivated inclusion of more complex tasks and constructs. Both these innovations have begun to be incorporated into computer-based language tests. Natural language processing has also provided some tools for computerized scoring of essays, particularly relevant in large-scale language testing programs. Although computer use has not revolutionized all aspects of language testing, recent efforts have produced some of the research, technological advances, and improved pedagogical understanding needed to support progress.

Language testing

04–378 Arkoudis, Sophie (U. of Melbourne, Australia; Email: s.arkoudis@unimelb.edu.au) and O'Loughlin, Kieran. Tensions between validity and outcomes: teacher assessment of written work of recently arrived ESL students. Language Testing (London, UK), 24, 3 (2004), 284–304.04–379 Cheng, Lyang (Queen's U. Canada; Email: chengl@educ.queensu.ca). Rogers, Todd and Hu, Huiqin. ESL/EFL instructors' classroom assessment practices: purpose, methods and procedures. Language Testing (London, UK), 24, 3 (2004), 360–389.04–380 Davison, Chris (U. of Hong Kong, China; Email: cdavison@hkucc.hku.hk). The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australian and Hong Kong secondary schools. Language Testing (London, UK), 24, 3 (2004), 305–334.04–381 Edelenbos, Peter (U. of Groningen and The Netherlands Language Academy, The Netherlands; Email: peter.edelenbos@talenacademie.nl) and Kubanek-German, Angelika. Teacher-assessment: the concept of ‘diagnostic competence'. Language Testing (London, UK), 24, 3 (2004), 259–283.04–382 Laufer, Batia and Goldstein, Zahava (U. of Haifa, Israel; Email: batialau@research.haifa.ac.il). Testing vocabulary knowledge: size, strength, and computer adaptiveness. Language Learning (Malden, Massachusetts, USA), 54, 3 (2004), 399–436.04–383 Lee, Soyoung (Inha U., South Korea; Email: soyoungl@inha.ac.kr). A study on comparability of paper-based and computer-based reading tests scores. English Teaching (Anseongunn, South Korea), 59, 2 (2004), 165–178.04–384 Leung, Constant (Kings College, London, UK; Email: leung@kcl.ac.uk) and Mohan, Bernard. Teacher formative assessment and talk in classroom contexts: assessment as discourse and assessment of discourse. Language Testing (London, UK), 24, 3 (2004), 335–359.04–385 MacDonald, Kim (St Francis Xavier U, Canada; Email: kmcdona@stfx.ca), Nielsen, Jean and Lai, Lisa. Selecting and using computer-based language tests (CBLTs) to assess language proficiency: guidelines for educators. TESL Canada Journal/Revue TESL du Canada (Burnaby, Canada), 21, 2 (2004), 93–104.