Abstract
This paper describes the steps to convert a paper-and-pencil English proficiency test for academic purposes, consisting of multiple choice items administered following the Admissible Probability Measurement Procedure [24], adopted by the graduate program at the Institute of Mathematics and Computer Sciences at the University of São Paulo (ICMC-USP), Brazil, to a computerized adaptive test (CAT) based on an Item Response Theory Model (IRT). Despite the fact that the program accepts various internationally recognized tests that attest non-native speakers English proficiency, such as the Test of English as a Foreign Language (TOEFL), the International English Language Testing System (IELTS) and the Cambridge English: Proficiency (CPE), for instance, its requirement is incoherent in public universities in Brazil due to the cost, which ranges from US$ 200.00 to US$ 300.00 per exam. The TAI-PI software (Computerized Adaptive Test for English Proficiency), which was developed in Java language and SQLite, started to be used to assess the English pro?ciency of students on the program from October, 2013. The statistical methodology used was defined considering the history and aims of the test and adopted Samejima's Graded Response Model [21], the Kullback-Leibler information criterion for item selection, the a posteriori estimation method for latent trait [2] and the Shadow Test approach [29] to impose restrictions (content and test length) on the test composition of each individual. A description of the test design, the statistical methods used, and the results of a real application of TAI-PI for graduate students are presented in this paper, as well as the validation studies of the new methodology for pass or fail classification, showing the good quality of the new evaluation system and examination of improvement using the IRT and CAT methods.
Highlights
Having a good command of English is fundamental for graduate students from various fields of science, so that they will be able to understand the course content properly, as well as develop and disseminate research carried out
To justify the choice of statistical methodology adopted in this study, in what follows we present the format in which the EPI was applied to the CCMC program between 2002 and 2013, providing the basis for building the Computerized Adaptive Test of English Proficiency (TAI-PI)
To contrast the Item Response Theory (IRT) results with the Admissible Probability Measurement Procedure (APM) results, which are the only available methods to evaluate the proficiency of theses students, the cutoff point definition was based on θ40
Summary
Having a good command of English is fundamental for graduate students from various fields of science, so that they will be able to understand the course content properly, as well as develop and disseminate research carried out. Besides the advantages mentioned above, CAT in combination with IRT make it possible to calculate comparable proficiencies between individuals who answered different sets of items, and at different times [14, 32] This greatly facilitates evaluating constructs on a large-scale resulting in its use in important examinations, such as the Graduate Record Examination (GRE) [6, 11], developed by the Educational Testing Service (ETS) in 1996; the TOEFL [10, 12, 33], developed by ETS and the Armed Services Vocational Aptitude Battery Test [23,24], developed by the United States Department of Defense to select potential recruits for military service. For students who failed according to this criterion, but almost passed if it was not for the answers of one or two items (maximum), an alternative criterion is available considering only Module 1 “Scientific Text Structure”
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
