Using TOEFL sub-scores to predict SPEAK test outcome: A multivariate Bayesian model

Abstract

The Test of English as a Foreign Language (TOEFL), produced by the Educational Testing Service (ETS), has been in use in institutions of higher American education since the 1960s as a means of measuring incoming international students' English proficiency. But like any test, the TOEFL is imperfect. For instance, whereas a high TOEFL score may be sufficient to admit an international student to an American graduate school, many colleges and universities require more rigorous proof of a student's English proficiency—often in the form of a passing score on a school-specific oral assessment—if he seeks employment as a Graduate Teaching Assistant (GTA). This is the case at the University of Virginia (UVa), where international graduate students are required to take the Speaking Proficiency English Assessment Kit (SPEAK) test if they apply for GTA positions; without a passing score, would-be GTAs are prohibited from interacting with undergraduate students in a teaching capacity. Academic departments sustain considerable risk extending offers of employment to GTAs based on the students' high TOEFL scores, as strong TOEFL performance does not guarantee a passing SPEAK test score. To mitigate this risk, forecasting models which use the TOEFL sub-scores of Speaking, Listening, Writing, and Reading to forecast SPEAK test outcome are applied. A student's sub-scores act as predictive inputs to each model, which outputs the posterior probability of his SPEAK test failure. Bayes Theorem provides the structure required to obtain this probability, and the multivariate meta-Gaussian distribution captures the stochastic dependence between the sub-scores. Therefore, these models are classified as Bayesian Meta-Gaussian Forecasters (BMGFs). Our findings are that (i) no combination of two, three, or four sub-scores is more informative than the Speaking sub-score alone, and (ii) in the absence of the Speaking sub-score, no combination of two, three, or four sub-scores is more informative than the Listening sub-score alone. Academic departments at UVa could use these probabilistic forecasts to better account for risk when dispensing offers of employment to potential GTAs.