Investigating and optimizing score dependability of a local ITA speaking test across language groups: A generalizability theory approach

Abstract

With the present study I investigated the sources of score variance and dependability in a local oral English proficiency test for potential international teaching assistants (ITAs) across four first language (L1) groups, and suggested alternative test designs. Using generalizability theory, I examined the relative importance of L1s (i.e., Indian, Korean, Mandarin, and Spanish), examinees, tasks, and ratings to score variability, and estimated dependability across the L1s. The analyses identified examinees as the largest contributor, which is important for high dependability and validity arguments for test scores. Effects of ratings and tasks were small, but L1 effects on score variance were considerable, with the Indian group’s dependability lowest. Unlike previous generalizability theory studies on L1 effects, however, further analyses revealed that the L1 effects highly likely reflect proficiency differences rather than strong bias when comparing the percent agreement of the ratings, external criteria of examinee English proficiency, and underlying score distributions. I discuss the proficiency differences related to varied socio-linguistic contexts of using and learning English. Lastly, I suggest an alternative design with fewer items and one additional rating for improved dependability. Considering multiple test purposes specific to ITA testing (i.e., efficiency, construct representation, formative advantages), I propose a flexible approach.