Abstract
With the present study I investigated the sources of score variance and dependability in a local oral English proficiency test for potential international teaching assistants (ITAs) across four first language (L1) groups, and suggested alternative test designs. Using generalizability theory, I examined the relative importance of L1s (i.e., Indian, Korean, Mandarin, and Spanish), examinees, tasks, and ratings to score variability, and estimated dependability across the L1s. The analyses identified examinees as the largest contributor, which is important for high dependability and validity arguments for test scores. Effects of ratings and tasks were small, but L1 effects on score variance were considerable, with the Indian group’s dependability lowest. Unlike previous generalizability theory studies on L1 effects, however, further analyses revealed that the L1 effects highly likely reflect proficiency differences rather than strong bias when comparing the percent agreement of the ratings, external criteria of examinee English proficiency, and underlying score distributions. I discuss the proficiency differences related to varied socio-linguistic contexts of using and learning English. Lastly, I suggest an alternative design with fewer items and one additional rating for improved dependability. Considering multiple test purposes specific to ITA testing (i.e., efficiency, construct representation, formative advantages), I propose a flexible approach.
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.