What the analytic versus holistic scoring of international teaching assistants can reveal: Lexical grammar matters

Abstract

Second-language (L2) testing researchers have explored the relationship between speakers’ overall speaking ability, reflected by holistic scores, and the speakers’ performance on speaking subcomponents, reflected by analytic scores (e.g., McNamara, 1990; Sato, 2011). These research studies have advanced applied linguists’ understanding of how raters view the components of effective speaking skills, but the authors of the studies either used analytic composite scores, instead of true holistic ratings, or ran regression analyses with highly correlated subscores, which is problematic. To address these issues, 10 experienced ITA raters rated the speaking of 127 international teaching assistant (ITA) candidates using a four-component analytic rubric. In addition, holistic ratings were provided for the 127 test takers from a separate (earlier) scoring by two experienced ITA raters. The two types of scores differentiated examinees in similar ways. The variability observed in students’ holistic scores was reflected in their analytic scores. However, among the four analytic subscales, examinees’ scores on Lexical and Grammatical Competence had the greatest differentiating power. Its scores indicated with a high level of accuracy who passed the test and who did not. The paper discusses the components contributing to ITAs’ L2 oral speaking proficiency, and reviews pedagogical implications.