A STUDY OF CHARACTERISTICS OF THE SPEAK TEST

Abstract

ABSTRACTThe purpose of this project was to investigate issues of reliability and validity arising from the local administration and scoring of the Spoken Proficiency English Assessment Kit (SPEAK®), the “off‐the‐shelf” version of the Test of Spoken English (TSE®), and to assess the validity of the SPEAK test as a predictor of classroom English ability for international teaching assistants (ITAs).The reliability issues addressed included interrater consistency within and across institutions, score comparability across institutions and Educational Testing Service (ETS®), and differential weightings by individual raters, institutions, and TSE raters of the three diagnostic language factors in assessing the overall comprehensibility of spoken English. SPEAK tests were administered to 119 international teaching assistants at three participating institutions, arranged in a counterbalanced design with 12 ETS‐scored tests of the same form, and scored by raters at each institution. High interrater reliability coefficients as measured by Pearson product‐moment correlations were obtained for the global comprehensibility score and the diagnostic subscores; other means of assessing interrater consistency, however, revealed some variation among individual raters. Scores were generally comparable across institutions and ETS in the high‐to‐mid range on the overall comprehensibility scale, although differences among institutions increased in the lower range of scores. Other interesting individual and institutional differences were found regarding correspondence with ETS scores. With regard to the weighting by different institutions of the three component subscales, regression analyses showed essentially no differences and thus indicate that the same construct is being measured by raters at each institution.For the predictive validity phase of the study, the SPEAK scores of the 59 ITAs who held classroom teaching responsibilities were used as predictive variables in multiple regression analyses. Criterion variables were student ratings of the ITA's spoken language use in the classroom and other instructional contexts as well as ratings of the ITA on several dimensions of teaching effectiveness from the Student Instructional Report (SIR), a standardized instructor/course rating instrument. Although squared multiple correlations for predicting student ratings of ITAs' spoken English proficiency from SPEAK scores were highly significant for nearly all of the student ratings, regression coefficients for the separate scales of SPEAK were consistently not significant due to the high multicollinearity of those scales, suggesting some redundancy in the information provided by the separate scales. SPEAK scales were also correlated with the more global aspects of teaching performance, as measured by student responses to relevant questions and question groupings on the SIR. These relationships were generally weak and not statistically significant. The SPEAK scales were thus effective predictors of student ratings of the classroom language proficiency of their instructors, although not of their teaching effectiveness. This pattern parallels the findings of an earlier study of the predictive validity of TSE.Results of the study indicate that the locally administered and scored SPEAK test, like TSE, is a reasonably reliable instrument for local screening and an appropriate measure of the general communicative proficiency in spoken English of ITAs in U.S. instructional settings.