RELIABILITY OF THE TEST OF SPOKEN ENGLISH REVISITED

Abstract

ABSTRACTThe Test of Spoken English (TSE) is an internationally administered instrument for assessing nonnative speakers' proficiency in speaking English. The research foundation of the TSE® examination is currently described in the Manual for Score Users. This publication describes technical characteristics of the test, including such psychometric characteristics as level of difficulty, reliability, and validity. Consistent with the ETS Standards for Quality and Fairness, the Manual refers to two sources of variation other than the achievement being measured: interrater reliability and internal consistency. Because the reported data on both issues were drawn from a 1980 study of reliabilities based on two raters, newer data and more extensive analyses were needed.The present study uses data from recent administrations of the TSE examination. Analysis of variance examined the effects of scale, section, examinee, and rater, as well as the interactions of these factors. Reliabilities were reported for item, section, and scale scores. Common and unique variance estimates were developed for each scale score for which a section was rated. Estimates of the effects of altering section lengths suggested that some sections should be lengthened and others omitted if reliability were to be maximized. Other suggestions were offered for improving reliability.