DIRECT EVALUATION OF SECOND‐LANGUAGE PROFICIENCY: THE EFFECT OF RATER AND SCALE IN ORAL INTERVIEWS

Abstract

Oral interviews have frequently been used as a means of testing speaking proficiency and are considered a supplement to the standard paper‐and‐pencil tests of second language proficiency, of which the TOEFL is an example. Although a properly conducted interview is a suitable means for examinees to demonstrate their ability to use the rules of the language effectively and proficiently, methods for evaluating such performances rest on scales and judgments. This paper reports the results of a study in which the same two judges used the same four scales in evaluating the performances of two different groups of subjects. It was found that there was no significant difference between the mean performance of the two groups, between the two judges, or among the four scales. It was also found that the greatest source of variation in scores was attributable to differences among subjects. It was also found that interaction effects between judge and scale and between judge and subject exist. Such results suggest that 1) oral proficiency interviews ought to be conducted by at least two judges at the same time so that scores are independent of a single judge; 2) the four scales ought not to be considered separate and distinct measures of oral proficiency if the same judges evaluate a performance on all four scales; 3) the procedure is capable of distinguishing the more proficient speakers from the less adept ones.