Deriving oral assessment scales across different tests and rater groups

Abstract

The purpose of this study is to derive the criteria/dimensions underlying learners' L2 oral ability scores across three tests: an oral interview, a narration and a read-aloud. A stimulus tape of 18 speech samples was presented to three native speaker rater groups for evaluation. The rater groups included teachers of Arabic as a foreign language in the USA, nonteaching Arabs residing in the USA for at least one year and nonteaching Arabs living in their home country (Lebanon). Each of the raters provided a holistic score for every speech sample. Holistic scores were analysed using the INDSCAL multidimensional scaling model. Results showed that the nonmetric three-dimensional solution provided a good fit to the data. Both regression and speech sample analyses were employed to identify those dimensions. Additionally, subject weights indicated that the three rater groups were emphasizing the three dimensions differentially, thus demon strating that native speaker groups with varied backgrounds perceive the L2 oral construct differently. The study contends that researchers might need to reconsider employing generic component scales. A research approach that derives scales empirically according to the given tests and audiences, and according to the purpose of assessment, is recommended. Finally, replicating this study using other languages, L2 oral ability levels, tests and rater groups is suggested.