The effect of rater variables in the development of an occupation-specific language performance test

Abstract

There has been little research into the effect of raters' background on assessments made in language tests. This article explores the effect that the rater's occupational and linguistic background has on assessments made in an occupation-specific oral language test, the Japanese Language Test for Tour Guides. Assessments of 51 test candidates made by 33 such assessors, including native and near-native speakers of Japanese with backgrounds either in teaching Japanese as a foreign language or in tour guiding in Japanese, were compared in order to determine what effect background has on assessments made on both linguistic and 'real-world' criteria. Multifaceted Rasch analysis was used, as this allows a range of facets to be modelled; for the purposes of this research we were interested in the facet 'raters', and in particular how the scale was applied for each of the assessment categories. These were then compared across the different types of rater. While differences were found in the assessments according to rater type, these were minor and did not point to the unsuitability of any group as a whole; there were no significant differences between the different types of rater in terms of the overall grade awarded. However, there were significant differences in ratings awarded for some individual criteria, and the fact that these reveal different perceptions of what constitutes good performances has obvious implications not only for the selection and training of raters for occupation- specific language performance tests but also for the development of assess ment procedures.