The influence of raters and scoring procedures on EIT outcomes: An exploratory study

Abstract

This mixed-methods study examines the role of rating modality (aural vs. written) and rater characteristics (language background, linguistics training) on outcomes on a Spanish elicited imitation task (EIT) and provides insight into rater cognition during the scoring process. Thirty Spanish-English bilingual raters were divided into two conditions—one that scored test-takers’ audio-recorded oral EIT productions, the other that scored written transcriptions. Within each condition, raters varied according to their language background (native vs. nonnative Spanish speaker) and linguistics training (no training, minimal training, moderate training, extensive training). All raters scored the same six 30-item EITs using Ortega et al.’s (2002) rubric. Ten raters additionally completed a think-aloud protocol during their scoring. Ordinal logistic mixed-effects modelling was used to explore quantitative interrater differences in scoring (k = 5,400), and content analysis (Krippendorf, 2018) was employed to investigate patterns in the verbal reports. Results reveal significant differences in EIT outcomes according to rating modality and language background but not linguistics training. Raters scoring in the aural condition and native Spanish speakers rated test items higher than their written condition and nonnative Spanish-speaking counterparts. Analyses of the verbal reports suggest some systematic differences in rating considerations according to rating modality such as differences introduced by punctuation in the written modality or by the need to determine what language was produced in the aural modality. The findings contribute novel information for EIT validity arguments and have implications for rating procedures for future EIT studies.