Inter-rater agreement on Mandarin tone categorization: Contributing factors and implications

Abstract

Factors that may/may not influence inter-rater reliability in assessing the accuracy of monosyllabic Mandarin tones produced by children and adults were examined in three experiments. Experiment 1 investigated inter-judge reliability in two groups of Mandarin-speaking adults—one group from China and the other from Taiwan—on their categorization of filtered tones produced by adults and children. The results showed that the magnitude of inter-rater agreement was associated with the production accuracy of the speakers; the judges attained lower agreement in categorizing children’s tones than adults’ tones. All judges who indicated that Mandarin was their strongest language and that they had learned and used Mandarin since birth performed similarly in their tone categorization despite the fact that they came from and were residing in different countries. Similar results was found in experiment 2, in which one group of the judges in experiment 1 categorized tones produced by a new and larger group of adults and children, and in experiment 3, in which a different group of adults categorized another new set of tones produced by a different group of speakers. Implications of the findings in research design will be discussed. [Work supported by NIH-NIDCD (1 F31 DC008479-01A1) and NSF (OISE-0611641).]