Non-Expert Raters’ Scoring Behavior and Cognition in Assessing Pragmatic Production in L2 Chinese

Abstract

AbstractThis chapter reports on a study investigating non-expert raters’ scoring behavior and cognitive processes involved in evaluating speech acts and pragmatic routines in L2 Chinese. Pragmatic production data were collected from 51 American learners of Chinese, who completed a 12-item oral Discourse Completion Test (DCT). The learners were divided into 15 groups, each including the same six learners and three different learners. A total of 101 non-expert, native Chinese raters evaluated the oral productions of one learner group and were encouraged to verbalize their scoring rationale. Results showed that, although the raters varied significantly in scoring severity, their scoring behaviors were consistent, with very limited instances of scoring bias. Qualitative analysis based on 2753 verbal protocols revealed that the raters predominantly oriented towards criteria related to holistic meaning expression in assessing speech acts and routines. They prioritized criteria related to linguistic expressions (notably those concerning vocabulary knowledge) in evaluating pragmatic routines, and they paid more attention to criteria related to interactional skills in assessing speech acts. Boundary crossing implications are discussed in relation to pragmatics assessment and L2 Chinese teaching.KeywordsPragmatics assessmentL2 ChineseRater cognitionScoring behavior