Interpreting quality assessment re-imagined: The synergy between human and machine scoring

Abstract

Assessment of interpreting quality is a ubiquitous social practice in the interpreting industry and academia. In this article, we focus on both psychometric and social dimensions of assessment practice, and analyse two major assessment paradigms, namely, human rater scoring and automatic machine scoring. Regarding human scoring, we describe five specific methods, including atomistic scoring, questionnaire-based scoring, multi-methods scoring, rubric scoring, and ranking, and critically analyse their respective strengths and weaknesses. In terms of automatic scoring, we highlight four assessment approaches that have been researched and operationalised in cognate disciplines and interpreting studies, including automatic assessment based on temporal variables, linguistic/surface features, machine translation metrics, and quality estimation methodology. Finally, we problematise the socio-technological tension between these two paradigms and envisage human–machine collaboration to produce psychometrically sound and socially responsible assessment. We hope that this article sparks more scholarly discussion of rater-mediated and automatic assessment of interpreting quality from a psychometric-social perspective.