Google Translate in Massive Open Online Courses: Evaluation of Accuracy and Comprehensibility among Female Users

Abstract

The increasing production of audiovisual texts online has led to the growing use of Machine Translation (MT) in its raw form to facilitate multilingual access. In this study, we examine the accuracy of Google Translate’s English–Arabic subtitles and their comprehensibility among female users through a case analysis of Massive Open Online Courses (MOOCs). We also seek to contribute to the research on MT by providing empirical evidence on the use of MT in audiovisual contexts, which is a relatively new and under-researched study area. The research questions considered in this study are: (i) What types of accuracy errors can be identified in Google Translate’s English–Arabic translation of MOOC subtitles? (ii) What effects do machine-translated MOOC subtitles have on female users’ comprehension? The study used a mixed-methods approach. First, 147 machine-translated subtitles of five MOOC videos were annotated by six evaluators based on five accuracy error types: mistranslation, terminology, omission, addition, and untranslated-segment errors. Second, the comprehensibility of the machine-translated subtitles was examined through a quasi-experiment. Sixty-two female participants were divided into two groups. Each group was presented with MOOC videos with either machine- or human-translated subtitles. Then, the participants completed a comprehension test. The qualitative analysis of the accuracy errors showed the occurrence of all five accuracy error types. The quantitative analysis of users’ comprehension revealed a lack of statistically significant differences between the examined groups. The study’s results suggested that MT can be useful in facilitating users’ access to MOOC video content.