Human or Neural Translation?

Shivendra Bhardwaj,Michel Simard,Gabriel Bernier-Colborne,David Alfonso Hermelo,Cyril Goutte,Phillippe Langlais

doi:10.18653/v1/2020.coling-main.576

Abstract

Deep neural models tremendously improved machine translation. In this context, we investigate whether distinguishing machine from human translations is still feasible. We trained and applied 18 classifiers under two settings: a monolingual task, in which the classifier only looks at the translation; and a bilingual task, in which the source text is also taken into consideration. We report on extensive experiments involving 4 neural MT systems (Google Translate, DeepL, as well as two systems we trained) and varying the domain of texts. We show that the bilingual task is the easiest one and that transfer-based deep-learning classifiers perform best, with mean accuracies around 85% in-domain and 75% out-of-domain .

Highlights

This work addresses the task of distinguishing between translations produced by humans and machines
We compare feature-based approaches with several deep learning methods, investigating the impact of text domains and MT systems, paying attention to cases where the translation engine at test time is different from the one used for training, which we found often not studied in related work
The best transfer learning method we tested recorded an in-domain accuracy of 87.6% and out-of-domain performances ranging between 65.4% and 84.2% depending on the domain of texts and MT system considered

Summary

Introduction

This work addresses the task of distinguishing between translations produced by humans and machines. Practical applications for this include: improving machine translation systems (Li et al, 2015), filtering parallel data mined from the Web (Arase and Zhou, 2013) and evaluating machine translation quality without reference translations (Aharoni et al, 2014). We compare feature-based approaches with several deep learning methods, investigating the impact of text domains and MT systems (in-house neural engines, Google Translate, DeepL), paying attention to cases where the translation engine at test time is different from the one used for training, which we found often not studied in related work. We believe our study offers many new data points, and hope it will foster research on this timely topic

Objectives

Methods

Findings

Conclusion