Searching for Poor Quality Machine Translated Text: Learning the Difference between Human Writing and Machine Translations

Abstract

As machine translation (MT) tools have become mainstream, machine translated text has increasingly appeared on multilingual websites. Trustworthy multilingual websites are used as training corpora for statistical machine translation tools; large amounts of MT text in training data may make such products less effective. We performed three experiments to determine whether a support vector machine (SVM) could distinguish machine translated text from human written text (both original text and human translations). Machine translated versions of the Canadian Hansard were detected with an F-measure of 0.999. Machine translated versions of six Government of Canada web sites were detected with an F-measure of 0.98. We validated these results with a decision tree classifier. An experiment to find MT text on Government of Ontario web sites using Government of Canada training data was unfruitful, with a high rate of false positives. Machine translated text appears to be learnable and detectable when using a similar training corpus.KeywordsSupport Vector MachineNatural Language ProcessingMachine TranslationOriginal TextStatistical Machine TranslationThese keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.