Misalignment Detection for Web-Scraped Corpora: A Supervised Regression Approach

Defauw Defauw,Everaert Everaert,Bogaert Bogaert,Scholte Scholte,Brabers Brabers,Bardadym Bardadym,Winckel Winckel,Szoc Szoc,Vanallemeersch Vanallemeersch,Mijic Mijic

doi:10.3390/informatics6030035

Abstract

To build state-of-the-art Neural Machine Translation (NMT) systems, high-quality parallel sentences are needed. Typically, large amounts of data are scraped from multilingual web sites and aligned into datasets for training. Many tools exist for automatic alignment of such datasets. However, the quality of the resulting aligned corpus can be disappointing. In this paper, we present a tool for automatic misalignment detection (MAD). We treated the task of determining whether a pair of aligned sentences constitutes a genuine translation as a supervised regression problem. We trained our algorithm on a manually labeled dataset in the FR–NL language pair. Our algorithm used shallow features and features obtained after an initial translation step. We showed that both the Levenshtein distance between the target and the translated source, as well as the cosine distance between sentence embeddings of the source and the target were the two most important features for the task of misalignment detection. Using gold standards for alignment, we demonstrated that our model can increase the quality of alignments in a corpus substantially, reaching a precision close to 100%. Finally, we used our tool to investigate the effect of misalignments on NMT performance.

Highlights

A machine translation (MT) system usually increases its performance when more training data is added
Our experiments showed that removing misalignments can be beneficial in terms of data selection, leaving misalignments in the training data did not result in a decrease in neural machine translation (NMT) performance
We performed an extrinsic evaluation by applying misalignment detection (MAD) on two web-scraped corpora, and examined the effect of removing misalignments on NMT performance

Summary

Introduction

A machine translation (MT) system usually increases its performance when more training data is added. Previous research showed that the performance of a neural machine translation (NMT) system decreases when the training data contains noisy sentence pairs [2,3], as an NMT model tends to assign high probabilities to rare events. Data crawled from the web typically contains a variety of noise: untranslated sentences, language and encoding errors, short segments, and misalignments. The effect of these types of noise on NMT and SMT was systematically investigated by Khayrallah and Koehn [4]. It was shown that alignment errors can impact SMT performance as well [5]

Objectives

Methods

Results

Conclusion