Predicting insertion positions in word-level machine translation quality estimation

Abstract

Word-level machine translation (MT) quality estimation (QE) is usually formulated as the task of automatically identifying which words need to be edited (either deleted or replaced) in a translation T produced by an MT system. The advantage of estimating MT quality at the word level is that this information can be used to guide post-editors since it enables the identification of the specific words in T that need to be edited in order to ease their work. However, word-level MT QE, as defined in the current literature, has an obvious limitation: it does not identify the positions in T in which missing words need to be inserted. To deal with this limitation, we propose a method which identifies both word deletions and insertion positions in T. This is, to the best of our knowledge, the first approach allowing the identification of insertion positions in word-level MT QE. The method proposed can use any source of bilingual information – such as MT, dictionaries, or phrase-level translation memories – to extract features that are then used by a neural network to produce a prediction for both words and insertion positions (gaps between words) in the translation T. In this paper, several feature sets and neural network architectures are explored and evaluated on publicly-available datasets used in previous evaluation campaigns for word-level MT QE. The results confirm the feasibility of the proposed approach, as well as the usefulness of sharing information between the two prediction tasks in order to obtain more reliable quality estimations.