Abstract

Within the field of Statistical Machine Translation (SMT), the neural approach (NMT) has recently emerged as the first technology able to challenge the long-standing dominance of phrase-based approaches (PBMT). In particular, at the IWSLT 2015 evaluation campaign, NMT outperformed well established state-of-the-art PBMT systems on English-German, a language pair known to be particularly hard because of morphology and syntactic differences. To understand in what respects NMT provides better translation quality than PBMT, we perform a detailed analysis of neural vs. phrase-based SMT outputs, leveraging high quality post-edits performed by professional translators on the IWSLT data. For the first time, our analysis provides useful insights on what linguistic phenomena are best modeled by neural models - such as the reordering of verbs - while pointing out other aspects that remain to be improved.

Highlights

  • The wave of neural models has eventually reached the field of Statistical Machine Translation (SMT)

  • We choose to focus on one language pair and one task because of the following advantages: (i) three state-of-the art Phrase-Based MT (PBMT) systems compared against the Neural MT (NMT) system on the same data and in the very same period; (ii) a challenging language pair in terms of morphology and word order differences; (iii) availability of MT outputs’ post-editing done by professional translators, which is very costly and rarely available

  • We analysed the output of four state-of-the-art MT systems that participated in the English-to-German task of the IWSLT 2015 evaluation campaign

Read more

Summary

Introduction

The wave of neural models has eventually reached the field of Statistical Machine Translation (SMT). The NMT systems described in (Jean et al, 2015b) ranked on par with the best phrase-based models on a couple of language pairs Such rapid progress stems from the improvement of the recurrent neural network encoderdecoder model, originally proposed in (Sutskever et al, 2014; Cho et al, 2014b), with the use of the attention mechanism (Bahdanau et al, 2015). We choose to focus on one language pair and one task because of the following advantages: (i) three state-of-the art PBMT systems compared against the NMT system on the same data and in the very same period (that of the evaluation campaign); (ii) a challenging language pair in terms of morphology and word order differences; (iii) availability of MT outputs’ post-editing done by professional translators, which is very costly and rarely available. Based on the finding that word reordering is the strongest aspect of NMT compared to the other systems, we carry out a finegrained analysis of word order errors (Section 6)

Previous Work
Experimental Setting
Task Data
Evaluation Data
MT Systems
Translation Edit Rate Measures
Overall Translation Quality
Translation quality by sentence length
Translation quality by talk
Analysis of Translation Errors
Morphology errors
Lexical errors
Word order errors
Fine-grained Word Order Error Analysis
Conclusions

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.