Abstract
AbstractExisting approaches to neural machine translation (NMT) generate the target language sequence token-by-token from left to right. However, this kind of unidirectional decoding framework cannot make full use of the target-side future contexts which can be produced in a right-to-left decoding direction, and thus suffers from the issue of unbalanced outputs. In this paper, we introduce a synchronous bidirectional–neural machine translation (SB-NMT) that predicts its outputs using left-to-right and right-to-left decoding simultaneously and interactively, in order to leverage both of the history and future information at the same time. Specifically, we first propose a new algorithm that enables synchronous bidirectional decoding in a single model. Then, we present an interactive decoding model in which left-to-right (right-to-left) generation does not only depend on its previously generated outputs, but also relies on future contexts predicted by right-to-left (left-to-right) decoding. We extensively evaluate the proposed SB-NMT model on large-scale NIST Chinese-English, WMT14 English-German, and WMT18 Russian-English translation tasks. Experimental results demonstrate that our model achieves significant improvements over the strong Transformer model by 3.92, 1.49, and 1.04 BLEU points, respectively, and obtains the state-of-the-art per- formance on Chinese-English and English- German translation tasks.1
Highlights
Neural machine translation has significantly improved the quality of machine translation in recent years (Sutskever et al, 2014; Bahdanau et al, 2015; Zhang and Zong, 2015; Wu et al, 2016; Gehring et al, 2017; Vaswani et al, 2017)
We propose a synchronous bidirectional neural machine translation (NMT) model that adopts one decoder to generate outputs with left-to-right and rightto-left directions simultaneously and interactively
Instead of multi-head intra-attention which prevents future information flow in the decoder to preserve the auto-regressive property, we propose a synchronous bidirectional attention (SBAtt) mechanism
Summary
Neural machine translation has significantly improved the quality of machine translation in recent years (Sutskever et al, 2014; Bahdanau et al, 2015; Zhang and Zong, 2015; Wu et al, 2016; Gehring et al, 2017; Vaswani et al, 2017). NMT adopts the encoder-decoder architecture and generates the target translation from left to right. Despite their remarkable success, NMT models suffer from several weaknesses (Koehn and Knowles, 2017). The statistical results show that L2R performs better in the first 4 tokens, whereas R2L translates better in terms of the last 4 tokens This problem is mainly caused by the left-toright unidirectional decoding, which conditions each output word on previously generated outputs only, but leaving the future information from target-side contexts unexploited during translation. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. For multi-head inter-attention of the decoder, Q are the hidden states of the previous decoder layer, and K-V pairs come from the output (z1, z2, ..., zn) of the encoder
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callMore From: Transactions of the Association for Computational Linguistics
Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
