Synchronous Bidirectional Neural Machine Translation

Long Zhou,Chengqing Zong,Jiajun Zhang

doi:10.1162/tacl_a_00256

Abstract

AbstractExisting approaches to neural machine translation (NMT) generate the target language sequence token-by-token from left to right. However, this kind of unidirectional decoding framework cannot make full use of the target-side future contexts which can be produced in a right-to-left decoding direction, and thus suffers from the issue of unbalanced outputs. In this paper, we introduce a synchronous bidirectional–neural machine translation (SB-NMT) that predicts its outputs using left-to-right and right-to-left decoding simultaneously and interactively, in order to leverage both of the history and future information at the same time. Specifically, we first propose a new algorithm that enables synchronous bidirectional decoding in a single model. Then, we present an interactive decoding model in which left-to-right (right-to-left) generation does not only depend on its previously generated outputs, but also relies on future contexts predicted by right-to-left (left-to-right) decoding. We extensively evaluate the proposed SB-NMT model on large-scale NIST Chinese-English, WMT14 English-German, and WMT18 Russian-English translation tasks. Experimental results demonstrate that our model achieves significant improvements over the strong Transformer model by 3.92, 1.49, and 1.04 BLEU points, respectively, and obtains the state-of-the-art per- formance on Chinese-English and English- German translation tasks.1

Highlights

Neural machine translation has significantly improved the quality of machine translation in recent years (Sutskever et al, 2014; Bahdanau et al, 2015; Zhang and Zong, 2015; Wu et al, 2016; Gehring et al, 2017; Vaswani et al, 2017)
We propose a synchronous bidirectional neural machine translation (NMT) model that adopts one decoder to generate outputs with left-to-right and rightto-left directions simultaneously and interactively
Instead of multi-head intra-attention which prevents future information flow in the decoder to preserve the auto-regressive property, we propose a synchronous bidirectional attention (SBAtt) mechanism

Summary

Introduction

Neural machine translation has significantly improved the quality of machine translation in recent years (Sutskever et al, 2014; Bahdanau et al, 2015; Zhang and Zong, 2015; Wu et al, 2016; Gehring et al, 2017; Vaswani et al, 2017). NMT adopts the encoder-decoder architecture and generates the target translation from left to right. Despite their remarkable success, NMT models suffer from several weaknesses (Koehn and Knowles, 2017). The statistical results show that L2R performs better in the first 4 tokens, whereas R2L translates better in terms of the last 4 tokens This problem is mainly caused by the left-toright unidirectional decoding, which conditions each output word on previously generated outputs only, but leaving the future information from target-side contexts unexploited during translation. Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. For multi-head inter-attention of the decoder, Q are the hidden states of the previous decoder layer, and K-V pairs come from the output (z1, z2, ..., zn) of the encoder

Objectives

Methods

Results

Conclusion