Exploring Graph-based Transformer Encoder for Low-Resource Neural Machine Translation

Abstract

The Transformer is commonly used in Neural Machine Translation (NMT), but it faces issues with over-parameterization in low-resource settings. This means that simply increasing the model parameters significantly will not lead to improved performance. In this study, we propose a graph-based approach that slightly increases the parameters while significantly outperforming the scaled version of the Transformer. We accomplish this by utilizing Graph Neural Networks to encode Universal Conceptual Cognitive Annotation (UCCA), allowing the linguistic features of UCCA to be incorporated into the word embeddings. This improves the performance of the NMT system since the word embedding is now more capable and informative. Experimental results demonstrate that the proposed method outperforms the scaled Transformer model by +0.4, +0.41, and +0.33 BLEU, respectively, in English-Vietnamese/French/Czech datasets. Furthermore, this method reduces the number of parameters by 47% when compared to the scaled Transformer. A thorough analysis of error patterns reveals that the proposed method provides structural awareness to translation systems. Our code is available at: https://github.com/nqbinh17/UCCA_GNN.