Unsupervised multilingual machine translation with pretrained cross-lingual encoders

Abstract

Multilingual Neural Machine Translation (MNMT) has recently made great progress in training models that can translate between multiple languages. However, MNMT faces a significant challenge due to the lack of sufficient parallel corpora for all language pairs. Unsupervised machine translation methods, which utilize monolingual data, have emerged as a solution to this challenge. In this paper, we propose a method that leverages cross-lingual encoders, such as XLM-R, in an unsupervised manner (i.e., using monolingual data and bilingual dictionaries) to train a MNMT model. Our method initializes the MNMT model with a pre-trained cross-lingual encoder and employs two levels of alignment to further align the representation space in MNMT model. Experimental results demonstrate that our method outperforms strong baseline systems and exhibits robust domain and language transfer capabilities while preserving the performance of the original pre-trained encoder on other downstream tasks.