Multilingual Machine Translation Research Articles

Multilingual Neural Machine Translation for Indic to Indic Languages

The method of translation from one language to another without human intervention is known as Machine Translation (MT). Multilingual neural machine translation (MNMT) is a technique for MT that builds a single model for multiple languages. It is preferred over other approaches, since it decreases training time and improves translation in low-resource contexts, i.e., for languages that have insufficient corpus. However, good-quality MT models are yet to be built for many scenarios such as for Indic-to-Indic Languages (IL-IL). Hence, this article is an attempt to address and develop the baseline models for low-resource languages i.e., IL-IL (for 11 Indic Languages (ILs)) in a multilingual environment. The models are built on the Samanantar corpus and analyzed on the Flores-200 corpus. All the models are evaluated using standard evaluation metrics i.e., Bilingual Evaluation Understudy (BLEU) score (with the range of 0 to 100). This article examines the effect of the grouping of related languages, namely, East Indo-Aryan (EI), Dravidian (DR), and West Indo-Aryan (WI) on the MNMT model. From the experiments, the results reveal that related language grouping is beneficial for the WI group only while it is detrimental for the EI group and it shows an inconclusive effect on the DR group. The role of pivot-based MNMT models in enhancing translation quality is also investigated in this article. Owing to the presence of large good-quality corpora from English (EN) to ILs, MNMT IL-IL models using EN as a pivot are built and examined. To achieve this, English-Indic Language (EN-IL) models are developed with and without the usage of related languages. Results show that the use of related language grouping is advantageous specifically for EN to ILs. Thus, related language groups are used for the development of pivot MNMT models. It is also observed that the usage of pivot models greatly improves MNMT baselines. Furthermore, the effect of transliteration on ILs is also analyzed in this article. To explore transliteration, the best MNMT models from the previous approaches (in most of cases pivot model using related groups) are determined and built on corpus transliterated from the corresponding scripts to a modified Indian language Transliteration script (ITRANS). The outcome of the experiments indicates that transliteration helps the models built for lexically rich languages, with the best increment of BLEU scores observed in Malayalam (ML) and Tamil (TA), i.e., 6.74 and 4.72, respectively. The BLEU score using transliteration models ranges from 7.03 to 24.29. The best model obtained is the Punjabi (PA)-Hindi (HI) language pair trained on PA-WI transliterated corpus.

Advancements in Neural Machine Translation: Methodological Innovations and Empirical Insights for Cross-Linguistic Discourse Preservation

Abstract: Natural Language Processing (NLP), in the form recognizable today, really began to take hold in the 1980s, when machine learning helped propel it to soaring heights. However, due to a lack of processing power, machine learning, and to an extent, NLP, started slowing down in innovation and ideas and had almost ground to a relative halt, until the last decade, when a sudden increase in both productivity and interest in the machine learning helped increase the amount of knowledge in the space itself. This review provides several different case studies using different methodologies. The first paper was a deep analysis on how researchers were able to use Tesseract and Google Vision in tandem with automatic data mining methods to enrich the Cherokee language database in order to preserve it from extinction. The second paper takes a query translation-based approach toward translating English to Indian languages and utilizes a Multilingual Cross-Language Information Retrieval (MLCIR) system with tools such as Part of Speech Tagger (POST), Stop-Word, and Porter Stemmer. The third paper presents CoVe, which transfers knowledge from machine translation to improve performance on NLP tasks like sentiment analysis and question answering by using contextualized word vectors along with word embeddings, achieving new state-of-the-art results on some datasets. The fourth paper aims to translate English to Pakistan Sign Language (PSL) and also uses POST and goes through dependency analysis, sentence classification, and PSL using PLS trees. The fifth paper uses a Multilingual Neural Machine Translation (NMT) system for LowResource languages and incorporates two main models: a recurrent NMT and a transformer NMT. The sixth paper analyzes how a fine-tuned transformer model seems to work better than transformer models trained from scratch on high-resource languages, while vice-versa seems to occur for low-resource languages. The seventh paper adds to this by talking about how multilingual translation seems to work better than a back-translation model. Given the diverse array of approaches that could be used, we aim to identify the most efficient and correct methodology for future researchers to use in their work, based on the papers in this literature review.

The translation teaching platform based on multilingual corpora of Xi Jinping: The Governance of China: Design, resources and applications

Over the last two decades, the domain of corpus-based translation studies has evolved into a significant paradigm, instrumental in probing a diverse array of scholarly inquiries. This article demonstrates the various uses of corpus analysis techniques that have gained popularity in translation studies, and it presents a new multilingual corpus. This corpus showcases the evolving nature of the field as it expands to accommodate new conceptual frameworks and methodologies. In addition to addressing the use of corpora in translating, learning, and analyzing translation within the context of multilingual corpora and machine translation, this article reports on a pilot project concerning the construction of an AI-based translation teaching platform. Utilizing multilingual corpora, the platform aims to create a foundation for extensive translation instruction and multilingual training at the tertiary level. Following a discussion on the evolution of corpus-based approaches in translation studies and their connection to the growth of translation education as a profession, the paper describes the actual design of the platform. This includes the theoretical framework, the methodology of annotation, and the execution of the software programme. These corpora are relevant not only to individuals studying translation but also to the related multilingual corpora, comprising collections of originals and their translations in the same language.

Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation.

When the amount of parallel sentences available to train a neural machine translation is scarce, a common practice is to generate new synthetic training samples from them. A number of approaches have been proposed to produce synthetic parallel sentences that are similar to those in the parallel data available. These approaches work under the assumption that non-fluent target-side synthetic training samples can be harmful and may deteriorate translation performance. Even so, in this paper we demonstrate that synthetic training samples with non-fluent target sentences can improve translation performance if they are used in a multilingual machine translation framework as if they were sentences in another language. We conducted experiments on ten low-resource and four high-resource translation tasks and found out that this simple approach consistently improves translation performance as compared to state-of-the-art methods for generating synthetic training samples similar to those found in corpora. Furthermore, this improvement is independent of the size of the original training corpus, the resulting systems are much more robust against domain shift and produce less hallucinations.

Unsupervised multilingual machine translation with pretrained cross-lingual encoders

Multilingual Neural Machine Translation (MNMT) has recently made great progress in training models that can translate between multiple languages. However, MNMT faces a significant challenge due to the lack of sufficient parallel corpora for all language pairs. Unsupervised machine translation methods, which utilize monolingual data, have emerged as a solution to this challenge. In this paper, we propose a method that leverages cross-lingual encoders, such as XLM-R, in an unsupervised manner (i.e., using monolingual data and bilingual dictionaries) to train a MNMT model. Our method initializes the MNMT model with a pre-trained cross-lingual encoder and employs two levels of alignment to further align the representation space in MNMT model. Experimental results demonstrate that our method outperforms strong baseline systems and exhibits robust domain and language transfer capabilities while preserving the performance of the original pre-trained encoder on other downstream tasks.

Language relatedness evaluation for multilingual neural machine translation

Multilingual neural machine translation (MNMT) has attracted more and more attention in recent days because it can use a single neural machine translation (NMT) model to translate between multiple languages. As several languages are involved in MNMT, recent studies have shown that using part of these languages rather than all of them to train the model leads to comparable results. However, previous work on this topic mainly focuses on language clustering and features defined by linguists. The semantic relationship and language distance are not fully considered. How to select the most related language pairs to current low-resource pair to optimize the performance of MNMT is still an open question. In this paper, we propose to take language relatedness computation as a ranking problem, where features such as language distance, linguistic typological information and semantic relatedness features are incorporated into a random decision forest to improve the language relatedness evaluation (LRE) for MNMT. Since the model only focuses on monolingual LRE in general cross-lingual natural language processing tasks, we also propose two features related to machine translation (data size and bilingual relatedness) to predict the final language pairs. Experimental results on IWSLT and WMT datasets show that our proposed LRE method can achieve significant improvements compared with other models. We also conducted several groups of experiments on IWSLT and WMT datasets to further evaluate the effectiveness of the proposed method on MNMT. The results show that the MNMT model trained on language pairs predicted by the LRE method outperforms other language selection methods.

Low-resource Multilingual Neural Translation Using Linguistic Feature-based Relevance Mechanisms

This article investigates approaches to effectively harness source-side linguistic features for low-resource multilingual neural machine translation (MNMT). Previous works focus on using various features of a word such as lemma, part-of-speech tag, dependency label, and so on, to improve translation quality in a low-resource scenario. However, these studies deal with bilingual translation and do not focus on using features in multilingual training setups. Our work focuses on this particular point and experiments with low-resource multilingual models incorporating source-side linguistic features. Although techniques for integrating features into an NMT model such as concatenation and feature relevance perform quite well in bilingual settings, they do not work well in multilingual settings. To remedy this, we propose the use of dummy features and language indicator features in MNMT models. Experiments are conducted on English to Asian language translation on a multilingual, multi-parallel corpus spanning English and eight Asian languages where for each language pair, the training data size does not exceed 20,000 parallel sentences. After establishing strong bilingual baselines using feature relevance mechanisms and multilingual baselines without any features, we show that our proposed dummy features and language indicator features, in combination with feature relevance mechanisms, yield significant improvements in BLEU points for all language pairs. We then analyze our models from the perspectives of model sizes, the impact of individual linguistic features, validation perplexity computed during training, visualization of the activations of the relevance mechanisms, and exhaustive tuning of hyperparameters. We also report preliminary results for multilingual multi-way models using linguistic features.

Retracted: Multilingual Machine Translation System Based on Decoder Recurrent Neural Network

Retracted: Multilingual Machine Translation System Based on Decoder Recurrent Neural Network

Improving Multilingual Neural Machine Translation System for Indic Languages

The Machine Translation System (MTS) serves as effective tool for communication by translating text or speech from one language to another language. Recently, neural machine translation (NMT) has become popular for its performance and cost-effectiveness. However, NMT systems are restricted in translating low-resource languages as a huge quantity of data is required to learn useful mappings across languages. The need for an efficient translation system becomes obvious in a large multilingual environment like India. Indian languages (ILs) are still treated as low-resource languages due to unavailability of corpora. In order to address such an asymmetric nature, the multilingual neural machine translation (MNMT) system evolves as an ideal approach in this direction. The MNMT converts many languages using a single model, which is extremely useful in terms of training process and lowering online maintenance costs. It is also helpful for improving low-resource translation. In this article, we propose an MNMT system to address the issues related to low-resource language translation. Our model comprises two MNMT systems, i.e., for English-Indic (one-to-many) and for Indic-English (many-to-one) with a shared encoder-decoder containing 15 language pairs (30 translation directions). Since most of IL pairs have a scanty amount of parallel corpora, not sufficient for training any machine translation model, we explore various augmentation strategies to improve overall translation quality through the proposed model. A state-of-the-art transformer architecture is used to realize the proposed model. In addition, the article addresses the use of language relationships (in terms of dialect, script, etc.), particularly about the role of high-resource languages of the same family in boosting the performance of low-resource languages. Moreover, the experimental results also show the advantage of back-translation and domain adaptation for ILs to enhance the translation quality of both source and target languages. Using all these key approaches, our proposed model emerges to be more efficient than the baseline model in terms of evaluation metrics, i.e., BLEU (BiLingual Evaluation Understudy) score for a set of ILs.

A case study on the evaluation of Korean-English legal translations by generic and custom neural machine translation engines

This paper addresses the quality of Korean-English legal translation outputs generated by three neural machine translation engines, namely NMT1, NMT2 and NMT3. NMT1 and NMT2 are widely available multilingual generic machine translation engines, whereas NMT3 is a custom machine translation engine for legal translation. In both automatic and human evaluations of the three engines’ English translations of Korean statutes, the custom engine outperformed the other two NMTs. BLEU and METEOR scores revealed that NMT3’s output was more similar to human reference translation. Human evaluation results confirmed NMT3’s strong performance, outperforming the generic engines in accuracy and fluency while underperforming in terminology. However, further analysis revealed that NMT3 was superior in handling terms comprising compound nouns and sentences over 25 words. Although the custom engine’s performance still requires substantial human involvement for quality translation, it showed a potential for the legal domain, demonstrating human parity in short sentences. The growing demand for speedy translation under budget constraints calls for further research on technology-based translation tools and human-computer interaction for legal translation.

Improving Many-to-Many Neural Machine Translation via Selective and Aligned Online Data Augmentation

Multilingual neural machine translation (MNMT) models are theoretically attractive for low- and zero-resource language pairs with the impact of cross-lingual knowledge transfer. Existing approaches mainly focus on English-centric directions and always underperform compared to their pivot-based counterparts for non-English directions. In this work, we aim to build a many-to-many MNMT system with an emphasis on the quality of non-English directions by exploring selective and aligned online data augmentation algorithms. Based on our findings showing that the augmented synthetic samples are not “the more, the better” we propose selective online back-translation (SOBT) and thoroughly study different selection criteria to pick suitable samples for training. Furthermore, we boost SOBT with cross-lingual online substitution (CLOS) to align token representations and encourage transfer learning. Our intuition is based on the hypothesis that a universal cross-lingual representation leads to a better multilingual translation performance, especially for non-English directions. Comparing to previous state-of-the-art many-to-many MNMT models and conventional pivot-based methods, experiments on IWSLT2014 and OPUS-100 translation benchmarks show that our approach achieves a competitive or even better performance on English-centric directions and achieves up to ∼12 BLEU for non-English directions. All of our models and codes are publicly available.

Rectifying Ill-Formed Interlingual Space: A Framework for Zero-Shot Translation on Modularized Multilingual NMT

The multilingual neural machine translation (NMT) model can handle translation between more than one language pair. From the perspective of industrial applications, the modularized multilingual NMT model (M2 model) that only shares modules between the same languages is a practical alternative to the model that shares one encoder and one decoder (1-1 model). Previous works have proven that the M2 model can benefit from multiway training without suffering from capacity bottlenecks and exhibits better performance than the 1-1 model. However, the M2 model trained on English-centric data is incapable of zero-shot translation due to the ill-formed interlingual space. In this study, we propose a framework to help the M2 model form an interlingual space for zero-shot translation. Using this framework, we devise an approach that combines multiway training with a denoising autoencoder task and incorporates a Transformer attention bridge module based on the attention mechanism. We experimentally show that the proposed method can form an improved interlingual space in two zero-shot experiments. Our findings further extend the use of the M2 model for multilingual translation in industrial applications.

Parameter Differentiation Based Multilingual Neural Machine Translation

Multilingual neural machine translation (MNMT) aims to translate multiple languages with a single model and has been proved successful thanks to effective knowledge transfer among different languages with shared parameters. However, it is still an open question which parameters should be shared and which ones need to be task-specific. Currently, the common practice is to heuristically design or search language-specific modules, which is difficult to find the optimal configuration. In this paper, we propose a novel parameter differentiation based method that allows the model to determine which parameters should be language-specific during training. Inspired by cellular differentiation, each shared parameter in our method can dynamically differentiate into more specialized types. We further define the differentiation criterion as inter-task gradient similarity. Therefore, parameters with conflicting inter-task gradients are more likely to be language-specific. Extensive experiments on multilingual datasets have demonstrated that our method significantly outperforms various strong baselines with different parameter sharing configurations. Further analysis reveals that the parameter sharing configuration obtained by our method correlates well with the linguistic proximities.

Interpreting Gender Bias in Neural Machine Translation: Multilingual Architecture Matters

Multilingual neural machine translation architectures mainly differ in the number of sharing modules and parameters applied among languages. In this paper, and from an algorithmic perspective, we explore whether the chosen architecture, when trained with the same data, influences the level of gender bias. Experiments conducted in three language pairs show that language-specific encoder-decoders exhibit less bias than the shared architecture. We propose two methods for interpreting and studying gender bias in machine translation based on source embeddings and attention. Our analysis shows that, in the language-specific case, the embeddings encode more gender information, and their attention is more diverted. Both behaviors help in mitigating gender bias.

The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation

Abstract One of the biggest challenges hindering progress in low-resource and multilingual machine translation is the lack of good evaluation benchmarks. Current evaluation benchmarks either lack good coverage of low-resource languages, consider only restricted domains, or are low quality because they are constructed using semi-automatic procedures. In this work, we introduce the Flores-101 evaluation benchmark, consisting of 3001 sentences extracted from English Wikipedia and covering a variety of different topics and domains. These sentences have been translated in 101 languages by professional translators through a carefully controlled process. The resulting dataset enables better assessment of model quality on the long tail of low-resource languages, including the evaluation of many-to-many multilingual translation systems, as all translations are fully aligned. By publicly releasing such a high-quality and high-coverage dataset, we hope to foster progress in the machine translation community and beyond.

Multilingual Machine Translation: Deep Analysis of Language-Specific Encoder-Decoders

State-of-the-art multilingual machine translation relies on a shared encoder-decoder. In this paper, we propose an alternative approach based on language-specific encoder-decoders, which can be easily extended to new languages by learning their corresponding modules. To establish a common interlingua representation, we simultaneously train N initial languages. Our experiments show that the proposed approach improves over the shared encoder-decoder for the initial languages and when adding new languages, without the need to retrain the remaining modules. All in all, our work closes the gap between shared and language-specific encoder-decoders, advancing toward modular multilingual machine translation systems that can be flexibly extended in lifelong learning settings.

Open and Competitive Multilingual Neural Machine Translation in Production

Open and Competitive Multilingual Neural Machine Translation in Production

Synchronous Inference for Multilingual Neural Machine Translation

Multilingual neural machine translation allows a single model to translate between multiple language pairs, which greatly reduces the cost of model training and receives much attention recently. Previous studies mainly focus on training stage optimization and improve positive knowledge transfer among languages with different levels of parameter sharing, but ignore the multilingual knowledge transfer during inference although the translation in one language may help the generation of other languages. This work enhances knowledge sharing among multiple target languages in the inference phase. To achieve this, we propose a synchronous inference method that can simultaneously generate translations in multiple languages. During generation, the model predicts the next word of each language not only based on source sentence and previously predicted segments, but also based on predicted words of other target languages. To maximize the inference stage knowledge sharing, we design a cross-lingual attention module which allows the model to dynamically select the most relevant information from multiple target languages. The synchronous inference model requires multi-way parallel training data which is scarce. We therefore propose to adopt multi-task learning to incorporate large-scale bilingual data. We evaluate our method on three multilingual translation datasets and prove that the proposed method significantly improve the translation quality and the decoding efficiency compared to strong bilingual and multilingual baselines.

A Survey of Orthographic Information in Machine Translation

Machine translation is one of the applications of natural language processing which has been explored in different languages. Recently researchers started paying attention towards machine translation for resource-poor languages and closely related languages. A widespread and underlying problem for these machine translation systems is the linguistic difference and variation in orthographic conventions which causes many issues to traditional approaches. Two languages written in two different orthographies are not easily comparable but orthographic information can also be used to improve the machine translation system. This article offers a survey of research regarding orthography’s influence on machine translation of under-resourced languages. It introduces under-resourced languages in terms of machine translation and how orthographic information can be utilised to improve machine translation. We describe previous work in this area, discussing what underlying assumptions were made, and showing how orthographic knowledge improves the performance of machine translation of under-resourced languages. We discuss different types of machine translation and demonstrate a recent trend that seeks to link orthographic information with well-established machine translation methods. Considerable attention is given to current efforts using cognate information at different levels of machine translation and the lessons that can be drawn from this. Additionally, multilingual neural machine translation of closely related languages is given a particular focus in this survey. This article ends with a discussion of the way forward in machine translation with orthographic information, focusing on multilingual settings and bilingual lexicon induction.

Synchronous Interactive Decoding for Multilingual Neural Machine Translation

To simultaneously translate a source language into multiple different target languages is one of the most common scenarios of multilingual translation. However, existing methods cannot make full use of translation model information during decoding, such as intra-lingual and inter-lingual future information, and therefore may suffer from some issues like the unbalanced outputs. In this paper, we present a new approach for synchronous interactive multilingual neural machine translation (SimNMT), which predicts each target language output simultaneously and interactively using historical and future information of all target languages. Specifically, we first propose a synchronous cross-interactive decoder in which generation of each target output does not only depend on its generated sequences, but also relies on its future information, as well as history and future contexts of other target languages. Then, we present a new interactive multilingual beam search algorithm that enables synchronous interactive decoding of all target languages in a single model. We take two target languages as an example to illustrate and evaluate the proposed SimNMT model on IWSLT datasets. The experimental results demonstrate that our method achieves significant improvements over several advanced NMT and MNMT models.