Training Neural Machine Translation Research Articles

An analysis of Google Translate and DeepL translation of source text typographical errors in the economic and legal fields

Training neural machine translation systems with noisy data has been shown to improve robustness (Heigold et al., 2018). The objective of the present study is to test Google Translate and DeepL performance in the detection and correction of typographical errors, by introducing 1,820 source text typos found in a previous work on specialised Spanish-English dictionaries (Rodríguez-Rubio & Fernández-Quesada, 2020a, 2020b; Rodríguez-Rubio Mediavilla, 2021). Typos were introduced in isolation and also in co-text. Results showed that Google Translate clearly outperformed DeepL. Moreover, the repetition of the source typo was found to be the most frequent phenomenon in the machine translation output of both systems. By shedding light on the capacity of systems to deal with source text typographical errors, our study aims to provide a starting point for their refinement.

Mutation prediction in the SARS-CoV-2 genome using attention-based neural machine translation.

Severe acute respiratory syndrome coronavirus 2 (SARS‑CoV‑2) has been evolving rapidly after causing havoc worldwide in 2020. Since then, it has been very hard to contain the virus owing to its frequently mutating nature. Changes in its genome lead to viral evolution, rendering it more resistant to existing vaccines and drugs. Predicting viral mutations beforehand will help in gearing up against more infectious and virulent versions of the virus in turn decreasing the damage caused by them. In this paper, we have proposed different NMT (neural machine translation) architectures based on RNNs (recurrent neural networks) to predict mutations in the SARS-CoV-2-selected non-structural proteins (NSP), i.e., NSP1, NSP3, NSP5, NSP8, NSP9, NSP13, and NSP15. First, we created and pre-processed the pairs of sequences from two languages using k-means clustering and nearest neighbors for training a neural translation machine. We also provided insights for training NMTs on long biological sequences. In addition, we evaluated and benchmarked our models to demonstrate their efficiency and reliability.

North Korean Neural Machine Translation through South Korean Resources

South and North Korea both use the Korean language. However, Korean natural language processing (NLP) research has mostly focused on South Korean language. Therefore, existing NLP systems in the Korean language, such as neural machine translation (NMT) systems, cannot properly process North Korean inputs. Training a model using North Korean data is the most straightforward approach to solving this problem, but the data to train NMT models are insufficient. To solve this problem, we constructed a parallel corpus to develop a North Korean NMT model using a comparable corpus. We manually aligned parallel sentences to create evaluation data and automatically aligned the remaining sentences to create training data. We trained a North Korean NMT model using our North Korean parallel data and improved North Korean translation quality using South Korean resources such as parallel data and a pre-trained model. In addition, we propose Korean-specific pre-processing methods, character tokenization, and phoneme decomposition to use the South Korean resources more efficiently. We demonstrate that the phoneme decomposition consistently improves the North Korean translation accuracy compared to other pre-processing methods.

Alien coding

We introduce a self-learning algorithm for synthesizing programs that provide explanations for OEIS sequences. The algorithm starts from scratch initially generating programs at random. Then it runs many iterations of a self-learning loop that interleaves (i) training neural machine translation to learn the correspondence between sequences and the programs discovered so far, and (ii) proposing many new programs for each OEIS sequence by the trained neural machine translator. The algorithm discovers on its own programs for more than 78000 OEIS sequences, sometimes developing unusual programming methods. We analyze its behavior and the invented programs in several experiments.

WCC-JC 2.0: A Web-Crawled and Manually Aligned Parallel Corpus for Japanese-Chinese Neural Machine Translation

Movie and TV subtitles are frequently employed in natural language processing (NLP) applications, but there are limited Japanese-Chinese bilingual corpora accessible as a dataset to train neural machine translation (NMT) models. In our previous study, we effectively constructed a corpus of a considerable size containing bilingual text data in both Japanese and Chinese by collecting subtitle text data from websites that host movies and television series. The unsatisfactory translation performance of the initial corpus, Web-Crawled Corpus of Japanese and Chinese (WCC-JC 1.0), was predominantly caused by the limited number of sentence pairs. To address this shortcoming, we thoroughly analyzed the issues associated with the construction of WCC-JC 1.0 and constructed the WCC-JC 2.0 corpus by first collecting subtitle data from movie and TV series websites. Then, we manually aligned a large number of high-quality sentence pairs. Our efforts resulted in a new corpus that includes about 1.4 million sentence pairs, an 87% increase compared with WCC-JC 1.0. As a result, WCC-JC 2.0 is now among the largest publicly available Japanese-Chinese bilingual corpora in the world. To assess the performance of WCC-JC 2.0, we calculated the BLEU scores relative to other comparative corpora and performed manual evaluations of the translation results generated by translation models trained on WCC-JC 2.0. We provide WCC-JC 2.0 as a free download for research purposes only.

DetTrans: A Lightweight Framework to Detect and Translate Noisy Inputs Simultaneously

Neural machine translation (NMT) systems trained on clean data usually suffer from performance degradation when translating noisy inputs. Existing works attempt to improve the robustness of NMT normally via data augmentation, where synthetic noisy data are mixed with original clean data, either for training NMT with the standard NMT loss alone, or for tuning auxiliary tasks in a multi-task learning manner. Typical auxiliary tasks include detecting and correcting noises, exploiting noisy outputs for contrastive learning etc. The aforementioned two auxiliary tasks are generally designed independently, and the modules for detecting and correcting noises are heavyweight. In this article, we propose a new framework, DetTransNet ( <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Det</b> ector- <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Trans</b> lator <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Net</b> work), aiming to detect positions of noises in the input and translate the input simultaneously. The newly introduced noise detector module is essentially a lightweight binary classifier built upon the final layer of the encoder of the original Transformer model for the translation task, which is to identify at which position of the input has potential noise. The module has a very few parameters. In order to help the model capture the relationship between clean instances and their noisy counterparts, an extra loss is further introduced to enhance the interaction between clean and noisy data. In this way, we combine noise detection and contrastive learning together. As the model is able to identify and locate noises, a heuristic method is proposed to correct detected noises, in order to achieve better translations. Experiments show that DetTransNet is robust to four types of noises (deletion, insertion, swapping, keyboard), and obtain a substantial improvement of up to 1.6 BLEU points across different datasets.

루간다어-영어 병렬코퍼스 구축과 번역 모델 학습

Neural machine translation (NMT) has achieved great successes with large datasets, so NMT is more premised on high-resource languages. This continuously underpins the low resource languages such as Luganda due to the lack of high-quality parallel corpora, so even 'Google translate' does not serve Luganda at the time of this writing. In this paper, we build a parallel corpus with 41,070 pairwise sentences for Luganda and English which is based on three different open-sourced corpora. Then, we train NMT models with hyper-parameter search on the dataset. Experiments gave us a BLEU score of 21.28 from Luganda to English and 17.47 from English to Luganda. Some translation examples show high quality of the translation. We believe that our model is the first Luganda-English NMT model. The bilingual dataset we built will be available to the public.

Research on Collaborative Machine English Translation Using the HIC Technology

Due to the rapid development of smart city, Hybrid Information Centric-Networking (HICN) emerges as a promising technology to enable the power of smart city. One of the most important application is the smart English translation, which becomes more and more popular with the process of Internationalization. In this work, we focus on studying the intelligent English translation is smart city using the HICN technology. Particularly, a method using collaborative machine learning and quality estimation technique is proposed, which sets a fixed threshold to filter pseudo-parallel data during unsupervised neural machine translation training. The quality estimation is used to evaluate and screen the pseudo-parallel data with high performance generated during reverse translation training. The results indicate that the proposed method outperforms the state-of-the-art methods.

Moment matching training for neural machine translation: An empirical study

In recent years, Neural Machine Translation (NMT), which harnesses the power of neural networks, has achieved astonishing achievements. Despite its promise, NMT models can still not model prior external knowledge. Recent investigations have necessitated the adaptation of past expertise to both training and inference methods, resulting in translation inference issues. This paper proposes an extension of the moment matching framework that incorporates advanced prior knowledge without interfering with the inference process by using a matching mechanism between the model and empirical distributions. Our tests show that the suggested expansion outperforms the baseline and effectively over various language combinations.

Addressing domain shift in neural machine translation via reinforcement learning

Domain Adaptation (DA) has been a well-known transfer learning algorithm used in Neural Machine Translation (NMT) task. Adding domain-related corpora in training data to train the model boosts the performance of NMT. Many low resource language pairs such as Hindi–Nepali and Spanish–Portuguese, for which sufficient parallel data are not available to train the model, face in-domain data scarcity issue. Non-availability of in-domain training data leads to the use of unrelated corpus in training the model, which degrades the NMT performance. This domain mismatch between training and test data results in a domain shift problem. To tackle the challenge of the domain shift problem in NMT, we propose REINFORCE-based Sentence Selection and Weighting (RSSW) method, which selects pseudo in-domain sentences from out-of-domain data and learns their weights based on Reinforcement Learning. The proposed method leverages the similarity between language pairs by encoding the source and target languages into a common encoding script for language model training. RSSW uses minimum risk training and maximum likelihood estimation as an objective function to train NMT on selected pseudo in-domain sentences. Furthermore, we employ multi-domain and fine-tuning approaches to compare proposed method on Hindi↔Nepali and Hindi↔Marathi language pairs through extensive experimental analysis. From the experimental results, we find that the proposed method outperforms the state-of-the-art and baseline approaches by ∼2 BLEU points in all translation directions of language pairs.

Improving neural machine translation for low-resource Indian languages using rule-based feature extraction

Languages help to unite the world socially, culturally and technologically. Different natives communicate in different languages; there is a tremendous requirement for inter-language information translation process to transfer and share information and ideas. Though Sanskrit is an ancient Indo-European language, a significant amount of work for processing the information is required to explore the full potential of this language to open vistas in computational linguistics and computer science domain. In this paper, we have proposed and presented the machine translation system for translating Sanskrit to the Hindi language. The developed technique uses linguistic features from rule-based feed to train neural machine translation system. The work is novel and applicable to any low-resource language with rich morphology. It is a generic system covering various domains with minimal human intervention. The performance analysis of work is performed on automatic and linguistic measures. The results show that proposed and developed approach outperforms earlier work for this language pair.

Iterative Training of Unsupervised Neural and Statistical Machine Translation Systems

Recent work achieved remarkable results in training neural machine translation (NMT) systems in a fully unsupervised way, with new and dedicated architectures that only rely on monolingual corpora. However, previous work also showed that unsupervised statistical machine translation (USMT) performs better than unsupervised NMT (UNMT), especially for distant language pairs. To take advantage of the superiority of USMT over UNMT, and considering that SMT suffers from well-known limitations overcome by NMT, we propose to define UNMT as NMT trained with the supervision of synthetic parallel data generated by USMT. This way we can exploit USMT up to its limits while ultimately relying on full-fledged NMT models to generate translations. We show significant improvements in translation quality over previous work and also that further improvements can be obtained by alternatively and iteratively training USMT and UNMT. Without the need of a dedicated architecture for UNMT, our simple approach can straightforwardly benefit from any recent and future advances in supervised NMT. Our systems achieve a new state-of-the-art for unsupervised machine translation in all of our six translation tasks for five diverse language pairs, surpassing even supervised SMT or NMT in some tasks. Furthermore, our analysis shows how crucial the comparability between the monolingual corpora used for unsupervised training is in improving translation quality.

Towards Automatic Subtitling: Assessing the Quality of Old and New Resources

Growing needs in localising multimedia content for global audiences have resulted in Neural Machine Translation (NMT) gradually becoming an established practice in the field of subtitling in order to reduce costs and turn-around times. Contrary to text translation, subtitling is subject to spatial and temporal constraints, which greatly increase the post-processing effort required to restore the NMT output to a proper subtitle format. In our previous work (Karakanta, Negri, and Turchi 2019), we identified several missing elements in the corpora available for training NMT systems specifically tailored for subtitling. In this work, we compare the previously studied corpora with MuST-Cinema, a corpus enabling end-to-end speech to subtitles translation, in terms of the conformity to the constraints of: 1) length and reading speed; and 2) proper line breaks. We show that MuST-Cinema conforms to these constraints and discuss the recent progress the corpus has facilitated in end-to-end speech to subtitles translation.

Acquiring Knowledge from Pre-Trained Model to Neural Machine Translation

Pre-training and fine-tuning have achieved great success in natural language process field. The standard paradigm of exploiting them includes two steps: first, pre-training a model, e.g. BERT, with a large scale unlabeled monolingual data. Then, fine-tuning the pre-trained model with labeled data from downstream tasks. However, in neural machine translation (NMT), we address the problem that the training objective of the bilingual task is far different from the monolingual pre-trained model. This gap leads that only using fine-tuning in NMT can not fully utilize prior language knowledge. In this paper, we propose an Apt framework for acquiring knowledge from pre-trained model to NMT. The proposed approach includes two modules: 1). a dynamic fusion mechanism to fuse task-specific features adapted from general knowledge into NMT network, 2). a knowledge distillation paradigm to learn language knowledge continuously during the NMT training process. The proposed approach could integrate suitable knowledge from pre-trained models to improve the NMT. Experimental results on WMT English to German, German to English and Chinese to English machine translation tasks show that our model outperforms strong baselines and the fine-tuning counterparts.

Automatic translation of spoken English based on improved machine learning algorithm

Due to the complexity of English machine translation technology and its broad application prospects, many experts and scholars have invested more energy to analyze it. In view of the complex and changeable English forms, the large difference between Chinese and English word order, and insufficient Chinese-English parallel corpus resources, this paper uses deep learning to complete the conversion between Chinese and English. The research focus of this paper is how to use language pairs with rich parallel corpus resources to improve the performance of Chinese-English neural machine translation, that is, to use multi-task learning to train neural machine translation models. Moreover, this research proposes a low-resource neural machine translation method based on weight sharing, which uses the weight-sharing method to improve the performance of Chinese-English low-resource neural machine translation. In addition, this study designs a control experiment to analyze the effectiveness of this study model. The research results show that the model proposed in this paper has a certain effect.

Speed Up the Training of Neural Machine Translation

Neural machine translation (NMT) has achieved notable achievements in recent years. Although existing models provide reasonable translation performance, they cost too much training time. Especially, when the corpus is enormous, their computational cost will be extremely high. In this paper, we propose a novel NMT model based on the conventional bidirectional recurrent neural network (bi-RNN). In this model, we apply a tanh activation function, which can learn the future and history context information more sufficiently, to speed up the training process. Experimental results on tasks of German–English and English–French translation demonstrate that the proposed model can save much training time compared with the state-of-the-art models and provide better translation performances.

Regularizing Neural Machine Translation by Target-Bidirectional Agreement

Although Neural Machine Translation (NMT) has achieved remarkable progress in the past several years, most NMT systems still suffer from a fundamental shortcoming as in other sequence generation tasks: errors made early in generation process are fed as inputs to the model and can be quickly amplified, harming subsequent sequence generation. To address this issue, we propose a novel model regularization method for NMT training, which aims to improve the agreement between translations generated by left-to-right (L2R) and right-to-left (R2L) NMT decoders. This goal is achieved by introducing two Kullback-Leibler divergence regularization terms into the NMT training objective to reduce the mismatch between output probabilities of L2R and R2L models. In addition, we also employ a joint training strategy to allow L2R and R2L models to improve each other in an interactive update process. Experimental results show that our proposed method significantly outperforms state-of-the-art baselines on Chinese-English and English-German translation tasks.

Effectively training neural machine translation models with monolingual data

Improving neural machine translation models (NMT) with monolingual data has aroused more and more interests in this area and back-translation for monolingual data augmentation Sennrich et al. (2016) has been taken as a promising development recently. While the naive back-translation approach improves the translation performance substantially, we notice that its usage for monolingual data is not so effective because traditional NMT models make no distinction between the true parallel corpus and the back translated synthetic parallel corpus. This paper proposes a gate-enhanced NMT model which makes use of monolingual data more effectively. The central idea is to separate the data flow of monolingual data and parallel data into different channels by the elegant designed gate, which enables the model to perform different transformations according to the type of the input sequence, i.e., monolingual data and parallel data. Experiments on Chinese-English and English-German translation tasks show that our approach achieves substantial improvements over strong baselines and the gate-enhanced NMT model can utilize the source-side and target-side monolingual data at the same time.

Sentence Selection and Weighting for Neural Machine Translation Domain Adaptation

Neural machine translation (NMT) has been prominent in many machine translation tasks. However, in some domain-specific tasks, only the corpora from similar domains can improve translation performance. If out-of-domain corpora are directly added into the in-domain corpus, the translation performance may even degrade. Therefore, domain adaptation techniques are essential to solve the NMT domain problem. Most existing methods for domain adaptation are designed for the conventional phrase-based machine translation. For NMT domain adaptation, there have been only a few studies on topics such as fine tuning, domain tags, and domain features. In this paper, we have four goals for sentence level NMT domain adaptation. First, the NMT's internal sentence embedding is exploited and the sentence embedding similarity is used to select out-of-domain sentences that are close to the in-domain corpus. Second, we propose three sentence weighting methods, i.e., sentence weighting, domain weighting, and batch weighting, to balance the data distribution during NMT training. Third, in addition, we propose dynamic training methods to adjust the sentence selection and weighting during NMT training. Fourth, to solve the multidomain problem in a real-world NMT scenario where the domain distributions of training and testing data often mismatch, we proposed a multidomain sentence weighting method to balance the domain distributions of training data and match the domain distributions of training and testing data. The proposed methods are evaluated in international workshop on spoken language translation (IWSLT) English-to-French/German tasks and a multidomain English-to-French task. Empirical results show that the sentence selection and weighting methods can significantly improve the NMT performance, outperforming the existing baselines.

Generative adversarial training for neural machine translation

Abstract Neural machine translation (NMT) is typically optimized to generate sentences which cover n-grams with ground target as much as possible. However, it is widely acknowledged that n-gram precisions, the manually designed approximate loss function, may mislead the model to generate suboptimal translations. To solve this problem, we train the NMT model to generate human-like translations directly by using the generative adversarial net, which has achieved great success in computer vision. In this paper, we build a conditional sequence generative adversarial net (CSGAN-NMT) which comprises of two adversarial sub models, a generative model (generator) which translates the source sentence into the target sentence as the traditional NMT models do and a discriminative model (discriminator) which discriminates the machine-translated target sentence from the human-translated one. The two sub models play a minimax game and achieve a win-win situation when reaching a Nash Equilibrium. As a variant of the single generator-discriminator model, the multi-CSGAN-NMT which contains multiple discriminators and generators, is also proposed. In the multi-CSGAN-NMT model, each generator is viewed as an agent which can interact with others and even transfer messages. Experiments show that the proposed CSGAN-NMT model obtains substantial improvements than the strong baseline and the improvement of the multi-CSGAN-NMT model is more remarkable.