Neural Machine Translation Models Research Articles

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

The development of deep learning techniques has allowed Neural Machine Translation (NMT) models to become extremely powerful, given sufficient training data and training time. However, systems struggle when translating text from a new domain with a distinct style or vocabulary. Fine-tuning on in-domain data allows good domain adaptation, but requires sufficient relevant bilingual data. Even if this is available, simple fine-tuning can cause overfitting to new data and catastrophic forgetting of previously learned behaviour. We survey approaches to domain adaptation for NMT, particularly where a system may need to translate across multiple domains. We divide techniques into those revolving around data selection or generation, model architecture, parameter adaptation procedure, and inference procedure. We finally highlight the benefits of domain adaptation and multidomain adaptation techniques to other lines of NMT research.

English-Chinese Machine Translation Based on Transfer Learning and Chinese-English Corpus.

This paper proposes an English-Chinese machine translation research method based on transfer learning. First, it expounds the theory of neural machine translation and transfer learning and related technologies. Neural machine translation is discussed, the advantages and disadvantages of various models are introduced, and the transformer neural machine translation model framework is selected. For low-resource Chinese-English parallel corpus and Tibetan-Chinese parallel corpus, 30 million Chinese-English parallel corpora, 100,000 Chinese-English low-resource parallel corpora, and 100,000 Tibetan-Chinese parallel corpora were used to pretrain the transformer machine translation architecture. The decoders are all composed of 6 identical hidden layers, the initialization of the model parameters is done by the transformer uniform distribution, and the model training uses Adam as the optimizer. In the model transfer part, the parameters with the better effect of the pretrained model are transferred to the low-resource Chinese-English and Tibetan-Chinese machine translation model training, so as to achieve the purpose of knowledge transfer. The results show that the model transfer learning of low-resource Chinese-English parallel corpus improves the translation system's translation by 3.97 BLEU values compared with the translation system without transfer learning at 0.34 BLEU values. Model transfer learning on low-resource Tibetan-Chinese parallel corpus increases the BLEU value by 2.64 BLEU compared to the translation system without transfer learning. The neural machine translation system that uses BPE technology for preprocessing plus model transfer learning is compared to the translation system that only performs transfer learning and shows an improved 0.26 BLEU value.It is verified that the transfer learning method proposed in this paper has a certain improvement in the effect of low-resource Chinese-English and Tibetan-Chinese neural machine translation models.

Creating a list of word alignments from parallel Russian simplification data.

This work describes the development of a list of monolingual word alignments taken from parallel Russian simplification data. This word lists can be used in such lexical simplification tasks as rule-based simplification applications and lexically constrained decoding for neural machine translation models. Moreover, they constitute a valuable source of information for developing educational materials for teaching Russian as a second/foreign language. In this work, a word list was compiled automatically and post-edited by human experts. The resulting list contains 1409 word pairs in which each “complex” word has an equivalent “simpler” (shorter, more frequent, modern, international) synonym. We studied the contents of the word list by comparing the frequencies of the words in the pairs and their levels in the special CEFR-graded vocabulary lists for learners of Russian as a foreign language. The evaluation demonstrated that lexical simplification by means of single-word synonym replacement does not occur often in the adapted texts. The resulting list also illustrates the peculiarities of the lexical simplification task for L2 learners, such as the choice of a less frequent but international word.

Research on the Analysis of Correlation Factors of English Translation Ability Improvement Based on Deep Neural Network.

This paper adopts the algorithm of the deep neural network to conduct in-depth research and analysis on the factors associated with the improvement of English translation ability. This study focuses on text complexity, adding discourse complexity features in addition to focusing on lexical and syntactic dimensions, exploring the application of neural network algorithm in the construction of text complexity grading model based on feature optimization, and examining the performance and generalization ability of the model. The rationality of the grading of the material is verified. After determining the model input features and training corpus, different classification algorithms were used to build the models and compare their performance. Meanwhile, compared with the models constructed based on common traditional readability formulas and other single-dimensional features, the models constructed based on the feature set of this study have significant advantages, with 20 to 30 percentage points higher in each performance evaluation index. The pseudo-parallel corpus is constructed, back translation is performed after obtaining the pseudo-parallel corpus, and finally, the data migration effect is measured and recorded on the low-resource Chinese-English parallel corpus and Tibetan-Chinese parallel corpus, and the cycle continues until the model performance is no longer improved. The low-resource neural machine translation model based on model migration learning improved 3.97 and 2.64 BLEU values in the low-resource English translation task, respectively, and reduced the training time; based on this, the data migration learning method further improved 2.26 and 2.52 BLEU values.

Uncertainty-aware non-autoregressive neural machine translation

Most existing non-autoregressive neural machine translation (NAT) models generally employ the posterior probability to indicate the model confidence during training, which seems to lag behind the novel uncertainty estimations (UEs) methods successfully deployed in other natural language processing (NLP) tasks. Previous research has practically ignored the large-scale exploration of UE methods in the NAT problem. In this paper, we propose a strategy based on Active Learning employed to investigate whether these sophisticated uncertainty-aware methods are more effective in the NAT problem. Besides, we provide an in-depth analysis of the impact of different widely employed UE methods and propose several tailored ones. In the end, we incorporate these exceptional ones into the practical one-pass GLAT model to obtain enhanced performance. Experimental results demonstrate that sophisticated uncertainty-aware UE methods with the two-step training paradigm are potentially superior to represent the model confidence in facilitating token-level decision-making compared to the posterior probability in NAT to a certain extent.

Research on the Construction of a Bidirectional Neural Network Machine Translation Model Fused with Attention Mechanism

With the development of deep learning, neural machine translation has also been paid attention and developed by researchers. Especially in the application of encoder-decoder in natural language processing, the translation performance has been significantly improved. In 2014, the attention mechanism was used in neural machine translation, the performance of translation was greatly improved, and the interpretability of the model was increased. This research proposes a research idea of sparsemax combined with AAN machine translation model and conducts multiple ablation experiments for experimental verification. This chapter first studies the problem of insufficient sparse normalization when generating target words in the attention mechanism and studies the neural machine translation model incorporating the sparse normalization calculation method. It solves the problem of inductive bias in the data transfer process of related sub-layers in the model. By combining the strategy of sparse normalization, the similarity value of related word vectors can be obtained more accurately when aligning words, which is more convenient for this chapter. Calculate and analyze the specific principles of the model. In addition, when the model faces a large vocabulary in the decoding stage, too many weights of scattered vocabulary vectors are not conducive to the generation of correct target values. After using the sparse normalization strategy, it can reduce the number of inconveniences. The calculation between related words optimizes the classification accuracy of the target vocabulary. In this chapter, aiming at the waste of the transformer’s decoder calculation in the inference stage, the average attention structure is used to replace the attention calculation layer of the first layer of the decoder part of the original model. Each moment is only related to the previous moment, which alleviates the waste of computing resources.

A Study on Chinese-English Machine Translation Based on Migration Learning and Neural Networks

This paper constructs a neural machine translation model for Chinese and English using encoder feature extraction with migration learning, allowing the machine to automatically perform feature migration learning, transforming Chinese and English data into word vectors using a centralized representation, and using neural networks to achieve a direct mapping between Chinese and English languages. In this paper, we study the construction of a neural machine translation model by the neural network structure. Firstly, the neural machine translation model is constructed using the neural network structure, which is simplified based on the neural network structure and achieves good performance while reducing the training complexity. To address the problem that neural machine translation cannot make good use of linguistic knowledge, a neural machine translation model with lexical sequence information is proposed. Based on the translation model of a bidirectional neural network incorporating attention mechanism, syntactic analysis is performed to obtain lexical sequence information, and then it is incorporated into the encoder part of the translation model in the form of bidirectional encoding, and the background vector is formed together by vector splicing. To address the problem that the Chinese-English language sequences of any length in the encoder part are encoded into a fixed dimensional background vector, an attention mechanism is introduced to dynamically adjust the degree of influence of the context at the source language end on the target language sequences to improve the translation model’s ability to deal with long-distance dependencies. To better reflect the contextual information, this paper further proposes a machine translation model based on migration learning and neural network and conducts a comparative analysis for various translation models to verify the effectiveness of the model performance improvement.

Improving neural machine translation with POS-tag features for low-resource language pairs

Integrating linguistic features has been widely utilized in statistical machine translation (SMT) systems, resulting in improved translation quality. However, for low-resource languages such as Thai and Myanmar, the integration of linguistic features in neural machine translation (NMT) systems has yet to be implemented. In this study, we propose transformer-based NMT models (transformer, multi-source transformer, and shared-multi-source transformer models) using linguistic features for two-way translation of Thai-to-Myanmar, Myanmar-to-English, and Thai-to-English. Linguistic features such as part-of-speech (POS) tags or universal part-of-speech (UPOS) tags are added to each word on either the source or target side, or both the source and target sides, and the proposed models are conducted. The multi-source transformer and shared-multi-source transformer models take two inputs (i.e., string data and string data with POS tags) and produce string data or string data with POS tags. A transformer model that utilizes only word vectors was used as the first baseline model for comparison with the proposed models. The second baseline model, an Edit-Based Transformer with Repositioning (EDITOR) model, was also used to compare with our proposed models in addition to the baseline transformer model. The findings of the experiments show that adding linguistic features to the transformer-based models enhances the performance of a neural machine translation in low-resource language pairs. Moreover, the best translation results were yielded using shared-multi-source transformer models with linguistic features resulting in more significant Bilingual Evaluation Understudy (BLEU) scores and character n-gram F-score (chrF) scores than the baseline transformer and EDITOR models.

Neural Machine Translation

Abstract: The project's novelty is not merely importing modules and preparing data and feeding the data to the model but understanding how the real language translation works and implementing the logics underlying each method utilized and creating every function from scratch, resulting in the creationof a Neural Machine Translation model. Initially, translation was accomplished by simply substituting words from one language for those from another. However, because languages are essentially different, a greater degree of knowledge (e.g., phrases/sentences) is required to achieveeffective results. With the introduction of deep learning, modern software now employs statisticaland neural techniques that have been shown to be more effective when translating. We are essentially translating German to English utilizing Sequence to Sequence models with attention and transformer models. Of course, everyone has access to Google Translates power, but if you want to learn how to implement translation in code, this project will show you how. We are writingour code from scratch, without using any libraries,in order to understand how each model works.While this design is a little out of date, it is still a great project to work on if you want to learn more about attention processes before moving on to Transformers. It is based on Effective Approaches to Attentionbased Neural Machine Translator and is a sequence to sequence (seq2seq) model for German to English translation.

SPARQL Generation with an NMT-based Approach

SPARQL is a powerful query language which has been widely used in various natural language question answering (QA) systems. As the advances of deep neural networks, Neural Machine Translation (NMT) models are employed to directly translate natural language questions to SPARQL queries in recent years. In this paper, we propose an NMT-based approach with Transformer model to generate SPARQL queries. Transformer model is chosen due to its relatively high efficiency and effectiveness. We design a format to encode a SPARQL query into a simple sequence with only RDF triples reserved. The main purpose of this step is to shorten the sequences and reduce the complexity of the target language. Moreover, we employ entity type tags to further resolve mistranslated problems. The proposed approach is evaluated against three open-domain question answering datasets (QALD-7, QALD-8, and LC-QuAD) on BLEU score and accuracy, and obtains outstanding results (83.49%, 90.13%, and 76.32% on BLEU score, respectively) which considerably outperform all known studies.

Self-feeding training method for semi-supervised grammatical error correction

Grammatical error correction (GEC) has been successful with deep and complex neural machine translation models, but the annotated data to train the model are scarce. We propose a novel self-feeding training method that generates incorrect sentences from freely available correct sentences. The proposed training method can generate appropriate wrong sentences from unlabeled sentences, using a data generation model trained as an autoencoder. It can also add artificial noise to correct sentences to automatically generate incorrect sentences. We show that the GEC models trained with the self-feeding training method are successful without extra annotated data or deeper neural network-based models, achieving F0.5 score of 0.5982 on the CoNLL-2014 Shared Task test data with a transformer model. The results also show that fully unlabeled training is possible for data-scarce domains and languages.

Moment matching training for neural machine translation: An empirical study

In recent years, Neural Machine Translation (NMT), which harnesses the power of neural networks, has achieved astonishing achievements. Despite its promise, NMT models can still not model prior external knowledge. Recent investigations have necessitated the adaptation of past expertise to both training and inference methods, resulting in translation inference issues. This paper proposes an extension of the moment matching framework that incorporates advanced prior knowledge without interfering with the inference process by using a matching mechanism between the model and empirical distributions. Our tests show that the suggested expansion outperforms the baseline and effectively over various language combinations.

Is the Corpus Ready for Machine Translation? A Case Study with Python to Pseudo-Code Corpus.

The availability of data is the driving force behind most of the state-of-the-art techniques for machine translation tasks. Understandably, this availability of data motivates researchers to propose new techniques and claim about the superiority of their techniques over the existing ones by using suitable evaluation measures. However, the performance of underlying learning algorithms can be greatly influenced by the correctness and the consistency of the corpus. We present our investigations for the relevance of a publicly available python to pseudo-code parallel corpus for automated documentation task, and the studies performed using this corpus. We found that the corpus had many visible issues like overlapping of instances, inconsistency in translation styles, incompleteness, and misspelled words. We show that these discrepancies can significantly influence the performance of the learning algorithms to the extent that they could have caused previous studies to draw incorrect conclusions. We performed our experimental study using statistical machine translation and neural machine translation models. We have recorded a significant difference (sim 10% on BLEU score) in the models’ performance after removing the issues from the corpus.

Analysis of English Translation Model Based on Artificial Intelligence Attention Mechanism

The particularity and quantity of English translation terms have a great impact on the quality and effect of machine translation and can not meet the requirements of English translation of terms. At the same time, technical exchange and communication in different fields need the expression of professional terms. In addition, although the neural machine translation model has good translation performance, it is not ideal for target languages with small translation needs and limited corpus resources. In order to solve the problems existing in the English translation model, this paper constructs the transformer model by replacing the cyclic neural network variables and introducing the attention mechanism. The term information is integrated into two pretraining models to improve the learning ability of model language sentence relationship. Maintain the integrity of terminology information by fully completing the training. The experimental results show that, compared with the other three term translation models, the translation model in this paper has the advantage of term information. At the same time, the deep neural network English term translation model can obtain more fine-grained word relevance. In different corpora, the Bleu score of the model is good, showing obvious translation performance advantages. This study provides a good professional reference value for the translation of English terms.

From Fully Trained to Fully Random Embeddings: Improving Neural Machine Translation with Compact Word Embedding Tables

Embedding matrices are key components in neural natural language processing (NLP) models that are responsible to provide numerical representations of input tokens (i.e. words or subwords). In this paper, we analyze the impact and utility of such matrices in the context of neural machine translation (NMT). We show that detracting syntactic and semantic information from word embeddings and running NMT systems with random embeddings is not as damaging as it initially sounds. We also show how incorporating only a limited amount of task-specific knowledge from fully-trained embeddings can boost the performance NMT systems. Our findings demonstrate that in exchange for negligible deterioration in performance, any NMT model can be run with partially random embeddings. Working with such structures means a minimal memory requirement as there is no longer need to store large embedding tables, which is a significant gain in industrial and on-device settings. We evaluated our embeddings in translating English into German and French and achieved a 5.3x compression rate. Despite having a considerably smaller architecture, our models in some cases are even able to outperform state-of-the-art baselines.

Non-autoregressive Translation with Layer-Wise Prediction and Deep Supervision

How do we perform efficient inference while retaining high translation quality? Existing neural machine translation models, such as Transformer, achieve high performance, but they decode words one by one, which is inefficient. Recent non-autoregressive translation models speed up the inference, but their quality is still inferior. In this work, we propose DSLP, a highly efficient and high-performance model for machine translation. The key insight is to train a non-autoregressive Transformer with Deep Supervision and feed additional Layer-wise Predictions. We conducted extensive experiments on four translation tasks (both directions of WMT'14 EN-DE and WMT'16 EN-RO). Results show that our approach consistently improves the BLEU scores compared with respective base models. Specifically, our best variant outperforms the autoregressive model on three translation tasks, while being 14.8 times more efficient in inference.

Non-parametric Online Learning from Human Feedback for Neural Machine Translation

We study the problem of online learning with human feedback in the human-in-the-loop machine translation, in which the human translators revise the machine-generated translations and then the corrected translations are used to improve the neural machine translation (NMT) system. However, previous methods require online model updating or additional translation memory networks to achieve high-quality performance, making them inflexible and inefficient in practice. In this paper, we propose a novel non-parametric online learning method without changing the model structure. This approach introduces two k-nearest-neighbor (KNN) modules: one module memorizes the human feedback, which is the correct sentences provided by human translators, while the other balances the usage of the history human feedback and original NMT models adaptively. Experiments conducted on EMEA and JRC-Acquis benchmarks demonstrate that our proposed method obtains substantial improvements on translation accuracy and achieves better adaptation performance with less repeating human correction operations.

Deep Fusing Pre-trained Models into Neural Machine Translation

Pre-training and fine-tuning have become the de facto paradigm in many natural language processing (NLP) tasks. However, compared to other NLP tasks, neural machine translation (NMT) aims to generate target language sentences through the contextual representation from the source language counterparts. This characteristic means the optimization objective of NMT is far from that of the universal pre-trained models (PTMs), leading to the standard procedure of pre-training and fine-tuning does not work well in NMT. In this paper, we propose a novel framework to deep fuse the pre-trained representation into NMT, fully exploring the potential of PTMs in NMT. Specifically, we directly replace the randomly initialized Transformer encoder with a pre-trained encoder and propose a layer-wise coordination structure to coordinate PTM and NMT decoder learning. Then, we introduce a partitioned multi-task learning method to fine-tune the pre-trained parameter, reducing the gap between PTM and NMT by progressively learning the task-specific representation. Experimental results show that our approach achieves considerable improvements on WMT14 En2De, WMT14 En2Fr, and WMT16 Ro2En translation benchmarks and outperforms previous work in both autoregressive and non-autoregressive NMT models.

An Improved Transformer-Based Neural Machine Translation Strategy: Interacting-Head Attention.

Transformer-based models have gained significant advances in neural machine translation (NMT). The main component of the transformer is the multihead attention layer. In theory, more heads enhance the expressive power of the NMT model. But this is not always the case in practice. On the one hand, the computations of each head attention are conducted in the same subspace, without considering the different subspaces of all the tokens. On the other hand, the low-rank bottleneck may occur, when the number of heads surpasses a threshold. To address the low-rank bottleneck, the two mainstream methods make the head size equal to the sequence length and complicate the distribution of self-attention heads. However, these methods are challenged by the variable sequence length in the corpus and the sheer number of parameters to be learned. Therefore, this paper proposes the interacting-head attention mechanism, which induces deeper and wider interactions across the attention heads by low-dimension computations in different subspaces of all the tokens, and chooses the appropriate number of heads to avoid low-rank bottleneck. The proposed model was tested on machine translation tasks of IWSLT2016 DE-EN, WMT17 EN-DE, and WMT17 EN-CS. Compared to the original multihead attention, our model improved the performance by 2.78 BLEU/0.85 WER/2.90 METEOR/2.65 ROUGE_L/0.29 CIDEr/2.97 YiSi and 2.43 BLEU/1.38 WER/3.05 METEOR/2.70 ROUGE_L/0.30 CIDEr/3.59 YiSi on the evaluation set and the test set, respectively, for IWSLT2016 DE-EN, 2.31 BLEU/5.94 WER/1.46 METEOR/1.35 ROUGE_L/0.07 CIDEr/0.33 YiSi and 1.62 BLEU/6.04 WER/1.39 METEOR/0.11 CIDEr/0.87 YiSi on the evaluation set and newstest2014, respectively, for WMT17 EN-DE, and 3.87 BLEU/3.05 WER/9.22 METEOR/3.81 ROUGE_L/0.36 CIDEr/4.14 YiSi and 4.62 BLEU/2.41 WER/9.82 METEOR/4.82 ROUGE_L/0.44 CIDEr/5.25 YiSi on the evaluation set and newstest2014, respectively, for WMT17 EN-CS.

Adoption of Wireless Network and Artificial Intelligence Algorithm in Chinese-English Tense Translation

In order to solve the problem of tense consistency in Chinese-English neural machine translation (NMT) system, a Chinese verb tense annotation model is proposed. Firstly, a neural network is used to build a Chinese tense annotation model. During the translation process, the source tense is passed to the target side through the alignment matrix of the traditional Attention mechanism. The probability of the candidate words inconsistent with the corresponding tense of source words in the candidate translation word set is also reduced. Then, the Chinese-English temporal annotation algorithm is integrated into the MT model, so as to build a Chinese-English translation system with temporal processing function. The essence of the system is that, in the process of translation, Chinese-English temporal annotation algorithm is used to obtain temporal information from Chinese sentences and transfer it to the corresponding English sentences, so as to realize the temporal processing of English sentences and obtain the English sentences corresponding to the tenses of the original Chinese sentences. The experimental results show that the Chinese tense annotation model of bidirectional long short-term memory (LSTM) is more accurate for the prediction of Chinese verb tense, so the improvement effect of NMT model is also the most obvious, especially on the NIST06 test set, where the BLEU value is increased by 1.07%. As the mainstream translation model, the transformer model contains multihead Attention mechanism, which can pay attention to some temporal information and has a certain processing ability for temporal translation. It solves the tense problems encountered in the process of MT and improves the credibility of Chinese-English machine translation (MT).