Neural Machine Translation Tasks Research Articles

Text to video generation via knowledge distillation

<p>Text-to-video generation (T2V) has recently attracted more attention due to the wide application scenarios of video media. However, compared with the substantial advances in text-to-image generation (T2I), the research on T2V remains in its early stage. The difficulty mainly lies in maintaining the text-visual semantic consistency and the video temporal coherence. In this paper, we propose a novel distillation and translation GAN (DTGAN) to address these problems. First, we leverage knowledge distillation to guarantee semantic consistency. We distill text-visual mappings from a well-performing T2I teacher model and transfer it to our DTGAN. This knowledge serves as shared abstract features and high-level constraints for each frame in the generated videos. Second, we propose a novel visual recurrent unit (VRU) to achieve video temporal coherence. The VRU can generate frame sequences as well as process the temporal information across frames. It enables our generator to act as a multi-modal variant of the language model in neural machine translation task, which iteratively predicts the next frame based on the input text and the previously generated frames. We conduct experiments on two synthetic datasets (SBMG and TBMG) and one real-world dataset (MSVD). Qualitative and quantitative comparisons with state-of-the-art methods demonstrate that our DTGAN can generate results with better text-visual semantic consistency and temporal coherence.<strong></strong></p>

Handling syntactic difference in Chinese-Vietnamese neural machine translation

As the typical distant language pair, Chinese and Vietnamese vary widely in syntactic structure, which significantly influences the performance of Chinese-Vietnamese machine translation. To address this problem, we present a simple approach with a pre-reordering model for closing syntactic gaps of the Chinese-Vietnamese language pair. Specifically, we first propose an algorithm for recognizing the modifier inverse, one of the most representative syntactic different in Chinese-Vietnamese language pair. Then we pre-train a pre-reordering model based on the former recognition algorithm and incorporate it into the attention-based translation framework for syntactic different reordering. We conduct empirical studies on Chinese-Vietnamese neural machine translation task, the results show that our approach achieves average improvement of 2.75 BLEU points in translation quality over the baseline model. In addition, the translation fluency can be significantly improved by over 2.44 RIBES points.

Towards better Chinese-centric neural machine translation for low-resource languages

The last decade has witnessed enormous improvements in science and technology, stimulating the growing demand for economic and cultural exchanges in various countries. Building a neural machine translation (NMT) system has become an urgent trend, especially in the low-resource setting. However, recent work tends to study NMT systems for low-resource languages centered on English, while a few studies focus on low-resource NMT systems centered on other languages such as Chinese. To achieve this, the low-resource multilingual translation challenge of the 2021 iFLYTEK AI Developer Competition provides the Chinese-centric multilingual low-resource NMT tasks, where participants are required to build NMT systems based on the provided low-resource samples. In this paper, we present the competition-winning system that leverages monolingual word embeddings data enhancement, bilingual curriculum learning, and contrastive re-ranking. In addition, a new Incomplete-Trust (In-trust) loss function is proposed to replace the traditional cross-entropy loss when training. The extensive experimental results demonstrate that the implementation of these ideas leads to better performance than other state-of-the-art methods. All the experimental codes are released at: https://github.com/WENGSYX/Low-resource-text-translation.

Diversity of Models Based on Sequence to Sequence for Neural Machine Translation Tasks

One of the most critical areas of Natural Language Processing study is machine translation, which realizes the translation process from source language to target language. After a rapid development, neural machine translation has become mainstream. Most of the tasks of natural language generation are implemented based on sequence to sequence model. The current technology of machine translation is the use of recurrent neural networks and attention mechanisms, which is essentially a set of encoding and decoding techniques. In order to handle the issue of excessive information compression introduced by decoder, attention mechanism is proposed. Transformer is also built with attention, which not only computes faster, but also achieves better results on translation tasks. In this paper, a series of comparison of these models and description of the performance on the dataset were shown.

Selective Knowledge Distillation for Non-Autoregressive Neural Machine Translation

Benefiting from the sequence-level knowledge distillation, the Non-Autoregressive Transformer (NAT) achieves great success in neural machine translation tasks. However, existing knowledge distillation has side effects, such as propagating errors from the teacher to NAT students, which may limit further improvements of NAT models and are rarely discussed in existing research. In this paper, we introduce selective knowledge distillation by introducing an NAT evaluator to select NAT-friendly targets that are of high quality and easy to learn. In addition, we introduce a simple yet effective progressive distillation method to boost NAT performance. Experiment results on multiple WMT language directions and several representative NAT models show that our approach can realize a flexible trade-off between the quality and complexity of training data for NAT models, achieving strong performances. Further analysis shows that distilling only 5% of the raw translations can help an NAT outperform its counterpart trained on raw data by about 2.4 BLEU.

Approximating to the Real Translation Quality for Neural Machine Translation via Causal Motivated Methods

It is hard to evaluate translations objectively and accurately, which limits the applications of machine translation. In this article, we assume that the above phenomenon is caused by noise interference during translation evaluation, and we handle the problem through a perspective of causal inference. We assume that the observable translation score is affected by the unobservable true translation quality and some noise simultaneously. If there is a variable that is related to the noise and independent to the true translation quality, the related noise can be eliminated by removing the effect of that variable from the observed score. Based on the above causality hypothesis, this article studies the length bias problem of beam search for neural machine translation (NMT) and the input related noise problem of translation quality estimation (QE). For the NMT length bias problem, we conduct the experiments on four typical NMT tasks (Uyghur–Chinese, Chinese–English, English–German, and English–French) with different scales of datasets. Comparing with previous approaches, the proposed causal motivated method is model-agnostic and does not require supervised training. For QE tasks, we conduct the experiments on the WMT’20 submissions. Experimental results show that the denoised QE results gain better Pearson’s correlation scores with human assessed scores compared to the original submissions. Further analyses on the NMT and QE tasks also demonstrate the rationality of the empirical assumptions made on our methods.

Deep Learning-Based Refactoring with Formally Verified Training Data

Refactoring source code has always been an active area of research. Since the uprising of various deep learning methods, there have been several attempts to perform source code transformation with the use of neural networks. More specifically, Encoder-Decoder architectures have been used to transform code similarly to a Neural Machine Translation task. In this paper, we present a deep learning-based method to refactor source code, which we have prototyped for Erlang. Our method has two major components: a localizer and a refactoring component. That is, we first localize the snippet to be refactored using a recurrent network, then we generate an alternative with a Sequence-to- Sequence architecture. Our method could be used as an extension for already existing AST-based approaches for refactoring since it is capable of transforming syntactically incomplete code. We train our models on automatically generated data sets, based on formally verified refactoring definitions and by using attribute grammar-based sampling.

An empirical study of cyclical learning rate on neural machine translation

AbstractIn training deep learning networks, the optimizer and related learning rate are often used without much thought or with minimal tuning, even though it is crucial in ensuring a fast convergence to a good quality minimum of the loss function that can also generalize well on the test dataset. Drawing inspiration from the successful application of cyclical learning rate policy to computer vision tasks, we explore how cyclical learning rate can be applied to train transformer-based neural networks for neural machine translation. From our carefully designed experiments, we show that the choice of optimizers and the associated cyclical learning rate policy can have a significant impact on the performance. In addition, we establish guidelines when applying cyclical learning rates to neural machine translation tasks.

Unified Distillation: Supervising the Student to Make Fewer Mistakes

Knowledge distillation has become a popular task in modern deep learning applications by performing knowledge transfer from a cumbersome neural network commonly called “teacher model” to a much smaller network called “student model”. In traditional knowledge distilling process, there are usually two objectives for training the student model, namely hard target and soft target. However, sometimes it’s hard to find a trade-off between them. We unify the two objectives into one making it easier to perform knowledge distillation, and propose a novel distilling method called “Unified Distillation” to supervise the student to make fewer mistakes. The model can correct the wrong predictions according to the hard target, and maintain the advantage of knowledge distillation. Although our method can be used in almost all fields suitable for knowledge distillation, we choose neural machine translation as a study object for its complexity. We conducted experiments on three neural machine translation tasks, using a finetuned language model BERT as the teacher, and a Transformer base model as the student. The experimental results indicate that our method is better than traditional knowledge distillation method.

Named Entity Correction in Neural Machine Translation Using the Attention Alignment Map

Neural machine translation (NMT) methods based on various artificial neural network models have shown remarkable performance in diverse tasks and have become mainstream for machine translation currently. Despite the recent successes of NMT applications, a predefined vocabulary is still required, meaning that it cannot cope with out-of-vocabulary (OOV) or rarely occurring words. In this paper, we propose a postprocessing method for correcting machine translation outputs using a named entity recognition (NER) model to overcome the problem of OOV words in NMT tasks. We use attention alignment mapping (AAM) between the named entities of input and output sentences, and mistranslated named entities are corrected using word look-up tables. The proposed method corrects named entities only, so it does not require retraining of existing NMT models. We carried out translation experiments on a Chinese-to-Korean translation task for Korean historical documents, and the evaluation results demonstrated that the proposed method improved the bilingual evaluation understudy (BLEU) score by 3.70 from the baseline.

BERT-JAM: Maximizing the utilization of BERT for neural machine translation

Pre-training based approaches have been demonstrated effective for a wide range of natural language processing tasks. Leveraging BERT for neural machine translation (NMT), which we refer to as BERT-enhanced NMT, has received increasing interest in recent years. However, there still exists a research gap in studying how to maximize the utilization of BERT for NMT tasks. Firstly, previous studies mostly focus on utilizing BERT’s last-layer representation, neglecting the linguistic features encoded by the intermediate layers. Secondly, it requires further architectural exploration to integrate the BERT representation with the NMT encoder/decoder layers efficiently. And thirdly, existing methods keep the BERT parameters fixed during training to avoid the catastrophic forgetting problem, wasting the chances of boosting the performance via fine-tuning. In this paper, we propose BERT-JAM to fill the research gap from three aspects: 1) we equip BERT-JAM with fusion modules for composing BERT’s multi-layer representations into a fused representation that can be leveraged by the NMT model, 2) BERT-JAM utilizes joint-attention modules to allow the BERT representation to be dynamically integrated with the encoder/decoder representations, and 3) we train BERT-JAM with a three-phase optimization strategy that progressively unfreezes different components to overcome catastrophic forgetting during fine-tuning. Experimental results show that BERT-JAM achieves state-of-the-art BLEU scores on multiple translation tasks.

Self-supervised Bilingual Syntactic Alignment for Neural Machine Translation

While various neural machine translation (NMT) methods have integrated mono-lingual syntax knowledge into the linguistic representation of sequence-to-sequence, no research is available on aligning the syntactic structures of target language with the corresponding source language syntactic structures. This work shows the first attempt of a source-target bilingual syntactic alignment approach SyntAligner by mutual information maximization-based self-supervised neural deep modeling. Building on the word alignment for NMT, our SyntAligner firstly aligns the syntactic structures of source and target sentences and then maximizes their mutual dependency by introducing a lower bound on their mutual information. In SyntAligner, the syntactic structure of span granularity is represented by transforming source or target word hidden state into a source or target syntactic span vector. A border-sensitive span attention mechanism then captures the correlation between the source and target syntactic span vectors, which also captures the self-attention between span border-words as alignment bias. Lastly, a self-supervised bilingual syntactic mutual information maximization-based learning objective dynamically samples the aligned syntactic spans to maximize their mutual dependency. Experiment results on three typical NMT tasks: WMT'14 English to German, IWSLT'14 German to English, and NC'11 English to French show the SyntAligner effectiveness and universality of syntactic alignment.

Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation

Large-scale pre-trained language models (e.g., BERT) have attracted great attention in recent years. It is straightforward to fine-tune them on natural language understanding tasks such as text classification, however, effectively and efficiently incorporating them into natural language generation tasks such as neural machine translation remains a challenging problem. In this paper, we integrate two pre-trained BERT models from the source and target language domains into a sequence-to-sequence model by introducing light-weight adapter modules. The adapters are inserted between BERT layers and tuned on downstream tasks, while the parameters of BERT models are fixed during fine-tuning. As pre-trained language models are usually very deep, inserting adapters into all layers will result in a considerable scale of new parameters. To deal with this problem, we introduce latent variables to decide whether using adapters or not in each layer, which are learned during fine-tuning. In this way, the model is able to automatically determine which adapters to use, therefore hugely promoting the parameter efficiency and decoding speed. We evaluate the proposed framework on various neural machine translation tasks. Equipped with parallel sequence decoding, our model consistently outperforms autoregressive baselines while reducing the inference latency by half. With automatic adapter selection, the proposed model further achieves 20% speedup while still outperforming autoregressive baselines. When applied to autoregressive decoding, the proposed model can also achieve comparable performance with the state-of-the-art baseline models.

Korean Historical Documents Analysis with Improved Dynamic Word Embedding

Historical documents refer to records or books that provide textual information about the thoughts and consciousness of past civilisations, and therefore, they have historical significance. These documents are used as key sources for historical studies as they provide information over several historical periods. Many studies have analysed various historical documents using deep learning; however, studies that employ changes in information over time are lacking. In this study, we propose a deep-learning approach using improved dynamic word embedding to determine the characteristics of 27 kings mentioned in the Annals of the Joseon Dynasty, which contains a record of 500 years. The characteristics of words for each king were quantitated based on dynamic word embedding; further, this information was applied to named entity recognition and neural machine translation.In experiments, we confirmed that the method we proposed showed better performance than other methods. In the named entity recognition task, the F1-score was 0.68; in the neural machine translation task, the BLEU4 score was 0.34. We demonstrated that this approach can be used to extract information about diplomatic relationships with neighbouring countries and the economic conditions of the Joseon Dynasty.

Towards Integrated Classification Lexicon for Handling Unknown Words in Chinese-Vietnamese Neural Machine Translation

In Neural Machine Translation (NMT), due to the limitations of the vocabulary, unknown words cannot be translated properly, which brings suboptimal performance of the translation system. For resource-scarce NMT that have small-scale training corpus, the effect is amplified. The traditional approach of amplifying the scale of the corpus is not applicable, because the parallel corpus is difficult to obtain in a resource-scarce setting; however, it is easy to obtain and utilize external knowledge, bilingual lexicon, and other resources. Therefore, we propose classification lexicon approach for processing unknown words in the Chinese-Vietnamese NMT task. Specifically, three types of unknown Chinese-Vietnamese words are classified and their corresponding classification lexicon are constructed by word alignment, Wikipedia extraction, and rule-based methods, respectively. After translation, the unknown words are restored by lexicon for post-processing. Experiment results on Chinese-Vietnamese, English-Vietnamese, and Mongolian-Chinese translations show that our approach significantly improves the accuracy and the performance of NMT especially in a resource-scarce setting.

Bi-Decoder Augmented Network for Neural Machine Translation

Neural Machine Translation (NMT) has become a popular technology in recent years, and the encoder–decoder framework is the mainstream among all the methods. It is obvious that the quality of the semantic representations from encoding is very crucial and can significantly affect the performance of the model. However, existing unidirectional source-to-target architectures may hardly produce a language-independent representation of the text because they rely heavily on the specific relations of the given language pairs. To alleviate this problem, in this paper, we propose a novel Bi-Decoder Augmented Network (BiDAN) for the neural machine translation task. Besides the original decoder which generates the target language sequence, we add an auxiliary decoder to generate back the source language sequence at the training time. Since each decoder transforms the representations of the input text into its corresponding language, jointly training with two target ends can make the shared encoder has the potential to produce a language-independent semantic space. We conduct extensive experiments on several NMT benchmark datasets and the results demonstrate the effectiveness of our proposed approach.

Neural Machine Translation: A Review of the Approaches

Neural Machine Translation (NMT) has presented promising results in Machine translation, convincingly replacing the traditional Statistical Machine Translation (SMT). This success of NMT in machine translation tasks therefore projects to more translation tasks using NMT. This paper systematically reviews the hitherto proposed NMT systems since 2014. 86 NMT papers have been selected and reviewed. The peak of NMT systems were proposed in 2016 and the same was the case for many machine translation workshops who provided datasets for NMT tasks. Most of the proposed systems covered English, German, French and Chinese translation tasks. BLEU score accompanied by significance tests has been seen to be the best metric for NMT systems evaluation. Human judgement for fluency and adequacy is also important to support the metrics. There is still room for further improvement in translations regarding rich source translations and rare words. There is also need for extensive NMT works in other languages to maximize the apparent capabilities of NMT systems. RNN Search and Moses are basically used to develop SMT baselines for model comparisons. Results provide futuristic and directional insights into further translation tasks.

Sentence-Wise Smooth Regularization for Sequence to Sequence Learning

Maximum-likelihood estimation (MLE) is widely used in sequence to sequence tasks for model training. It uniformly treats the generation/prediction of each target token as multiclass classification, and yields non-smooth prediction probabilities: in a target sequence, some tokens are predicted with small probabilities while other tokens are with large probabilities. According to our empirical study, we find that the non-smoothness of the probabilities results in low quality of generated sequences. In this paper, we propose a sentence-wise regularization method which aims to output smooth prediction probabilities for all the tokens in the target sequence. Our proposed method can automatically adjust the weights and gradients of each token in one sentence to ensure the predictions in a sequence uniformly well. Experiments on three neural machine translation tasks and one text summarization task show that our method outperforms conventional MLE loss on all these tasks and achieves promising BLEU scores on WMT14 English-German and WMT17 Chinese-English translation task.

Expressive Ontology Learning as Neural Machine Translation

Automated ontology learning from unstructured textual sources has been proposed in literature as a way to support the difficult and time-consuming task of knowledge modeling for semantic applications. In this paper we propose a system, based on a neural network in the encoder-decoder configuration, to translate natural language definitions into Description Logics formulae through syntactic transformation. The model has been evaluated to asses its capacity to generalize over different syntactic structures, tolerate unknown words, and improve its performance by enriching the training set with new annotated examples. The results obtained in our evaluation show how approaching the ontology learning problem as a neural machine translation task can be a valid way to tackle long term expressive ontology learning challenges such as language variability, domain independence, and high engineering costs.

Expressive ontology learning as neural machine translation

Automated ontology learning from unstructured textual sources has been proposed in literature as a way to support the difficult and time-consuming task of knowledge modeling for semantic applications. In this paper we propose a system, based on a neural network in the encoder–decoder configuration, to translate natural language definitions into Description Logics formulæ through syntactic transformation. The model has been evaluated to assess its capacity to generalize over different syntactic structures, tolerate unknown words, and improve its performance by enriching the training set with new annotated examples. The results obtained in our evaluation show how approaching the ontology learning problem as a neural machine translation task can be a valid way to tackle long term expressive ontology learning challenges such as language variability, domain independence, and high engineering costs.