Neural Machine Translation Models Research Articles

Neural Machine Translation (NMT): Deep learning approaches through Neural Network Models

Abstract. This paper explores the significant advancements in Neural Machine Translation (NMT) models, focusing on the impact of different architectures, training methodologies, and optimization techniques on translation quality. The study contrasts the performance of Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and the Transformer model, highlighting the superior capabilities of the Transformer in handling long-range dependencies and providing contextually accurate translations. Key optimization techniques, such as learning rate scheduling, dropout regularization, and gradient clipping, are discussed in detail, emphasizing their roles in enhancing model performance and training efficiency. Furthermore, the paper presents a comparative analysis of NMT and traditional Statistical Machine Translation (SMT) systems, showcasing NMT's superior BLEU scores and fluency. The application of model distillation is also examined, demonstrating how smaller models can achieve high performance with reduced computational resources. These findings underscore the transformative potential of NMT in achieving state-of-the-art translation quality and efficiency.

Enhancing Neural Machine Translation Quality for Kannada–Tulu Language Pairs through Transformer Architecture: A Linguistic Feature Integration

The rise of intelligent systems demands good machine translation models that are less data hungry and more efficient, especially for low- and extremely-low-resource languages with few or no data available. By integrating a linguistic feature to enhance the quality of translation, we have developed a generic Neural Machine Translation (NMT) model for Kannada–Tulu language pairs. The NMT model uses Transformer architecture and a state-of-the-art model for translating text from Kannada to Tulu and learns based on the parallel data. Kannada and Tulu are both low-resource Dravidian languages, with Tulu recognised as an extremely-low-resource language. Dravidian languages are morphologically rich and are highly agglutinative in nature and there exist only a few NMT models for Kannada–Tulu language pairs. They exhibit poor translation scores as they fail to capture the linguistic features of the language. The proposed generic approach can benefit other low-resource Indic languages that have smaller parallel corpora for NMT tasks. Evaluation metrics like Bilingual Evaluation Understudy (BLEU), character-level F-score (chrF) and Word Error Rate (WER) are considered to obtain the improved translation scores for the linguistic-feature-embedded NMT model. These results hold promise for further experimentation with other low- and extremely-low-resource language pairs.

C anton MT: Investigating Back-Translation and Model-Switch Mechanisms for Cantonese-English Neural Machine Translation

This paper investigates the development and evaluation of machine translation models from Cantonese to English (and backward), where we propose a novel approach to tackle low-resource language translations. Despite recent improvements in Neural Machine Translation (NMT) models with Transformer-based architectures, Cantonese, a language with over 80 million native speakers, has below-par State-of-the-art commercial translation models due to a lack of resources. The main objectives of the study are to develop a model that can effectively translate Cantonese to English and evaluate it against state-of-the-art commercial models. To achieve this, a new parallel corpus has been created by combining different available corpora online with preprocessing and cleaning. In addition, a monolingual Cantonese dataset has been created through web scraping to aid the synthetic parallel corpus generation. Following the data collection process, several approaches, including fine-tuning models, back-translation , and model switch , have been used. The translation quality of models has been evaluated with multiple quality metrics, including lexicon-based metrics (SacreBLEU and hLEPOR) and embedding-space metrics (COMET and BERTscore). Based on the automatic metrics, the best model is selected and compared against the 2 best commercial translators using a new human evaluation framework HOPES . The best model proposed in this investigation (NLLB-mBART) with model switch mechanisms has reached comparable and even better automatic evaluation scores against State-of-the-art commercial models (Bing and Baidu Translators), with a SacreBLEU score of 16.8 on our test set. Furthermore, an open-source web application has been developed to allow users to translate between Cantonese and English, with the different trained models available for effective comparisons between models from this investigation and users. CantonMT is available at https://github.com/kenrickkung/CantoneseTranslation

Improved Urdu-English Neural Machine Translation with a fully Convolutional Neural Network Encoder

Neural machine translation (NMT) approaches driven by artificial intelligence (AI) has gained more and more attention in recent years, mainly due to their simplicity yet state-of-the-art performance. Despite NMT models with attention mechanism relying heavily on the accessibility of substantial parallel corpora, they have demonstrated efficacy even for languages with limited linguistic resources. The convolutional neural network (CNN) is frequently employed in tasks involving visual and speech recognition. Implementing CNN for MT is still challenging compared to the predominant approaches. Recent research has shown that the CNN-based NMT model cannot capture long-term dependencies present in the source sentence. The CNN-based model can only capture the word dependencies within the width of its filters. This unnatural character often causes a worse performance for CNN-based NMT than the RNN-based NMT models. This study introduces a simple method to improve neural translation of a low-resource language, specifically Urdu-English (UR-EN). In this paper, we use a Fully Convolutional Neural Network (FConv-NN) based NMT architecture to create a powerful MT encoder for UR-EN translation that can capture the long dependency of words in a sentence. Although the model is quite simple, it yields strong empirical results. Experimental results show that the FConv-NN model consistently outperforms the traditional CNN-based model with filters. On the Urdu-English Dataset, the FConv-NN model produces translation with a gain of 18.42 BLEU points. Moreover, the quantitative and comparative analysis shows that in a low-resource setting, FConv-NN-based NMT outperforms conventional CNN-based NMT models.

Efficient incremental training using a novel NMT-SMT hybrid framework for translation of low-resource languages.

The data-hungry statistical machine translation (SMT) and neural machine translation (NMT) models offer state-of-the-art results for languages with abundant data resources. However, extensive research is imperative to make these models perform equally well for low-resource languages. This paper proposes a novel approach to integrate the best features of the NMT and SMT systems for improved translation performance of low-resource English-Tamil language pair. The suboptimal NMT model trained with the small parallel corpus translates the monolingual corpus and selects only the best translations, to retrain itself in the next iteration. The proposed method employs the SMT phrase-pair table to determine the best translations, based on the maximum match between the words of the phrase-pair dictionary and each of the individual translations. This repeating cycle of translation and retraining generates a large quasi-parallel corpus, thus making the NMT model more powerful. SMT-integrated incremental training demonstrates a substantial difference in translation performance as compared to the existing approaches for incremental training. The model is strengthened further by adopting a beam search decoding strategy to produce k best possible translations for each input sentence. Empirical findings prove that the proposed model with BLEU scores of 19.56 and 23.49 outperforms the baseline NMT with scores 11.06 and 17.06 for Eng-to-Tam and Tam-to-Eng translations, respectively. METEOR score evaluation further corroborates these results, proving the supremacy of the proposed model.

Knowledge Graph Guided Neural Machine Translation with Dynamic Reinforce-selected Triples

Previous methods incorporating knowledge graphs (KGs) into neural machine translation (NMT) adopt a static knowledge utilization strategy, that introduces many useless knowledge triples and makes the useful triples difficult be utilized by NMT. To address this problem, we propose a KG guided NMT model with dynamic reinforce-selected triples. The proposed methods could dynamically select the different useful knowledge triples for different source sentences. Specifically, the proposed model contains two components: 1) knowledge selector, that dynamically selects useful knowledge triples for a source sentence, and 2) knowledge guided NMT (KgNMT), that utilizes the selected triples to guide the translation of NMT. Meanwhile, to overcome the non-differentiable problem and guide the training procedure, we propose a policy gradient strategy to encourage the model to select useful triples and improve the generation probability of gold target sentence. Various experimental results show that the proposed method can significantly outperform the baseline models in both translation quality and handling the entities.

Reinforcement of low-resource language translation with neural machine translation and backtranslation synergies

<p><span lang="EN-US">This research investigates challenges and advancements in neural machine translation (NMT), specifically targeting English-to-Kannada translation. Emphasizing the scarcity of data and linguistic complexity in low-resource languages (LRL), particularly Kannada, the study underscores the need for specialized techniques. Starting with exploration of Kannada's historical and cultural significance, the paper highlights critical importance of linguistic comprehension. The primary objective is to develop robust NMT models for precise and contextually relevant translations in low-resource scenarios. The novelty of this research lies in its innovative approach to Kannada NMT challenges, incorporating comprehensive examination of historical and cultural context to establish strong linguistic foundation. Motivated by the urgency to address translation needs in LRL, the paper proposes novel strategies, advocating notably for backtranslation to generate synthetic parallel corpora. Rigorous testing, including bilingual evaluation understudy (BLEU) score assessments, evaluates effectiveness of these proposed approaches. Beyond assessing backtranslation, the study explores challenges faced by Kannada NMT in handling dialectical and spelling variations. The research reports substantial 83-percentage-point average increase in BLEU scores, contingent on aligning unique Kannada terms with the same domain as existing occurrences. This study contributes significantly to Kannada natural language processing by offering novel insights into NMT intricacies and providing practical solutions for enhancing translation accuracy in low-resource settings.</span></p>

Exploration on Advanced Intelligent Algorithms of Artificial Intelligence for Verb Recognition in Machine Translation

This article aimed to address the problems of word order confusion, context dependency, and ambiguity in traditional machine translation (MT) methods for verb recognition. By applying advanced intelligent algorithms of artificial intelligence, verb recognition can be better processed and the quality and accuracy of MT can be improved. Based on Neural machine translation (NMT), basic attention mechanisms, historical attention information, dynamically obtain information related to the generated words, and constraint mechanisms were introduced to embed semantic information, represent polysemy, and annotate semantic roles of verbs. This article used the Workshop on MT (WMT), British National Corpus (BNC), Gutenberg, Reuters Corpus, and OpenSubtitles corpus, and enhanced the data in the corpora. The improved NMT model was compared with traditional NMT models, Rule-Based MT (RBMT), and Statistical MT (SMT). The experimental results showed that the average verb semantic matching degree of the improved NMT model in five corpora was 0.85, and the average Bilingual Evaluation Understudy (BLEU) score in five corpora was 0.90. The improved NMT model in this article can effectively improve the accuracy of verb recognition in MT, providing new methods for verb recognition in MT.

Bilingual Neural Machine Translation From English To Yoruba Using A Transformer Model

The necessity for language translation in Nigeria arises from its linguistic diversity, facilitating effective communication and understanding across communities. Yoruba, considered a language with limited resources, has potential for greater online presence. This research proposes a neural machine translation model using a transformer architecture to convert English text into Yoruba text. While previous studies have addressed this area, challenges such as vanishing gradients, translation accuracy, and computational efficiency for longer sequences persist. This research proposes to address these limitations by employing a transformer- based model, which has demonstrated efficacy in overcoming issues associated with Recurrent Neural Networks (RNNs). Unlike RNNs, transformers utilize attention mechanisms to establish comprehensive connections between input and output, improving translation quality and computational efficiency.

NmTHC: a hybrid error correction method based on a generative neural machine translation model with transfer learning

BackgroundsThe single-pass long reads generated by third-generation sequencing technology exhibit a higher error rate. However, the circular consensus sequencing (CCS) produces shorter reads. Thus, it is effective to manage the error rate of long reads algorithmically with the help of the homologous high-precision and low-cost short reads from the Next Generation Sequencing (NGS) technology.MethodsIn this work, a hybrid error correction method (NmTHC) based on a generative neural machine translation model is proposed to automatically capture discrepancies within the aligned regions of long reads and short reads, as well as the contextual relationships within the long reads themselves for error correction. Akin to natural language sequences, the long read can be regarded as a special “genetic language” and be processed with the idea of generative neural networks. The algorithm builds a sequence-to-sequence(seq2seq) framework with Recurrent Neural Network (RNN) as the core layer. The before and post-corrected long reads are regarded as the sentences in the source and target language of translation, and the alignment information of long reads with short reads is used to create the special corpus for training. The well-trained model can be used to predict the corrected long read.ResultsNmTHC outperforms the latest mainstream hybrid error correction methods on real-world datasets from two mainstream platforms, including PacBio and Nanopore. Our experimental evaluation results demonstrate that NmTHC can align more bases with the reference genome without any segmenting in the six benchmark datasets, proving that it enhances alignment identity without sacrificing any length advantages of long reads.ConclusionConsequently, NmTHC reasonably adopts the generative Neural Machine Translation (NMT) model to transform hybrid error correction tasks into machine translation problems and provides a novel perspective for solving long-read error correction problems with the ideas of Natural Language Processing (NLP). More remarkably, the proposed methodology is sequencing-technology-independent and can produce more precise reads.

Automatic translation from English to Amazigh using transformer learning

Due to the lack of parallel data, to our knowledge, no study has been conducted on the Amazigh-English language pair, despite the numerous machine translation studies completed between major European language pairs. We decided to utilize the neural machine translation (NMT) method on a parallel corpus of 137,322 sentences. The attention-based encoder-decoder architecture is used to construct statistical machine translation (SMT) models based on Moses, as well as NMT models using long short-term memory (LSTM), gated recurrent units (GRU), and transformers. Various outcomes were obtained for each strategy after several simulations: 80.7% accuracy was achieved using the statistical approach, 85.2% with the GRU model, 87.9% with the LSTM model, and 91.37% with the transformer.

Weight Saliency search with Semantic Constraint for Neural Machine Translation attacks

Text adversarial attack is an effective way to improve the robustness of Neural Machine Translation (NMT) models. Existing NMT attack tasks are often completed by replacing words. However, most of previous works pursue a high attack success rate but produce semantic inconsistency sentences, leading to wrong translations even for humans. In this paper, we propose a Weight Saliency search with Semantic Constraint (WSSC) algorithm to make semantic consistency word modifications to the input sentence for black-box NMT attacks. Specifically, our WSSC has two major merits. First, it optimizes the word substitution with a word saliency method, which is helpful to reduce word replacement rate. Second, it constrains the objective function with a semantic similarity loss, ensuring every modification does not lead to significant semantic changes. We evaluate the effectiveness of the proposed WSSC by attacking three popular NMT models, i.e., T5, Marian, and BART, on three widely used datasets, i.e., WMT14, WMT16, and TED. Experimental results validate that our WSSC improves Attack Success Rate (ASR) by 4.02% and Semantic Similarity score (USE) by 1.28% on average. Besides, our WSSC also shows good properties in keeping grammar correctness and transfer attack.

E-learning application in english writing classroom based on neural machine translation and semantic analysis algorithms

At present, there are some problems in traditional English writing teaching, such as the lack of practice opportunities and the lack of timely feedback. The aim of this study is to design an e-learning platform to improve students’ writing ability through neural machine translation and semantic analysis algorithms. This paper introduces the basic principles of neural machine translation and semantic analysis algorithms and their applications in translation and semantic understanding. In this paper, a computer English writing correction model based on neural machine translation and semantic analysis algorithm is proposed and applied to e-learning platform. The platform structure is constructed. The neural machine translation model is used to translate and transform the original writing content into standard English expressions. Then, semantic analysis algorithm is used to detect errors and propose suggestions for modification. The results show that the e-learning platform can provide personalized and timely feedback, and students’ writing ability has been significantly improved on the e-learning platform. They have improved their writing skills and presentation through exercises and submissions through the e-learning platform.

Comparison of deep learning approaches to estimate injury severity from the International Classification of Diseases codes

Objective The injury severity classification based on the Abbreviated Injury Scale (AIS) provides information that allows for standardized comparisons for injury research. However, the majority of injury data is captured using the International Classification of Diseases (ICD), which lacks injury severity information. It has been shown that the encoder-decoder-based neural machine translation (NMT) model is more accurate than other methods for determining injury severity from ICD codes. The objectives of this project were to determine if feed-forward neural networks (FFNN) perform as well as NMT and to determine if direct estimation of injury severity is more accurate than using AIS codes as an intermediary (indirect method). Methods Patient data from the National Trauma Data Bank were used to develop and test the four models (NMT/Indirect, NMT/Direct, FFNN/Indirect, FFNN/Direct). There were 2,031,793 cases from 2017–2018 used to train and 1,091,792 cases from 2019 were used for testing. The primary outcome of interest was the percent of cases with the correct binary classification of Injury Severity Score (ISS) ≥16, using ISS values recorded in NTDB for benchmarking. The secondary outcome was the percent of predicted ISS exactly matching the recorded ISS. Results The results show that indirect estimation through first converting to AIS using an NMT was the most accurate in predicting ISS ≥ 16 (94.0%), followed by direct estimation with FFNN (93.4%), direct estimation with NMT (93.1%), and then indirect estimation with FFNN (93.1%), with statistically significant differences in pairwise comparison. The rankings were the same when evaluating models based on exactly matches of ISS. Training times were similar for all models (range 11–14 h), but testing was much faster for FFNN models (GPU: 1–2 min) compared to the NMT models (GPU: 69–82 min). Conclusions The most accurate method for obtaining injury severity from ICD was NMT using AIS codes as an intermediary (indirect method), although all methods performed well. The indirect NMT model was the most resource intensive in terms of processing time. The optimal approach for researchers will be based on their needs and the computing resources available.

Ainu–Japanese Bi-directional Neural Machine Translation: A Step Towards Linguistic Preservation of Ainu, An Under-Resourced Indigenous Language in Japan

This study presents a groundbreaking approach to preserving the Ainu language, recognized as critically endangered by UNESCO, by developing a bi-directional neural machine translation (MT) system between Ainu and Japanese. Utilizing the Marian MT framework, known for its effectiveness with resource-scarce languages, the research aims to overcome the linguistic complexities inherent in Ainu's polysynthetic structure. The paper delineates a comprehensive methodology encompassing data collection from diverse Ainu text sources, meticulous preprocessing, and the deployment of neural MT models, culminating in the achievement of significant SacreBLEU scores that underscore the models' translation accuracy. The findings illustrate the potential of advanced MT technology to facilitate linguistic preservation and educational endeavors, advocating for integrating such technologies in safeguarding endangered languages. This research not only underscores the critical role of MT in bridging language divides but also sets a precedent for employing computational linguistics to preserve cultural and linguistic heritage.

Learning Domain Specific Sub-layer Latent Variable for Multi-Domain Adaptation Neural Machine Translation

Domain adaptation proves to be an effective solution for addressing inadequate translation performance within specific domains. However, the straightforward approach of mixing data from multiple domains to obtain the multi-domain neural machine translation (NMT) model can give rise to the parameter interference between domains problem, resulting in a degradation of overall performance. To address this, we introduce a multi-domain adaptive NMT method aimed at learning domain specific sub-layer latent variable and employ the Gumbel-Softmax reparameterization technique to concurrently train both model parameters and domain specific sub-layer latent variable. This approach facilitates the learning of private domain-specific knowledge while sharing common domain-invariant knowledge, effectively mitigating the parameter interference problem. The experimental results show that our proposed method significantly improved by up to 7.68 and 3.71 BLEU compared with the baseline model in English-German and Chinese-English public multi-domain datasets, respectively.

Neural machine translation for low resource Indian language: Hindi-Kangri

Neural Machine Translation (NMT) for low resource languages is a challenging task due to unavailability of large parallel corpus. The efficacy of Transformer based NMT models largely depends on scale of the parallel corpus and the configuration of hyperparameters implemented during model training. This study aims to delve into and elucidate the impact of hyperparameters on the performance of NMT models for low resource languages. To accomplish this, a series of experiments are conducted using an open-source Hindi-Kangri corpus to train both supervised and semi-supervised NMT models. Throughout the experimentation process, a significant number of discrepancies were identified within the data-set, necessitating manual correction. The best translation performance evaluated with respect to the metrics such as BLEU (0–1), SacreBLEU (0–100), Chrf (0–100), Chrf+ (0–100), Chrf++ (0–100) and TER (%) is (0.15, 14.98, 41.43, 41.49, 38.77, 68.20) for Hindi to Kangri direction, and (0.283, 28.17, 49.71, 50.64, 48.63, 51.25) for Kangri to Hindi direction.

A multimodal approach to cross-lingual sentiment analysis with ensemble of transformer and LLM

Sentiment analysis is an essential task in natural language processing that involves identifying a text’s polarity, whether it expresses positive, negative, or neutral sentiments. With the growth of social media and the Internet, sentiment analysis has become increasingly important in various fields, such as marketing, politics, and customer service. However, sentiment analysis becomes challenging when dealing with foreign languages, particularly without labelled data for training models. In this study, we propose an ensemble model of transformers and a large language model (LLM) that leverages sentiment analysis of foreign languages by translating them into a base language, English. We used four languages, Arabic, Chinese, French, and Italian, and translated them using two neural machine translation models: LibreTranslate and Google Translate. Sentences were then analyzed for sentiment using an ensemble of pre-trained sentiment analysis models: Twitter-Roberta-Base-Sentiment-Latest, bert-base-multilingual-uncased-sentiment, and GPT-3, which is an LLM from OpenAI. Our experimental results showed that the accuracy of sentiment analysis on translated sentences was over 86% using the proposed model, indicating that foreign language sentiment analysis is possible through translation to English, and the proposed ensemble model works better than the independent pre-trained models and LLM.

Low resource neural machine translation model optimization based on semantic confidence weighted alignment

Low resource neural machine translation model optimization based on semantic confidence weighted alignment

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

The attention mechanism performs well for the Neural Machine Translation (NMT) task, but heavily depends on the context vectors generated by the attention network to predict target words. This reliance raises the issue of long-term dependencies. Indeed, it is very common to combine predicates with postpositions in sentences, and the same predicate may have different meanings when combined with different postpositions. This usually poses an additional challenge to the NMT study. In this work, we observe that the embedding vectors of different target tokens can be classified by part-of-speech, thus we analyze the Natural Language Processing (NLP) related Content-Adaptive Recurrent Unit (CARU) unit and apply it to our attention model (CAAtt) and embedding layer (CAEmbed). By encoding the source sentence with the current decoded feature through the CARU, CAAtt is capable of achieving translation content-adaptive representations, which attention weights are contributed and enhanced by our proposed L1expNx normalization. Furthermore, CAEmbed aims to alleviate long-term dependencies in the target language through partial recurrent design, performing the feature extraction in a local perspective. Experiments on the WMT14, WMT17, and Multi30k translation tasks show that the proposed model achieves improvements in BLEU scores and enhancement of convergence over the attention-based plain NMT model. We also investigate the attention weights generated by the proposed approaches, which indicate that refinement over the different combinations of adposition can lead to different interpretations. Specifically, this work provides local attention to some specific phrases translated in our experiment. The results demonstrate that our approach is effective in improving performance and achieving a more reasonable attention distribution compared to the state-of-the-art models.