Machine Translation Evaluation Research Articles

A survey of context in neural machine translation and its evaluation

Abstract The question of context in neural machine translation often focuses on topics related to document-level translation or intersentential context. However, there is a wide range of other aspects that can be considered under the umbrella of context. In this work, we survey ways that researchers have incorporated context into neural machine translation systems and the evaluation thereof. This includes building translation systems that operate at the paragraph level or the document level or ones that translate at the sentence level but incorporate information from other sentences. We also consider how issues like terminology consistency, anaphora, and world knowledge or external information can be considered as types of context relevant to the task of machine translation and its evaluation. Closely tied to these topics is the question of how to best evaluate machine translation output in a way that is sensitive to the contexts in which it appears. To this end, we discuss work on incorporating context into both human and automatic evaluations of machine translation quality. Furthermore, we also discuss recent experiments in the field as they relate to the use of large language models in translation and evaluation. We conclude with a view of the future of machine translation, where we expect to see issues of context continue to come to the forefront.

Evaluating NMT using the non-inferiority principle

Abstract The aim of this article is to propose a new neural machine translation (NMT) evaluation method based on the non-inferiority principle. In order to do that, we evaluate raw machine translation (MT) in terms of naturalness, which for this research is defined as not just the lack of fluency errors but also meeting the linguistic expectations of Galician end users when reading original texts in Galician. Our main objective is, in the first place, to validate the new methodology presented in our previous study by evaluating an NMT engine from Spanish into Galician for the social media domain that was retrained with a new Twitter corpus. This new methodology and NMT engine were applied after analyzing the conclusions of a pilot survey conducted among Twitter users to evaluate their perception of tweets translated from Spanish into Galician with our NMT engine created with a corpus of tweets. As in our preliminary study, our aim is to propose a robust quality approximation method based on the reception parameters of end users’ perceptions. This new survey was conducted in December of 2022 with the participation of 228 Galician-speaking Twitter users. Among the main changes proposed are the inclusion of more information about the participant profile, so the non-inferiority principle can be also evaluated according to these parameters; the inclusion of a new typology of tweets, the threads; the provision of context by means of presenting the tweets in their original display as shown in the Twitter app; a change in the number of tweets evaluated and the number of different questionnaires; the change in the distribution of the questionnaires; and the inclusion of an error classification human evaluation conducted by professional linguists to correlate the findings. We will present the steps carried out following the conclusions of the pilot study, describe the new study’s design, analyze the new findings, and present the final conclusions regarding the engine and the evaluation method based on the non-inferiority principle. Finally, we will also provide some examples of the use of this new methodology in the translation industry.

The use of residual analysis to improve the error rate accuracy of machine translation

The aim of the study is to compare two different approaches to machine translation—statistical and neural—using automatic MT metrics of error rate and residuals. We examined four available online MT systems (statistical Google Translate, neural Google Translate, and two European commission’s MT tools—statistical mt@ec and neural eTranslation) through their products (MT outputs). We propose using residual analysis to improve the accuracy of machine translation error rate. Residuals represent a new approach to comparing the quality of statistical and neural MT outputs. The study provides new insights into evaluating machine translation quality from English and German into Slovak through automatic error rate metrics. In the category of prediction and syntactic-semantic correlativeness, statistical MT showed a significantly higher error rate than neural MT. Conversely, in the category of lexical semantics, neural MT showed a significantly higher error rate than statistical MT. The results indicate that relying solely on the reference when determining MT quality is insufficient. However, when combined with residuals, it offers a more objective view of MT quality and facilitates the comparison of statistical MT and neural MT.

Evaluation of Instagram's Neural Machine Translation for Literary Texts: An MQM-Based Analysis

Evaluation of Instagram's Neural Machine Translation for Literary Texts: An MQM-Based Analysis

A comparative analysis of lexical-based automatic evaluation metrics for different Indic language pairs

<p>With the rise of machine translation systems, it has become essential to evaluate the quality of translations produced by these systems. However, the existing evaluation metrics designed for English and other European languages may not always be suitable or apply to other Indic languages due to their complex morphology and syntax. Machine translation evaluation (MTE) is a process of assessing the quality and accuracy of the machine-translated text. MTE involves comparing the machine-translated output with the reference translation to calculate the level of similarity and correctness. Therefore, this study evaluates different metrics, namely, BLEU, METEOR, and TER to identify the most suitable evaluation metric for Indic languages. The study uses datasets for Indic languages and evaluates the metrics on various translation systems. The study contributes to the field of MT by providing insights into suitable evaluation metrics for Indic languages. This research paper aims to study and compare several lexical automatic machine translation evaluation metrics for Indic languages. For this research analysis, we have selected five language pairs of parallel corpora from the low-resource domain, such as English–Hindi, English-Punjabi, English-Gujarati, English-Marathi, and English-Bengali. All these languages belong to the Indo-Aryan language family and are resource-poor. A comparison of the state of art MT is presented and shows which translator works better on these language pairs. For this research work, the natural language toolkit tokenizers are used to assess the analysis of the experimental results. These results have been performed by taking two different datasets for all these language pairs using fully automatic MT evaluation metrics. The research study explores the effectiveness of these metrics in assessing the quality of machine translations between various Indic languages. Additionally, this dataset and analysis will make it easier to do future research in Indian MT evaluation.</p>

Evaluating machine translation of literature through rhetorical analysis

This paper looks at how well ChatGPT and DeepL, two AI tools, translate literary works. Not only can ChatGPT translate text, but it can also carry out other jobs. DeepL is a service that performs computer translation and uses neural networks. The paper looks at how ChatGPT and DeepL translate books, poems, and dialogues compared to translations done by humans. The paper also talks about the pros and cons of using machine translation for literary reasons, including issues of creativity, style, and adapting to different cultures. The paper uses both new and old studies on machine translation technologies and how they work with human translation. The paper comes to the conclusion that ChatGPT and DeepL are useful but imperfect tools for translating literature, and they require human review and improvement. The paper adds to the fields of machine translation and natural language processing by looking at how two cutting-edge AI tools, ChatGPT and DeepL, can be used to translate literary works. The paper also adds to literature studies and digital humanities by looking into what machine translation can and can't do for creative writing and dialog systems. The goal of the paper is to encourage researchers, translators, writers, and users from different fields to work together and talk to each other. ⁤

A review of existing Machine Translation Approaches, their Challenges and Evaluation Metrics

Machine translation is the process of translating a natural language into another. The primary goal of machine translation is to bridge the linguistic gap between languages. A significant job that may be utilized is to extract information from texts written in different languages through machine translation (MT). Due to the remarkable developments in Machine Translation over the past several years, we have entered the era of Neural Machine Translation (NMT). A review of MT is essential for better understanding of MT in the domain of Natural Language Processing. This paper focuses on Statistical, Corpus, Example, Rule based, Transfer based, direct based and Nural Machine Translation approach, their challenges, and existing evaluation metrics such as BLEU score and Human evaluation metrics.

Evaluation of Neural Machine translation for conversion of International Classification of disease codes to the Abbreviated injury Scale

The Abbreviated Injury Scale (AIS) is an essential tool for injury research since it allows for comparisons of injury severity among patients, however, the International Classification of Diseases (ICD) is more widely used to capture medical information. The problem of conversion between these two medical coding systems has similarities to the challenges encountered in language translation. We therefore hypothesize that neural machine translation (NMT), a deep learning technique which is commonly used for human language translation, could be used to convert ICD codes to AIS. The objective of this study was to compare the accuracy of a NMT model for determining injury severity compared to two established methods of conversion. The injury severity classifications used for this study were Injury Severity Score (ISS) ≥ 16, Maximum AIS severity (MAIS) ≥ 3, and MAIS ≥ 2. Data from a US national trauma registry, which has patient injuries coded in both AIS and ICD, was used to train a NMT model. Testing data from a separate year was used to determine the accuracy of the NMT model predictions against the actual ISS recorded in the registry. The prediction accuracy of the NMT model was compared to that of the official Association for the Advancement of Automotive Medicine (AAAM) ICD-AIS map and the R package ‘ICD Program for Injury Categorization in R’ (ICDPIC-R). The results show that the NMT model was the most accurate across all injury severity classifications, followed by the ICD-AIS map and then ICDPIC-R package. The NMT model also showed the highest correlation between the predicted and observe ISS scores. Overall, NMT appears to be a promising method for predicting injury severity from ICD codes, however, validation in external databases is needed.

Recovering Word Forms by Context for Morphologically Rich Languages

In this work, we focus on “sentence-level unlemmatization,” the task of generating a grammatical sentence given a lemmatized one; this task is usually easy to do for humans but may present problems for machine learning models. We treat this setting as a machine translation problem and, as a first try, apply a sequence-to-sequence model to the texts of Russian Wikipedia articles, evaluate the effect of the different training sets sizes quantitatively and achieve the BLUE score of 67, 3 using the largest training set available. We discuss preliminary results and flaws of traditional machine translation evaluation methods for this task and suggest directions for future research.

SemSyn: Semantic-Syntactic Similarity Based Automatic Machine Translation Evaluation Metric

Machine translation evaluation is difficult and challenging for natural languages because different languages behave differently for the same dataset. Lexical-based metrics have been poorly represented semantic relationships and impose strict identity matching. However, translation and assessment become difficult for target morphologically rich languages with relatively free word order. Most of the standard evaluation metrics consider word order but do not effectively consider sentence structure. In this paper, we propose a novel machine translation evaluation metric SemSyn which incorporates both semantic and syntactic similarity. We incorporate the term frequency-inverse document frequency with the earth mover’s distance and word embedding to cover the semantic similarity. The part of speech and dependency parsing tags assist in covering syntactic similarity in the sentence structure. Part of speech and dependency parsing tags are extracted from universal dependencies and trained on the SpaCy library. Experimental results show that SemSyn has a higher correlation with human judgment than other evaluation metrics for morphologically rich language and other languages.

Utilizing Artificial Intelligence Technologies in Saudi EFL Tertiary Level Classrooms

This study focuses on the employment of AI technology in regular, day-to-day activities, such as when Google Translate or Bing Translator are encouraged alongside various programs and applications. It also evaluates and empirically demonstrates the subjects of writing with AI technologies, computer-assisted language learning (CALL), machine translation (MT), and automatic evaluation systems (AESs) in order to offer solutions for enhanced communication training in Saudi Arabia's EFL system. Word tune is an artificial intelligence (AI)-driven writing assistant that can understand the writer's ideas and suggest alternative rewrites (e.g., shorten, expand). This program assists writers of English as a foreign language to maintain a steady flow and acquire useful English expressions. This research made use of questionnaires as a method for collecting data and then ran those responses through SPSS for analysis. The use of artificial intelligence (AI) technology in English as a foreign language (EFL) settings has been shown to facilitate the English language learning (ELT) process and to keep both teachers and students up to date on recent technological developments. This exploratory investigation demonstrated that all digital and AI-powered devices have the potential to assist in teaching and learning. Consequently, the pedagogical component of future education can be developed using an AI framework.

Challenging machine translation engines: some Spanish-English linguistic problems put to the test

This work is an evaluation of machine translation engines completed in 2018 and 2021, inspired by Isabelle, Cherry & Foster (2017), and Isabelle & Kuhn (2018). The challenge consisted of testing MTs Google Translate and Bing and DeepL in the translation of certain linguistic problems normally found when translating from Spanish into English. The divergences representing a “challenge” to the engines were of morphological and lexical-syntactical types. The absolute winner of the challenge was DeepL, in second place was Bing from Microsoft, and Google was the engine that was the poorest in the management of the linguistic problems. In terms of time, when comparing the engines three years apart, it was found that DeepL was the only one that enhanced its performance by correcting a problem it had before in a test sentence. This was not the case for the other two, on the contrary, their translations were of lower quality. These machines do not seem to be consistent in the manner in which they are improved. These findings may be valuable for translators who may work with these systems as pre or post-editors so that their efforts may be better directed.

A Survey on Evaluation Metrics for Machine Translation

The success of Transformer architecture has seen increased interest in machine translation (MT). The translation quality of neural network-based MT transcends that of translations derived using statistical methods. This growth in MT research has entailed the development of accurate automatic evaluation metrics that allow us to track the performance of MT. However, automatically evaluating and comparing MT systems is a challenging task. Several studies have shown that traditional metrics (e.g., BLEU, TER) show poor performance in capturing semantic similarity between MT outputs and human reference translations. To date, to improve performance, various evaluation metrics have been proposed using the Transformer architecture. However, a systematic and comprehensive literature review on these metrics is still missing. Therefore, it is necessary to survey the existing automatic evaluation metrics of MT to enable both established and new researchers to quickly understand the trend of MT evaluation over the past few years. In this survey, we present the trend of automatic evaluation metrics. To better understand the developments in the field, we provide the taxonomy of the automatic evaluation metrics. Then, we explain the key contributions and shortcomings of the metrics. In addition, we select the representative metrics from the taxonomy, and conduct experiments to analyze related problems. Finally, we discuss the limitation of the current automatic metric studies through the experimentation and our suggestions for further research to improve the automatic evaluation metrics.

COVID-19 MT Evaluator: A Platform for the Evaluation of Machine Translation of Public Health Information Related to COVID-19

COVID-19 MT Evaluator: A Platform for the Evaluation of Machine Translation of Public Health Information Related to COVID-19

English Machine translation System Based on Neural Network Algorithm

In order to solve the problems of machine translation efficiency and translation quality, this paper proposes a research method of machine English translation evaluation system based on BP neural network algorithm. This method provides users with a more intelligent machine translation service experience. With the help of BP neural network algorithm, taking English online translation as the research object, Google's translation quality is the best, with an error frequency of only 167, while Baidu translation and iFLYTEK translation in China have a high error rate of 266 and 301 respectively, which is much higher than Google translation. A model of machine translation evaluation system based on neural network algorithm is proposed to better solve the disadvantages of traditional English machine translation. The results show that the machine translation system based on neural network algorithm can further optimize the problems existing in machine translation, such as insufficient use of information and large scale of model parameters, and further improve the performance of neural network machine translation.

Implicit Cross-Lingual Word Embedding Alignment for Reference-Free Machine Translation Evaluation

As we know, cross-lingual word embedding alignment is critically important for reference-free machine translation evaluation, where source texts are directly compared with system translations. In this paper, it is revealed that multilingual knowledge distillation for sentence embedding alignment could achieve cross-lingual word embedding alignment implicitly. A simplified analysis is given to explain the implicit alignment reason. And according to the analysis, it could be deduced that using the last layer embeddings of the distilled student model will have the best alignment effect, which is also validated by the experimental results on the WMT19 datasets. Furthermore, with the assistant of a target-side language model, BERTScore andWord Mover’s Distance using the cross-lingual word embeddings get very competitive results (4 best average scores on 3 types of language directions and ranking first among more than half of all 18 language pairs for the system-level evaluations) in the WMT19’s reference-free machine translation evaluation tasks when the current state-of-the-art (SOTA) metrics are chosen for comparison.

Post-editing Eğitimi, Acemi Post-editörlerin Performansını Etkiliyor mu? Deneysel Bir Çalışma

Machine translation systems led to the creation of a new role for translators: the post-editor. With the birth of neural machine translation systems, the demand for post-editing has been increasing in the recent years, and it has now become a common service given by language service providers and professional translators. Such a change in the landscape of the translation industry might evolve the translation training programs worldwide. It is still heavily discussed whether post-editing and translation skills overlap, and post-editing courses are now included into the curriculum by several translation departments. We set out to investigate whether post-editing training influences the performance of student post-editors in order to explore the necessary background and skills in post-editing tasks. We measured productivity parameters and quality of the final outputs produced by two groups of participants, one of which was previously trained on post-editing. Our results show that, the experimental and control groups did not differ significantly from each other in terms of productivity. There was also little to no difference when we evaluated the post-edited outputs produced by both groups against a reference text using automatic machine translation evaluation metrics. However, we detected a statistical significance between the groups when we analyzed the number of errors in the final output. The post-editors in the experimental group were more aware of the typical errors of machine translation engines.

Machine English Translation Evaluation System Based on BP Neural Network Algorithm.

In order to solve the problems of machine translation efficiency and translation quality, this paper proposes an English translation evaluation system based on the BP neural network algorithm. This method provides users with a more intelligent machine translation service experience. With the help of the BP neural network algorithm, taking English online translation as the research object, Google's translation quality is the best, with an error frequency of only 167, while Baidu translation and iFLYTEK translation in China have a high error rate of 266 and 301, respectively, which is much higher than Google translation. A model of machine translation evaluation based on the neural network algorithm is proposed to better solve the disadvantages of traditional English machine translation. The results show that the machine translation system based on the neural network algorithm can further optimize the problems existing in machine translation, such as insufficient use of information and large scale of model parameters, and further improve the performance of neural network machine translation.

A human evaluation of English-Slovak machine translation

ABSTRACT The paper aims to obtain an error profile for machine translation (MT) from English into Slovak. We present an adjusted framework for MT evaluation, which is based on Vanko's categorical framework, but reflects machine translation peculiarities of synthetic and/or inflectional languages. Based on the framework, we analyse the errors generated by Google Translate and identify the most frequent categories of errors occurring in machine translation when translating newspaper articles from English into Slovak. While we have seen research on widely-spoken languages, such as English or other major official EU languages, little is known about Slovak, which is also an official EU language. This paper provides the first human MT evaluation study of English-Slovak machine translation using professional translators for a more detailed depiction of translation quality. Our research has revealed that the highest numbers of errors occurred in the sphere of lexical semantics, as well as in syntactic-semantic correlativeness, both being closely related. Additionally, based on the results of the Cochran Q test, we show how individual MT errors located in the examined categories differ in co-incidence and in how they impact translation quality.

Evaluation of Machine Translation in English-Chinese Automatic Subtitling of TED Talks

With technology becoming an essential competitive factor for subtitle translation services, the quality of machine subtitle translation in the current era deserves further examination. This paper examines the Chinese translation of 20 Ted Talks subtitled by an automatic subtitling software under the frame of the FAR model. This paper examines if machine subtitle translation has semantic, contextual, grammatical or spelling errors, idiomatic errors, syncopation and inappropriate synchronization, as well as wrong reading speed and paragraph length. It also emphasizes the need to broaden new perspectives on the quality of machine subtitle translation.