Machine Translation Evaluation Research Articles

DiBiMT: A Gold Evaluation Benchmark for Studying Lexical Ambiguity in Machine Translation

Abstract Despite the remarkable progress made in the field of Machine Translation (MT), current systems still struggle when translating ambiguous words, especially when these express infrequent meanings. In order to investigate and analyze the impact of lexical ambiguity on automatic translations, several tasks and evaluation benchmarks have been proposed over the course of the last few years. However, works in this research direction suffer from critical shortcomings. Indeed, existing evaluation datasets are not entirely manually curated, which significantly compromises their reliability. Furthermore, current literature fails to provide detailed insights into the nature of the errors produced by models translating ambiguous words, lacking a thorough manual analysis across languages. With a view to overcoming these limitations, we propose Disambiguation Biases in MT (DiBiMT), an entirely manually-curated evaluation benchmark for investigating disambiguation biases in eight language combinations and assessing the ability of both commercial and non-commercial systems to handle ambiguous words. We also examine and detail the errors produced by models in this scenario by carrying out a manual error analysis in all language pairs. Additionally, we perform an extensive array of experiments aimed at studying the behavior of models when dealing with ambiguous words. Finally, we show the ineffectiveness of standard MT evaluation settings for assessing the disambiguation capabilities of systems and highlight the need for additional efforts in this research direction and ad-hoc testbeds such as DiBiMT.Our benchmark is available at: https://nlp.uniroma1.it/dibimt/.

Cross-Lingual Short-Text Semantic Similarity for Kannada–English Language Pair

Analyzing the semantic similarity of cross-lingual texts is a crucial part of natural language processing (NLP). The computation of semantic similarity is essential for a variety of tasks such as evaluating machine translation systems, quality checking human translation, information retrieval, plagiarism checks, etc. In this paper, we propose a method for measuring the semantic similarity of Kannada–English sentence pairs that uses embedding space alignment, lexical decomposition, word order, and a convolutional neural network. The proposed method achieves a maximum correlation of 83% with human annotations. Experiments on semantic matching and retrieval tasks resulted in promising results in terms of precision and recall.

The link between translation difficulty and the quality of machine translation: a literature review and empirical investigation

We survey the relevant literature on translation difficulty and automatic evaluation of machine translation (MT) quality and investigate whether source text’s translation difficulty features contain any information about MT quality. We analyse the 2017–2019 Conferences on Machine Translation (WMT) data of machine translation quality of English news text translated to eleven different languages (Chinese, Czech, Estonian, Finnish, Latvian, Lithuanian, German, Gujarati, Kazakh, Russian, and Turkish). We find (weak) negative correlation between the source text’s length, polysemy and structural complexity and the corresponding human evaluated quality of machine translation. This suggests a potentially important but measureable influence of source text’s translation difficulty on MT quality.

Calibration and context in human evaluation of machine translation

Abstract Human evaluation of machine translation is considered the “gold standard” for evaluation, but it remains a challenging task for which to define best practices. Recent work has focused on incorporating intersentential context into human evaluation, to better distinguish between high-performing machine translation systems and human translations. In this work, we examine several ways that such context influences evaluation and evaluation protocols. We take a close look at annotator variation through the lens of calibration sets and focus on the implications for context-aware evaluation protocols. We then demonstrate one way in which degraded target-side intersentential context can influence annotator scores of individual sentences, a finding that supports the context-aware approach to evaluation and which also has implications for best practices in evaluation protocols.

A survey of context in neural machine translation and its evaluation

Abstract The question of context in neural machine translation often focuses on topics related to document-level translation or intersentential context. However, there is a wide range of other aspects that can be considered under the umbrella of context. In this work, we survey ways that researchers have incorporated context into neural machine translation systems and the evaluation thereof. This includes building translation systems that operate at the paragraph level or the document level or ones that translate at the sentence level but incorporate information from other sentences. We also consider how issues like terminology consistency, anaphora, and world knowledge or external information can be considered as types of context relevant to the task of machine translation and its evaluation. Closely tied to these topics is the question of how to best evaluate machine translation output in a way that is sensitive to the contexts in which it appears. To this end, we discuss work on incorporating context into both human and automatic evaluations of machine translation quality. Furthermore, we also discuss recent experiments in the field as they relate to the use of large language models in translation and evaluation. We conclude with a view of the future of machine translation, where we expect to see issues of context continue to come to the forefront.

Evaluating NMT using the non-inferiority principle

Abstract The aim of this article is to propose a new neural machine translation (NMT) evaluation method based on the non-inferiority principle. In order to do that, we evaluate raw machine translation (MT) in terms of naturalness, which for this research is defined as not just the lack of fluency errors but also meeting the linguistic expectations of Galician end users when reading original texts in Galician. Our main objective is, in the first place, to validate the new methodology presented in our previous study by evaluating an NMT engine from Spanish into Galician for the social media domain that was retrained with a new Twitter corpus. This new methodology and NMT engine were applied after analyzing the conclusions of a pilot survey conducted among Twitter users to evaluate their perception of tweets translated from Spanish into Galician with our NMT engine created with a corpus of tweets. As in our preliminary study, our aim is to propose a robust quality approximation method based on the reception parameters of end users’ perceptions. This new survey was conducted in December of 2022 with the participation of 228 Galician-speaking Twitter users. Among the main changes proposed are the inclusion of more information about the participant profile, so the non-inferiority principle can be also evaluated according to these parameters; the inclusion of a new typology of tweets, the threads; the provision of context by means of presenting the tweets in their original display as shown in the Twitter app; a change in the number of tweets evaluated and the number of different questionnaires; the change in the distribution of the questionnaires; and the inclusion of an error classification human evaluation conducted by professional linguists to correlate the findings. We will present the steps carried out following the conclusions of the pilot study, describe the new study’s design, analyze the new findings, and present the final conclusions regarding the engine and the evaluation method based on the non-inferiority principle. Finally, we will also provide some examples of the use of this new methodology in the translation industry.

The use of residual analysis to improve the error rate accuracy of machine translation

The aim of the study is to compare two different approaches to machine translation—statistical and neural—using automatic MT metrics of error rate and residuals. We examined four available online MT systems (statistical Google Translate, neural Google Translate, and two European commission’s MT tools—statistical mt@ec and neural eTranslation) through their products (MT outputs). We propose using residual analysis to improve the accuracy of machine translation error rate. Residuals represent a new approach to comparing the quality of statistical and neural MT outputs. The study provides new insights into evaluating machine translation quality from English and German into Slovak through automatic error rate metrics. In the category of prediction and syntactic-semantic correlativeness, statistical MT showed a significantly higher error rate than neural MT. Conversely, in the category of lexical semantics, neural MT showed a significantly higher error rate than statistical MT. The results indicate that relying solely on the reference when determining MT quality is insufficient. However, when combined with residuals, it offers a more objective view of MT quality and facilitates the comparison of statistical MT and neural MT.

Evaluation of Instagram's Neural Machine Translation for Literary Texts: An MQM-Based Analysis

Evaluation of Instagram's Neural Machine Translation for Literary Texts: An MQM-Based Analysis

A comparative analysis of lexical-based automatic evaluation metrics for different Indic language pairs

<p>With the rise of machine translation systems, it has become essential to evaluate the quality of translations produced by these systems. However, the existing evaluation metrics designed for English and other European languages may not always be suitable or apply to other Indic languages due to their complex morphology and syntax. Machine translation evaluation (MTE) is a process of assessing the quality and accuracy of the machine-translated text. MTE involves comparing the machine-translated output with the reference translation to calculate the level of similarity and correctness. Therefore, this study evaluates different metrics, namely, BLEU, METEOR, and TER to identify the most suitable evaluation metric for Indic languages. The study uses datasets for Indic languages and evaluates the metrics on various translation systems. The study contributes to the field of MT by providing insights into suitable evaluation metrics for Indic languages. This research paper aims to study and compare several lexical automatic machine translation evaluation metrics for Indic languages. For this research analysis, we have selected five language pairs of parallel corpora from the low-resource domain, such as English–Hindi, English-Punjabi, English-Gujarati, English-Marathi, and English-Bengali. All these languages belong to the Indo-Aryan language family and are resource-poor. A comparison of the state of art MT is presented and shows which translator works better on these language pairs. For this research work, the natural language toolkit tokenizers are used to assess the analysis of the experimental results. These results have been performed by taking two different datasets for all these language pairs using fully automatic MT evaluation metrics. The research study explores the effectiveness of these metrics in assessing the quality of machine translations between various Indic languages. Additionally, this dataset and analysis will make it easier to do future research in Indian MT evaluation.</p>

Evaluating machine translation of literature through rhetorical analysis

This paper looks at how well ChatGPT and DeepL, two AI tools, translate literary works. Not only can ChatGPT translate text, but it can also carry out other jobs. DeepL is a service that performs computer translation and uses neural networks. The paper looks at how ChatGPT and DeepL translate books, poems, and dialogues compared to translations done by humans. The paper also talks about the pros and cons of using machine translation for literary reasons, including issues of creativity, style, and adapting to different cultures. The paper uses both new and old studies on machine translation technologies and how they work with human translation. The paper comes to the conclusion that ChatGPT and DeepL are useful but imperfect tools for translating literature, and they require human review and improvement. The paper adds to the fields of machine translation and natural language processing by looking at how two cutting-edge AI tools, ChatGPT and DeepL, can be used to translate literary works. The paper also adds to literature studies and digital humanities by looking into what machine translation can and can't do for creative writing and dialog systems. The goal of the paper is to encourage researchers, translators, writers, and users from different fields to work together and talk to each other. ⁤

English translation evaluation method based on BP neural network

In order to solve the problems of machine translation efficiency and translation quality, this paper proposes an English translation evaluation system based on BP neural network algorithm. This method provides users with a more intelligent machine translation service experience. With the help of BP neural network algorithm, taking English online translation as the research object, Google's translation quality is the best, with an error frequency of only 167, while Baidu translation and iFLYTEK translation in China have a high error rate of 266 and 301 respectively, which is much higher than Google translation. A model of machine translation evaluation based on neural network algorithm is proposed to better solve the disadvantages of traditional English machine translation. The results show that the machine translation system based on neural network algorithm can further optimize the problems existing in machine translation, such as insufficient use of information and large scale of model parameters, and further improve the performance of neural network machine translation.

A review of existing Machine Translation Approaches, their Challenges and Evaluation Metrics

Machine translation is the process of translating a natural language into another. The primary goal of machine translation is to bridge the linguistic gap between languages. A significant job that may be utilized is to extract information from texts written in different languages through machine translation (MT). Due to the remarkable developments in Machine Translation over the past several years, we have entered the era of Neural Machine Translation (NMT). A review of MT is essential for better understanding of MT in the domain of Natural Language Processing. This paper focuses on Statistical, Corpus, Example, Rule based, Transfer based, direct based and Nural Machine Translation approach, their challenges, and existing evaluation metrics such as BLEU score and Human evaluation metrics.

Development of English-to-Igala machine translation system using neural machine translation with attention mechanism

Translation is a matter of absolute necessity in the globally united and yet; linguistically and culturally separated world in which we live. Translation allows individuals who speak different languages to communicate and understand one another. The obvious limitations in human translation in terms of speed, scalability and high cost gave birth to the quest for alternative means of translation which is machine translation. Machine translation involves the use of computers to translate from one natural language (source) to another natural language (target). Machine translation applications can be developed using rule-based, example- based and statistical-based technologies. Machine translation applications have been developed to translate English language text to Igala language text using rule-based and example-based approaches. Recently, Neural Machine Translation (NMT) emerged as a new paradigm that swiftly superseded other methods of machine translation evolved with the development of deep learning. This study is aimed at developing a system that can translate contents rendered in English language to Igala language using the Neural Machine Translation with attention mechanism approach. The Neural Machine Translation model for English-to-Igala translation was built with encoder-decoder architecture with an attention layer. The model was trained with a dataset using English-Igala parallel corpus that contains 50,000 parallel sentences. The dataset was partitioned into training, validation and test set in the ratio of 80/20 percent. The model was implemented using python programming language. After training, validation and testing, the output of the model was tested on a corpus of 276 selected English texts using the Bilingual Evaluation Understudy (BLEU) method for evaluating Machine Translation system. An accuracy of 71.0% was obtained. The adoption of this model will play crucial role in facilitating everyday interaction, information exchange and active participation in society among Igala people. It will give Igala people access to information in Igala language that will enhance their standard of living as it will enable them to participate in abundant business opportunities available in the online community, and ensure proper integration in the emerging information society.

Evaluation of Neural Machine translation for conversion of International Classification of disease codes to the Abbreviated injury Scale

The Abbreviated Injury Scale (AIS) is an essential tool for injury research since it allows for comparisons of injury severity among patients, however, the International Classification of Diseases (ICD) is more widely used to capture medical information. The problem of conversion between these two medical coding systems has similarities to the challenges encountered in language translation. We therefore hypothesize that neural machine translation (NMT), a deep learning technique which is commonly used for human language translation, could be used to convert ICD codes to AIS. The objective of this study was to compare the accuracy of a NMT model for determining injury severity compared to two established methods of conversion. The injury severity classifications used for this study were Injury Severity Score (ISS) ≥ 16, Maximum AIS severity (MAIS) ≥ 3, and MAIS ≥ 2. Data from a US national trauma registry, which has patient injuries coded in both AIS and ICD, was used to train a NMT model. Testing data from a separate year was used to determine the accuracy of the NMT model predictions against the actual ISS recorded in the registry. The prediction accuracy of the NMT model was compared to that of the official Association for the Advancement of Automotive Medicine (AAAM) ICD-AIS map and the R package ‘ICD Program for Injury Categorization in R’ (ICDPIC-R). The results show that the NMT model was the most accurate across all injury severity classifications, followed by the ICD-AIS map and then ICDPIC-R package. The NMT model also showed the highest correlation between the predicted and observe ISS scores. Overall, NMT appears to be a promising method for predicting injury severity from ICD codes, however, validation in external databases is needed.

Recovering Word Forms by Context for Morphologically Rich Languages

In this work, we focus on “sentence-level unlemmatization,” the task of generating a grammatical sentence given a lemmatized one; this task is usually easy to do for humans but may present problems for machine learning models. We treat this setting as a machine translation problem and, as a first try, apply a sequence-to-sequence model to the texts of Russian Wikipedia articles, evaluate the effect of the different training sets sizes quantitatively and achieve the BLUE score of 67, 3 using the largest training set available. We discuss preliminary results and flaws of traditional machine translation evaluation methods for this task and suggest directions for future research.

SemSyn: Semantic-Syntactic Similarity Based Automatic Machine Translation Evaluation Metric

Machine translation evaluation is difficult and challenging for natural languages because different languages behave differently for the same dataset. Lexical-based metrics have been poorly represented semantic relationships and impose strict identity matching. However, translation and assessment become difficult for target morphologically rich languages with relatively free word order. Most of the standard evaluation metrics consider word order but do not effectively consider sentence structure. In this paper, we propose a novel machine translation evaluation metric SemSyn which incorporates both semantic and syntactic similarity. We incorporate the term frequency-inverse document frequency with the earth mover’s distance and word embedding to cover the semantic similarity. The part of speech and dependency parsing tags assist in covering syntactic similarity in the sentence structure. Part of speech and dependency parsing tags are extracted from universal dependencies and trained on the SpaCy library. Experimental results show that SemSyn has a higher correlation with human judgment than other evaluation metrics for morphologically rich language and other languages.

Utilizing Artificial Intelligence Technologies in Saudi EFL Tertiary Level Classrooms

This study focuses on the employment of AI technology in regular, day-to-day activities, such as when Google Translate or Bing Translator are encouraged alongside various programs and applications. It also evaluates and empirically demonstrates the subjects of writing with AI technologies, computer-assisted language learning (CALL), machine translation (MT), and automatic evaluation systems (AESs) in order to offer solutions for enhanced communication training in Saudi Arabia's EFL system. Word tune is an artificial intelligence (AI)-driven writing assistant that can understand the writer's ideas and suggest alternative rewrites (e.g., shorten, expand). This program assists writers of English as a foreign language to maintain a steady flow and acquire useful English expressions. This research made use of questionnaires as a method for collecting data and then ran those responses through SPSS for analysis. The use of artificial intelligence (AI) technology in English as a foreign language (EFL) settings has been shown to facilitate the English language learning (ELT) process and to keep both teachers and students up to date on recent technological developments. This exploratory investigation demonstrated that all digital and AI-powered devices have the potential to assist in teaching and learning. Consequently, the pedagogical component of future education can be developed using an AI framework.

Challenging machine translation engines: some Spanish-English linguistic problems put to the test

This work is an evaluation of machine translation engines completed in 2018 and 2021, inspired by Isabelle, Cherry & Foster (2017), and Isabelle & Kuhn (2018). The challenge consisted of testing MTs Google Translate and Bing and DeepL in the translation of certain linguistic problems normally found when translating from Spanish into English. The divergences representing a “challenge” to the engines were of morphological and lexical-syntactical types. The absolute winner of the challenge was DeepL, in second place was Bing from Microsoft, and Google was the engine that was the poorest in the management of the linguistic problems. In terms of time, when comparing the engines three years apart, it was found that DeepL was the only one that enhanced its performance by correcting a problem it had before in a test sentence. This was not the case for the other two, on the contrary, their translations were of lower quality. These machines do not seem to be consistent in the manner in which they are improved. These findings may be valuable for translators who may work with these systems as pre or post-editors so that their efforts may be better directed.

A Survey on Evaluation Metrics for Machine Translation

The success of Transformer architecture has seen increased interest in machine translation (MT). The translation quality of neural network-based MT transcends that of translations derived using statistical methods. This growth in MT research has entailed the development of accurate automatic evaluation metrics that allow us to track the performance of MT. However, automatically evaluating and comparing MT systems is a challenging task. Several studies have shown that traditional metrics (e.g., BLEU, TER) show poor performance in capturing semantic similarity between MT outputs and human reference translations. To date, to improve performance, various evaluation metrics have been proposed using the Transformer architecture. However, a systematic and comprehensive literature review on these metrics is still missing. Therefore, it is necessary to survey the existing automatic evaluation metrics of MT to enable both established and new researchers to quickly understand the trend of MT evaluation over the past few years. In this survey, we present the trend of automatic evaluation metrics. To better understand the developments in the field, we provide the taxonomy of the automatic evaluation metrics. Then, we explain the key contributions and shortcomings of the metrics. In addition, we select the representative metrics from the taxonomy, and conduct experiments to analyze related problems. Finally, we discuss the limitation of the current automatic metric studies through the experimentation and our suggestions for further research to improve the automatic evaluation metrics.

COVID-19 MT Evaluator: A Platform for the Evaluation of Machine Translation of Public Health Information Related to COVID-19

COVID-19 MT Evaluator: A Platform for the Evaluation of Machine Translation of Public Health Information Related to COVID-19