Automatic Machine Translation Evaluation Research Articles

A comparative analysis of lexical-based automatic evaluation metrics for different Indic language pairs

<p>With the rise of machine translation systems, it has become essential to evaluate the quality of translations produced by these systems. However, the existing evaluation metrics designed for English and other European languages may not always be suitable or apply to other Indic languages due to their complex morphology and syntax. Machine translation evaluation (MTE) is a process of assessing the quality and accuracy of the machine-translated text. MTE involves comparing the machine-translated output with the reference translation to calculate the level of similarity and correctness. Therefore, this study evaluates different metrics, namely, BLEU, METEOR, and TER to identify the most suitable evaluation metric for Indic languages. The study uses datasets for Indic languages and evaluates the metrics on various translation systems. The study contributes to the field of MT by providing insights into suitable evaluation metrics for Indic languages. This research paper aims to study and compare several lexical automatic machine translation evaluation metrics for Indic languages. For this research analysis, we have selected five language pairs of parallel corpora from the low-resource domain, such as English–Hindi, English-Punjabi, English-Gujarati, English-Marathi, and English-Bengali. All these languages belong to the Indo-Aryan language family and are resource-poor. A comparison of the state of art MT is presented and shows which translator works better on these language pairs. For this research work, the natural language toolkit tokenizers are used to assess the analysis of the experimental results. These results have been performed by taking two different datasets for all these language pairs using fully automatic MT evaluation metrics. The research study explores the effectiveness of these metrics in assessing the quality of machine translations between various Indic languages. Additionally, this dataset and analysis will make it easier to do future research in Indian MT evaluation.</p>

Post-editing Eğitimi, Acemi Post-editörlerin Performansını Etkiliyor mu? Deneysel Bir Çalışma

Machine translation systems led to the creation of a new role for translators: the post-editor. With the birth of neural machine translation systems, the demand for post-editing has been increasing in the recent years, and it has now become a common service given by language service providers and professional translators. Such a change in the landscape of the translation industry might evolve the translation training programs worldwide. It is still heavily discussed whether post-editing and translation skills overlap, and post-editing courses are now included into the curriculum by several translation departments. We set out to investigate whether post-editing training influences the performance of student post-editors in order to explore the necessary background and skills in post-editing tasks. We measured productivity parameters and quality of the final outputs produced by two groups of participants, one of which was previously trained on post-editing. Our results show that, the experimental and control groups did not differ significantly from each other in terms of productivity. There was also little to no difference when we evaluated the post-edited outputs produced by both groups against a reference text using automatic machine translation evaluation metrics. However, we detected a statistical significance between the groups when we analyzed the number of errors in the final output. The post-editors in the experimental group were more aware of the typical errors of machine translation engines.

An English Translation Quality Evaluation Model Integrating Knowledge Transfer and Wireless Network

Machine translation is to automatically translate one natural language text into another. It is one of the important applications of natural language processing. The research of machine translation is closely related to machine translation evaluation technology. Therefore, with the progress of machine translation technology in recent years, researchers are also committed to designing more reasonable evaluation methods. Manual evaluation is costly, time-consuming, and subjective, so it has been difficult to provide practical help for the development of machine translation systems. Finding a feasible automatic machine translation evaluation method has been a hot spot in machine translation evaluation research in recent years. Based on this, this paper proposes an English translation quality evaluation model integrating knowledge transfer and wireless network. Firstly, wireless network technology is used to coordinate multitasks, and then knowledge transfer optimization technology is used to construct an English translation quality evaluation model. In this paper, simulation experiments are designed to verify the effectiveness of the model. The experimental results show that the accuracy of the evaluation method based on fusion technology is significantly improved compared with the existing evaluation methods.

Statistical Analysis of Machine Translation Evaluation Systems for English- Hindi Language Pair

Background: Automatic Machine Translation (AMT) Evaluation Metrics have become popular in the Machine Translation Community in recent times. This is because of the popularity of Machine Translation engines and Machine Translation as a field itself. Translator is a very important tool to break barriers between communities especially in countries like India, where people speak 22 different languages and their many variations. With the onset of Machine Translation engines, there is a need for a system that evaluates how well these are performing. This is where machine translation evaluation enters. Objective: This paper discusses the importance of Automatic Machine Translation Evaluation and compares various Machine Translation Evaluation metrics by performing Statistical Analysis on various metrics and human evaluations to find out which metric has the highest correlation with human scores. Methods: The correlation between the Automatic and Human Evaluation Scores and the correlation between the five Automatic evaluation scores are examined at the sentence level. Moreover, a hypothesis is set up and p-values are calculated to find out how significant these correlations are. Results: The results of the statistical analysis of the scores of various metrics and human scores are shown in the form of graphs to see the trend of the correlation between the scores of Automatic Machine Translation Evaluation metrics and human scores. Conclusion: Out of the five metrics considered for the study, METEOR shows the highest correlation with human scores as compared to the other metrics.

Adaptation in Statistical Machine Translation for Low-resource Domains in English-Vietnamese Language

Adaptation in Statistical Machine Translation for Low-resource Domains in English-Vietnamese Language

Metric for Automatic Machine Translation Evaluation based on Pre-trained Sentence Embeddings

Metric for Automatic Machine Translation Evaluation based on Pre-trained Sentence Embeddings

Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

Automatic Machine Translation (MT) evaluation is an active field of research, with a handful of new metrics devised every year. Evaluation metrics are generally benchmarked against manual assessment of translation quality, with performance measured in terms of overall correlation with human scores. Much work has been dedicated to the improvement of evaluation metrics to achieve a higher correlation with human judgments. However, little insight has been provided regarding the weaknesses and strengths of existing approaches and their behavior in different settings. In this work we conduct a broad meta-evaluation study of the performance of a wide range of evaluation metrics focusing on three major aspects. First, we analyze the performance of the metrics when faced with different levels of translation quality, proposing a local dependency measure as an alternative to the standard, global correlation coefficient. We show that metric performance varies significantly across different levels of MT quality: Metrics perform poorly when faced with low-quality translations and are not able to capture nuanced quality distinctions. Interestingly, we show that evaluating low-quality translations is also more challenging for humans. Second, we show that metrics are more reliable when evaluating neural MT than the traditional statistical MT systems. Finally, we show that the difference in the evaluation accuracy for different metrics is maintained even if the gold standard scores are based on different criteria.

Vergleich der menschlichen, maschinellen und Post-Editing-Übersetzung aus dem Slowakischen ins Deutsche mittels automatischer Evaluation

SummaryThe aim of the study is to compare translation quality and effectiveness (translator productivity) using measures of the automatic evaluation of machine translation output. The examined translation(s) was a legal text, translated from Slovak (mother tongue) into German. We distinguish human translation (HT), machine translation (MT) and post-edited MT (PEMT). For the evaluation we used our own tool, wherein were implemented the metrics of automatic MT evaluation. Analysis of Variance and Multiply Comparisons (Tukey HSD test) were applied to textual data. The results proved that in terms of productivity, regarding the expert and terminological knowledge of human translators, post-editing of MT is more effective than HT. On the other hand, in terms of quality (sentence structure or stylistics), regardless of time, HT is of a higher quality then MT, but similar to PEMT. The post-editors were affected by grammar and by stylistics of MT output, but post-edited a larger volume of MT than human translators translated.

Analysis and evaluation on the quality of news text machine translation based on neural network

Machine translation quality estimation is an important task in natural language processing. Unlike traditional automatic machine translation evaluation methods, the quality of machine translation is evaluated by the translation quality estimation method without using manual reference translation. In view of the fact that the feature extraction of current sentence-level machine translation quality estimation relies heavily on linguistic analysis, which leads to insufficient generalization ability and restricts the performance of subsequent support vector regression algorithm, it is proposed to extract the features of sentence vectors by using contextual word prediction model and matrix decomposition model in deep learning, and combine them with the features of recurrent neural network language model to improve the correlation between automatic estimation of translation quality and manual evaluation. The experimental results on WMT15 and WMT16 translation quality estimation subtask data sets show that the performance statistics of the method of extracting sentence vector features by adopting context word prediction model are consistently superior to that of the traditional Quest method and continuous space language model sentence vector feature extraction method. It reveals that the proposed feature extraction method not only requires no linguistic analysis, but also significantly improves the effect of translation quality estimation.

Optimizing Automatic Evaluation of Machine Translation with the ListMLE Approach

Automatic evaluation of machine translation is critical for the evaluation and development of machine translation systems. In this study, we propose a new model for automatic evaluation of machine translation. The proposed model combines standard n-gram precision features and sentence semantic mapping features with neural features, including neural language model probabilities and the embedding distances between translation outputs and their reference translations. We optimize the model with a representative list-wise learning to rank approach, ListMLE, in terms of human ranking assessments. The experimental results on WMT’2015 Metrics task indicated that the proposed approach yields significantly better correlations with human assessments than several state-of-the-art baseline approaches. In particular, the results confirmed that the proposed list-wise learning to rank approach is useful and powerful for optimizing automatic evaluation metrics in terms of human ranking assessments. Deep analysis also demonstrated that optimizing automatic metrics with the ListMLE approach is a reasonable method and adding the neural features can gain considerable improvements compared with the traditional features.

VERTa: a linguistic approach to automatic machine translation evaluation

Machine translation (MT) is directly linked to its evaluation in order to both compare different MT system outputs and analyse system errors so that they can be addressed and corrected. As a consequence, MT evaluation has become increasingly important and popular in the last decade, leading to the development of MT evaluation metrics aiming at automatically assessing MT output. Most of these metrics use reference translations in order to compare system output, and the most well-known and widely spread work at lexical level. In this study we describe and present a linguistically-motivated metric, VERTa, which aims at using and combining a wide variety of linguistic features at lexical, morphological, syntactic and semantic level. Before designing and developing VERTa a qualitative linguistic analysis of data was performed so as to identify the linguistic phenomena that an MT metric must consider (Comelles et al. 2017). In the present study we introduce VERTa’s design and architecture and we report the experiments performed in order to develop the metric and to check the suitability and interaction of the linguistic information used. The experiments carried out go beyond traditional correlation scores and step towards a more qualitative approach based on linguistic analysis. Finally, in order to check the validity of the metric, an evaluation has been conducted comparing the metric’s performance to that of other well-known state-of-the-art MT metrics.

Development of machine translation technology for assisting health communication: A systematic review.

To (1) characterize how machine translation (MT) is being developed to overcome language barriers in health settings; and (2) based on evaluations presented in the literature, determine which MT approaches show evidence of promise and what steps need to be taken to encourage adoption of MT technologies in health settings. We performed a systematic literature search covering 2006-2016 in major health, engineering, and computer science databases. After removing duplicates, two levels of screening identified 27 articles for full text review and analysis. Our review and qualitative analysis covered application setting, target users, underlying technology, whether MT was used in isolation or in combination with human editing, languages tested, evaluation methods, findings, and identified gaps. Of 27 studies, a majority focused on MT systems for use in clinical settings (n = 18), and eight of these involved speech-based MT systems for facilitating patient-provider communications. Text-based MT systems (n = 19) aimed at generating a range of multilingual health materials. Almost a third of all studies (n = 8) pointed to MT's potential as a starting point before human input. Studies employed a variety of human and automatic MT evaluation methods. In comparison studies, statistical machine translation (SMT) systems were more accurate than rule-based systems when large corpora were available. For a variety of systems, performance was best for translations of simple, less technical sentences and from English to Western European languages. Only one system has been fully deployed. MT is currently being developed primarily through pilot studies to improve multilingual communication in health settings and to increase access to health resources for a variety of languages. However, continued concerns about accuracy limit the deployment of MT systems in these settings. The variety of piloted systems and the lack of shared evaluation criteria will likely continue to impede adoption in health settings, where excellent accuracy and a strong evidence base are critical. Greater translation accuracy and use of standard evaluation criteria would encourage deployment of MT into health settings. For now, the literature points to using MT in health communication as an initial step to be followed by human correction.

Guiding automatic MT evaluation by means of linguistic features

Machine translation (MT) has become increasingly important and popular in the past decade, leading to the development of MT evaluation metrics aiming at automatically assessing MT output. Most of these metrics use reference translations to compare systems output, therefore, they should not only detect MT errors but also be able to identify correct equivalent expressions so as not to penalize them when those are not displayed in the reference translations. With the aim of improving MT evaluation metrics a study has been carried out of a wide panorama of linguistic features and their implications. For that purpose a Spanish and an English corpora containing hypothesis and reference translations have been analysed from a linguistic point of view, so that common errors can be detected and positive equivalencies highlighted. This article focuses on this qualitative analysis describing the linguistic phenomena that should be considered when developing an automatic MT evaluation metric. The results of this analysis have been used to develop an automatic MT evaluation metric that takes into account different dimensions of language. A brief review of the metric and its evaluation are also provided.

Comparative Study Between METEOR and BLEU Methods of MT: Arabic into English Translation as a Case Study

The Internet provides its users with a variety of services, and these services include free online machine translators, which translate free of charge between many of the world's languages such as Arabic, English, Chinese, German, Spanish, French, Russian, etc. Machine translators facilitate the transfer of information between different languages, thus eliminating the language barrier, since the amount of information and knowledge available varies from one language to another, Arabic content on the internet, for example, accounts 1% of the total internet content, while Arabs constitute 5% of the population of the earth, which means that the intellectual productivity of the Arabs is low because within internet use Internet's Arabic content represents 20% of their natural proportion, which in turn encouraged some Arab parties to improve Arabic content within the internet. So, many of those interested specialists rely on machine translators to bridge the knowledge gap between the information available in the Arabic language and those in other living languages such as English. This empirical study aims to identify the best Arabic to English Machine translation system, in order to help the developers of these systems to enhance the effectiveness of these systems. Furthermore, such studies help the users to choose the best. This study involves the construction of a system for Automatic Machine Translation Evaluation System of the Arabic language into language. This study includes assessing the accuracy of the translation by the two known machine translators, Google Translate, and the second, which bears the name of Babylon machine translation from Arabic into English. BLEU and METEOR methods are used the MT quality, and to identify the closer method to human judgments. The authors conclude that BLEU is closer to human judgments METEOR method.

Automated MT evaluation metrics and their limitations

This paper gives a general overview of the main classes of methods for automatic evaluation of Machine Translation (MT) quality, their limitations and their value for professional translators and MT developers. Automated evaluation of MT characterizes performance of MT systems on specific text or a corpus. Automated scores are expected to correlate with certain parameters of MT quality scored by human evaluators, such as adequacy of fluency of translation. Automated evaluation is now part of MT development cycle, but it also contributes to fundamental research on MT and improving MT technology..

Research on the Machine Translation Based on the Hybrid Approach

This paper firstly introduces the recent research on machine translation and describes the hybrid strategies on machine translation in detail, and discusses the applications of machine learning for machine translation. The hybrid approach is a method for translation which integrates an example-pattern-based method and a rule-based method, to the design and implementation of an English-Chinese machine translation system. It focuses discussion on language model, knowledge base, design ideas and implementation strategies. Our system has been tested based on requirement details listed in the Outlines for Automatic Evaluation of Machine Translation constituted by National Hi-Tech Project, and compared with the assigned system. Experiment results indicate that our system has high translation speed and accuracy.

Machine Translation Based on Translation Rules for Processing Natural Language

This paper presents a hybrid approach, which integrates an example-pattern-based method and a rule-based method, to the design and implementation of an English-Chinese machine translation system. It focuses discussion on language model , knowledge base, design ideas and implementation strategies. Our system has been tested based on requirement details listed in the Outlines for Automatic Evaluation of Machine Translation constituted by National Hi-Tech Project 863, and compared with the Huajian system. Experiment results indicate that our system has high translation speed and accuracy.

O avtomatski evalvaciji strojnega prevajanja

Stalen del razvoja strojnega prevajanja je evalvacija prevodov, pri čemer se v glavnem uporabljajo avtomatski postopki. Ti vedno temeljijo na referenčnem prevodu. V tem prispevku pokažemo, kako zelo različni so lahko referenčni prevodi za področje podnaslavljanja ter kako lahko to vpliva na oceno – ista metrika lahko isti prevajalnik oceni kot neuporaben ali kot zelo uspešen samo na podlagi tega, da uporabimo referenčne prevode, ki so pridobljeni po različnih postopkih, vendar vedno jezikovno in pomensko povsem ustrezni.

Extractive summarization using complex networks and syntactic dependency

The realization that statistical physics methods can be applied to analyze written texts represented as complex networks has led to several developments in natural language processing, including automatic summarization and evaluation of machine translation. Most importantly, so far only a few metrics of complex networks have been used and therefore there is ample opportunity to enhance the statistics-based methods as new measures of network topology and dynamics are created. In this paper, we employ for the first time the metrics betweenness, vulnerability and diversity to analyze written texts in Brazilian Portuguese. Using strategies based on diversity metrics, a better performance in automatic summarization is achieved in comparison to previous work employing complex networks. With an optimized method the Rouge score (an automatic evaluation method used in summarization) was 0.5089, which is the best value ever achieved for an extractive summarizer with statistical methods based on complex networks for Brazilian Portuguese. Furthermore, the diversity metric can detect keywords with high precision, which is why we believe it is suitable to produce good summaries. It is also shown that incorporating linguistic knowledge through a syntactic parser does enhance the performance of the automatic summarizers, as expected, but the increase in the Rouge score is only minor. These results reinforce the suitability of complex network methods for improving automatic summarizers in particular, and treating text in general.

Linguistic measures for automatic machine translation evaluation

Assessing the quality of candidate translations involves diverse linguistic facets. However, most automatic evaluation methods in use today rely on limited quality assumptions, such as lexical simi...