Machine Translation Evaluation Research Articles

The Quality of Machine Translation Assessment On Gender Markers Lingual Units

Machine Translation (MT) is one of the most advanced and elaborate research fields within Translation Technology, the quality of MT output has always been a great concern, and MT evaluation is a popular research topic. This research aims to assess the quality translation on the gender markers lingual unit of the Arabic short story "عَبْدُ اللهِ وَالْعُصْفُوْرُ" which is translated to English and Indonesian using Machine Translation. The research was qualitatively-based method. The subject is gender markers lingual unit that taken from the Arabic short story "عَبْدُ اللهِ وَالْعُصْفُوْرُ". The key instrument of this research is human instrument. Additional instruments used to support this research consisted tables of the lingual units of gender markers and table of rating scales based on Nababans' theory (2012). The research findings showed that this analysis has discovered were 72 lingual units of gender markers in the short story. Further, the dominant type was personal pronoun. Based on the results, it can be concluded that the google translate translation on the gender markers lingual units have the high quality on the accuracy, acceptance, and readability level.

Bridging the gap between machine translation output and images in multimodal documents

The aim of this article is to report on recent findings concerning the use of Google Translate outputs in multimodal contexts. Development and evaluation of machine translation often focus on verbal mode, but accounts by the area on the exploration text-image relations in multimodal documents translated automatically are rare. Thus, this work seeks to describe just what are such relations and how to describe them, organized in two parts: firstly, by exploring the problem through an interdisciplinary interface, involving Machine Translation and Multimodality to analyze some examples from the Wikihow website; secondly, by reporting on recent investigation on suitable tools and methods to properly annotate these issues from within a long-term purpose to assemble a corpus. Finally, this article provides a discussion on the findings, including some limitations and perspectives for future research.

Innovatively Fused Deep Learning with Limited Noisy Data for Evaluating Translations from Poor into Rich Morphology

Evaluation of machine translation (MT) into morphologically rich languages has not been well studied despite its importance. This paper proposes a classifier, that is, a deep learning (DL) schema for MT evaluation, based on different categories of information (linguistic features, natural language processing (NLP) metrics and embeddings), by using a model for machine learning based on noisy and small datasets. The linguistic features are string based for the language pairs English (EN)–Greek (EL) and EN–Italian (IT). The paper also explores the linguistic differences that affect evaluation accuracy between different kinds of corpora. A comparative study between using a simple embedding layer (mathematically calculated) and pre-trained embeddings is conducted. Moreover, an analysis of the impact of feature selection and dimensionality reduction on classification accuracy has been conducted. Results show that using a neural network (NN) model with different input representations produces results that clearly outperform the state-of-the-art for MT evaluation for EN–EL and EN–IT, by an increase of almost 0.40 points in correlation with human judgments on pairwise MT evaluation. It is observed that the proposed algorithm achieved better results on noisy and small datasets. In addition, for a more integrated analysis of the accuracy results, a qualitative linguistic analysis has been carried out in order to address complex linguistic phenomena.

Post-Editing Oriented Human Quality Evaluation of Neural Machine Translation in Translator Training: A Study on Perceived Difficulties and Benefits

The aim of this article is to investigate translation trainees’ perceived difficulties and benefits of a post-editing oriented neural machine translation (NMT) error annotation and quality evaluation task which was carried out for the language pair English-Turkish and for two separate text types and domains, i.e., environmental blogs and movie/TV reviews, within the scope of an MA course on translation quality standards. The data to be analyzed were collected through a semi-structured questionnaire which was given to the trainees attending the course after the completion of the task. The questionnaire was prepared with the aim of understanding perceived difficulties and benefits of the task. Analysis of the answers revealed that most of the trainees were in the opinion that the task was difficult. Majority of the trainees also believed that the task was beneficial and enabled them to feel empowered to make decisions on human translation quality evaluation of machine translation (MT) and to carry out error annotation and post-editing activities in the future. According to a significant number of trainees, error annotation facilitated their post-editing process and reduced the effort in post-editing. Enhanced understanding of MT error annotation and enhanced ability to perform post-editing were the significant benefits stated by the trainees. The difficulties were associated with being introduced and assigned to perform tasks they were not familiar with. Yet, as displayed in the answers to the questionnaire, a considerable majority of the trainees were positive about the learning experience. The results have also shown that integration of MT-related activities into translator training with a focus on the empowerment of the human translator has its difficulties and benefits also for translation trainers. While the difficulties for the trainers concern the decisions on the design, implementation, and planning of the task and the responsibility to carry out the task in constant interaction and collaboration with the trainees, the benefits are the sense of fulfillment and enrichment brought by positive feedback from the trainees and the discovery of the fact that the so-called ‘teaching experience’ becomes a ‘learning experience’ for trainers as well as trainees.

Detecting Source Contextual Barriers for Understanding Neural Machine Translation

In machine translation evaluation, the traditional wisdom measures model's generalization ability in an average sense, for example by using corpus BLEU. However, the statistics of corpus BLEU cannot provide comprehensive understanding and fine-grained analysis on model's generalization ability. As a remedy, this paper attempts to understand NMT at fine-grained level, by detecting contextual barriers within an unseen input sentence that cause the degradation in model's translation quality. It proposes a principled definition of source contextual barriers as well as its modified version which is tractable in computation and operates at word-level. Based on the modified one, three simple methods are proposed for barrier detection by search-aware risk estimation through counterfactual generation. Extensive analyses are conducted on those detected contextual barrier words on both Zh $\Leftrightarrow$ En NIST benchmarks. Potential usages motivated from barrier words are also discussed.

A comprehensive understanding of popular machine translation evaluation metrics

A comprehensive understanding of popular machine translation evaluation metrics

A Crowdsourcing-Based Framework for the Development and Validation of Machine Readable Parallel Corpus for Sign Languages

Sign languages are used by the deaf and mute community of the world. These are gesture based languages where the subjects use hands and facial expressions to perform different gestures. There are hundreds of different sign languages in the world. Furthermore, like natural languages, there exist different dialects for many sign languages. In order to facilitate the deaf community several different repositories of video gestures are available for many sign languages of the world. These video based repositories do not support the development of an automated language translation systems. This research aims to investigate the idea of engaging the deaf community for the development and validation of a parallel corpus for a sign language and its dialects. As a principal contribution, this research presents a framework for building a parallel corpus for sign languages by harnessing the powers of crowdsourcing with editorial manager, thus it engages a diversified set of stakeholders for building and validating a repository in a quality controlled manner. It further presents processes to develop a word-level parallel corpus for different dialects of a sign language; and a process to develop sentence-level translation corpus comprising of source and translated sentences. The proposed framework has been successfully implemented and involved different stakeholders to build corpus. As a result, a word-level parallel corpus comprising of the gestures of almost 700 words of Pakistan Sign Language (PSL) has been developed. While, a sentence-level translation corpus comprising of more than 8000 sentences for different tenses has also been developed for PSL. This sentence-level corpus can be used in developing and evaluating machine translation models for natural to sign language translation and vice-versa. While the machine-readable word level parallel corpus will help in generating avatar based videos for the translated sentences in different dialects of a sign language.

DiaBLa: a corpus of bilingual spontaneous written dialogues for machine translation

We present a new English–French dataset for the evaluation of Machine Translation (MT) for informal, written bilingual dialogue. The test set contains 144 spontaneous dialogues (5700+ sentences) between native English and French speakers, mediated by one of two neural MT systems in a range of role-play settings. The dialogues are accompanied by fine-grained sentence-level judgments of MT quality, produced by the dialogue participants themselves, as well as by manually normalised versions and reference translations produced a posteriori. The motivation for the corpus is twofold: to provide (i) a unique resource for evaluating MT models, and (ii) a corpus for the analysis of MT-mediated communication. We provide an initial analysis of the corpus to confirm that the participants’ judgments reveal perceptible differences in MT quality between the two MT systems used.

Statistical Analysis of Machine Translation Evaluation Systems for English- Hindi Language Pair

Background: Automatic Machine Translation (AMT) Evaluation Metrics have become popular in the Machine Translation Community in recent times. This is because of the popularity of Machine Translation engines and Machine Translation as a field itself. Translator is a very important tool to break barriers between communities especially in countries like India, where people speak 22 different languages and their many variations. With the onset of Machine Translation engines, there is a need for a system that evaluates how well these are performing. This is where machine translation evaluation enters. Objective: This paper discusses the importance of Automatic Machine Translation Evaluation and compares various Machine Translation Evaluation metrics by performing Statistical Analysis on various metrics and human evaluations to find out which metric has the highest correlation with human scores. Methods: The correlation between the Automatic and Human Evaluation Scores and the correlation between the five Automatic evaluation scores are examined at the sentence level. Moreover, a hypothesis is set up and p-values are calculated to find out how significant these correlations are. Results: The results of the statistical analysis of the scores of various metrics and human scores are shown in the form of graphs to see the trend of the correlation between the scores of Automatic Machine Translation Evaluation metrics and human scores. Conclusion: Out of the five metrics considered for the study, METEOR shows the highest correlation with human scores as compared to the other metrics.

Adaptation in Statistical Machine Translation for Low-resource Domains in English-Vietnamese Language

Adaptation in Statistical Machine Translation for Low-resource Domains in English-Vietnamese Language

Quality and Machine Translation: An Evaluation of Online Machine Translation of English into Arabic Texts

This study compares the translation outputs of an English into Arabic text using the three machine translators of Google Translate, Microsoft Bing, and Ginger. To carry this evaluation of the machine translation (MT) outputs, an English text and its Arabic counterpart were selected from the UN records. The English source text was segmented into 84 semantic chunks. Depending on the Arabic counterpart model text, each chunk was rated as “correct or incorrect” at the two levels of the translation attributes: fidelity and intelligibility. To perform the quantitative description of the evaluation process, the numbers of fidelity and intelligibility errors and their percentages were calculated. Results of this evaluation process revealed that none of the three translated versions of the source text was perfectly translated. Although the translation of Microsoft Bing was rated the best, Google’s translation was found the least accurate due to the high percentage of fidelity and intelligibility errors detected in its translation output. However, the quality of Ginger’s translation was found slightly less accurate than that of Microsoft Bing, but remarkably better than Google’s translation. The findings of this study imply that these MT applications can be implemented to perform English into Arabic translation to get the broad gist of a source text, but a deep and thorough post-editing process looks essential for a full and accurate understanding of an English into Arabic MT output. The study recommends that more studies are encouraged to continue to assess the quality of MT that will further highlight its weaknesses and the strategies that should be adopted to overcome them.

기계번역 프로그램 품질에 대한 사용자 평가와 사용자의 L2 수준 간 상관관계 고찰 - 한중 언어 쌍을 중심으로 -

While the quality of machine translation is getting better and better, it is still not perfect. In this case, how the user treats the imperfect machine translation results determines whether the user achieves the purpose for using machine translation. In this process, user evaluations are a key factor. User evaluations are not objective nor accurate. User evaluations are a subjective evaluation of the user and is related to the user. Therefore, this paper attempts to analyze the relationship between the level of the user's L2 and user evaluations. This paper surveyed 69 Chinese users to understand the current status of their use of machine translation, including frequency of use, purpose of use, favorite machine translation, reasons for preferences, and satisfaction with the quality of the machine translation used. At the same time, for specific translation articles, it lets users evaluate the accuracy and fluency of machine translation articles. The survey results show that users with low L2 levels used machine translation at a higher frequency and were more inclined to evaluate machine translation from the perspective of ease of use. In addition, statistical analysis of the evaluation results found that users’ evaluation of machine translation was related to the user's own level L2. The lower the L2 level, the higher the evaluation of the adequacy and fluency of machine translation, and the higher the assessment of the overall quality of the machine translation.

Metric for Automatic Machine Translation Evaluation based on Pre-trained Sentence Embeddings

Metric for Automatic Machine Translation Evaluation based on Pre-trained Sentence Embeddings

How to evaluate machine translation: A review of automated and human metrics

AbstractThis article presents the most up-to-date, influential automated, semiautomated and human metrics used to evaluate the quality of machine translation (MT) output and provides the necessary background for MT evaluation projects. Evaluation is, as repeatedly admitted, highly relevant for the improvement of MT. This article is divided into three parts: the first one is dedicated to automated metrics; the second, to human metrics; and the last, to the challenges posed by neural machine translation (NMT) regarding its evaluation. The first part includes reference translation–based metrics; confidence or quality estimation (QE) metrics, which are used as alternatives for quality assessment; and diagnostic evaluation based on linguistic checkpoints. Human evaluation metrics are classified according to the criterion of whether human judges directly express a so-called subjective evaluation judgment, such as ‘good’ or ‘better than’, or not, as is the case in error classification. The former methods are based on directly expressed judgment (DEJ); therefore, they are called ‘DEJ-based evaluation methods’, while the latter are called ‘non-DEJ-based evaluation methods’. In the DEJ-based evaluation section, tasks such as fluency and adequacy annotation, ranking and direct assessment (DA) are presented, whereas in the non-DEJ-based evaluation section, tasks such as error classification and postediting are detailed, with definitions and guidelines, thus rendering this article a useful guide for evaluation projects. Following the detailed presentation of the previously mentioned metrics, the specificities of NMT are set forth along with suggestions for its evaluation, according to the latest studies. As human translators are the most adequate judges of the quality of a translation, emphasis is placed on the human metrics seen from a translator-judge perspective to provide useful methodology tools for interdisciplinary research groups that evaluate MT systems.

Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments

Automatic Machine Translation (MT) evaluation is an active field of research, with a handful of new metrics devised every year. Evaluation metrics are generally benchmarked against manual assessment of translation quality, with performance measured in terms of overall correlation with human scores. Much work has been dedicated to the improvement of evaluation metrics to achieve a higher correlation with human judgments. However, little insight has been provided regarding the weaknesses and strengths of existing approaches and their behavior in different settings. In this work we conduct a broad meta-evaluation study of the performance of a wide range of evaluation metrics focusing on three major aspects. First, we analyze the performance of the metrics when faced with different levels of translation quality, proposing a local dependency measure as an alternative to the standard, global correlation coefficient. We show that metric performance varies significantly across different levels of MT quality: Metrics perform poorly when faced with low-quality translations and are not able to capture nuanced quality distinctions. Interestingly, we show that evaluating low-quality translations is also more challenging for humans. Second, we show that metrics are more reliable when evaluating neural MT than the traditional statistical MT systems. Finally, we show that the difference in the evaluation accuracy for different metrics is maintained even if the gold standard scores are based on different criteria.

A Better Gauging Model for the Evaluation of Automatic Machine Translation of English – Hindi Language

The problem of language translation has prevailed in society for so long. However, up to some extent the problem is being reduced by the online available machine translation systems like Google, Bing, Babelfish, etc. But with the emergence of these Machine Translation Systems, there arises the problem of their validation. Can we trust on such translation systems blindly? Is there no scope of improvement? Are these Machine Translation systems not prone to errors? The answer to all these questions is No. So, for this purpose, we need a mechanism that can test or assess these Machine Translation systems. In this paper, we have proposed an algorithm that will evaluate such Machine Translation systems. Our algorithm is being compared with a very well-known BLEU algorithm that works very well for non-Indian languages. The accuracy of the designed algorithm is evaluated using the standard datasets like Tides and EMILLE.

Neural machine translation of clinical texts between long distance languages.

To analyze techniques for machine translation of electronic health records (EHRs) between long distance languages, using Basque and Spanish as a reference. We studied distinct configurations of neural machine translation systems and used different methods to overcome the lack of a bilingual corpus of clinical texts or health records in Basque and Spanish. We trained recurrent neural networks on an out-of-domain corpus with different hyperparameter values. Subsequently, we used the optimal configuration to evaluate machine translation of EHR templates between Basque and Spanish, using manual translations of the Basque templates into Spanish as a standard. We successively added to the training corpus clinical resources, including a Spanish-Basque dictionary derived from resources built for the machine translation of the Spanish edition of SNOMED CT into Basque, artificial sentences in Spanish and Basque derived from frequently occurring relationships in SNOMED CT, and Spanish monolingual EHRs. Apart from calculating bilingual evaluation understudy (BLEU) values, we tested the performance in the clinical domain by human evaluation. We achieved slight improvements from our reference system by tuning some hyperparameters using an out-of-domain bilingual corpus, obtaining 10.67 BLEU points for Basque-to-Spanish clinical domain translation. The inclusion of clinical terminology in Spanish and Basque and the application of the back-translation technique on monolingual EHRs significantly improved the performance, obtaining 21.59 BLEU points. This was confirmed by the human evaluation performed by 2 clinicians, ranking our machine translations close to the human translations. We showed that, even after optimizing the hyperparameters out-of-domain, the inclusion of available resources from the clinical domain and applied methods were beneficial for the described objective, managing to obtain adequate translations of EHR templates. We have developed a system which is able to properly translate health record templates from Basque to Spanish without making use of any bilingual corpus of clinical texts or health records.

Machine Translation Evaluation Metric Based on Dependency Parsing Model

Most of the syntax-based metrics obtain the similarity by comparing the sub-structures extracted from the trees of hypothesis and reference. These sub-structures cannot represent all the information in the trees because their lengths are limited. To sufficiently use the reference syntax information, a new automatic evaluation metric is proposed based on the dependency parsing model. First, a dependency parsing model is trained using the reference dependency tree for each sentence. Then, the hypothesis is parsed by this dependency parsing model and the corresponding hypothesis dependency tree is generated. The quality of hypothesis can be judged by the quality of the hypothesis dependency tree. Unigram F-score is included in the new metric so that lexicon similarity is obtained. According to experimental results, the proposed metric can perform better than METEOR and BLEU on system level and get comparable results with METEOR on sentence level. To further improve the performance, we also propose a combined metric which gets the best performance on the sentence level and on the system level.

Tradução automática e poesia: o caso do inglês e do português

This article sets out to analyse the translations into Portuguese of three poems written I English, “A Letter is a Joy of Earth” by Emily Dickinson, “To a Stranger” by Walt Whitman and “Sandra” by Charles Bukowski. Three translations by three human translators, Geir Campos, Pedro Gonzaga and Jorge de Sena, are compared to the translations made by Google Translate in order to evaluate machine translation quality. This research shows that machine translations are less “ludicrous” than some would think and are in fact quite acceptable. In the cases investigated, machine translations are sometimes as acceptable as the ones made by the professional translators, and they could even help them to make mistakes through a lack of attention or by ignoring all the possibilities in the case of polysemous words. The Google translations are obviously plainer and there are a number of mistakes in them of the kind one expects: wrong concordances, wrong interpretations of polysemous words, wrong interpretation of gendered words. However, overall the results are far more satisfying than forecast. Google Translate, or similar programmes, may help translators with different, albeit sound alternatives. Additionally, machine translations provide a useful tool to analyse the idiosyncrasies of translators.

Machine Translation Evaluation: Unveiling the Role of Dense Sentence Vector Embedding for Morphologically Rich Language

Machine Translation (MT) evaluation metrics like BiLingual Evaluation Understudy (BLEU) and Metric for Evaluation of Translation with Explicit Ordering (METEOR) are known to have poor performance for word-order and morphologically rich languages. Application of linguistic knowledge to evaluate MTs for morphologically rich language like Hindi as a target language, is shown to be more effective and accurate [S. Tripathi and V. Kansal, Using linguistic knowledge for machine translation evaluation with Hindi as a target language, Comput. Sist.21(4) (2017) 717–724]. Leveraging the recent progress made in the domain of word vector and sentence vector embedding [T. Mikolov and J. Dean, Distributed representations of words and phrases and their compositionality, Adv. Neural Inf. Process. Syst. 2 (2013) 3111–3119], authors have trained a large corpus of pre-processed Hindi text ([Formula: see text] million tokens) for obtaining the word vectors and sentence vector embedding for Hindi. The training has been performed on high end system configuration utilizing Google Cloud platform resources. This sentence vector embedding is further used to corroborate the findings through linguistic knowledge in evaluation metric. For morphologically rich language as target, evaluation metric of MT systems is considered as an optimal solution. In this paper, authors have demonstrated that MT evaluation using sentence embedding-based approach closely mirrors linguistic evaluation technique. The relevant codes used to generate the vector embedding for Hindi have been uploaded on code sharing platform Github. a