Machine Translation Evaluation Research Articles

Implicit Cross-Lingual Word Embedding Alignment for Reference-Free Machine Translation Evaluation

As we know, cross-lingual word embedding alignment is critically important for reference-free machine translation evaluation, where source texts are directly compared with system translations. In this paper, it is revealed that multilingual knowledge distillation for sentence embedding alignment could achieve cross-lingual word embedding alignment implicitly. A simplified analysis is given to explain the implicit alignment reason. And according to the analysis, it could be deduced that using the last layer embeddings of the distilled student model will have the best alignment effect, which is also validated by the experimental results on the WMT19 datasets. Furthermore, with the assistant of a target-side language model, BERTScore andWord Mover’s Distance using the cross-lingual word embeddings get very competitive results (4 best average scores on 3 types of language directions and ranking first among more than half of all 18 language pairs for the system-level evaluations) in the WMT19’s reference-free machine translation evaluation tasks when the current state-of-the-art (SOTA) metrics are chosen for comparison.

Does a Formal Post-editing Training Affect the Performance of Novice Post-editors? An Experimental Study

Machine translation systems led to the creation of a new role for translators: the post-editor. With the birth of neural machine translation systems, the demand for post-editing has been increasing in the recent years, and it has now become a common service given by language service providers and professional translators. Such a change in the landscape of the translation industry might evolve the translation training programs worldwide. It is still heavily discussed whether post-editing and translation skills overlap, and post-editing courses are now included into the curriculum by several translation departments. We set out to investigate whether post-editing training influences the performance of student post-editors in order to explore the necessary background and skills in post-editing tasks. We measured productivity parameters and quality of the final outputs produced by two groups of participants, one of which was previously trained on post-editing. Our results show that, the experimental and control groups did not differ significantly from each other in terms of productivity. There was also little to no difference when we evaluated the post-edited outputs produced by both groups against a reference text using automatic machine translation evaluation metrics. However, we detected a statistical significance between the groups when we analyzed the number of errors in the final output. The post-editors in the experimental group were more aware of the typical errors of machine translation engines.

Machine English Translation Evaluation System Based on BP Neural Network Algorithm.

In order to solve the problems of machine translation efficiency and translation quality, this paper proposes an English translation evaluation system based on the BP neural network algorithm. This method provides users with a more intelligent machine translation service experience. With the help of the BP neural network algorithm, taking English online translation as the research object, Google's translation quality is the best, with an error frequency of only 167, while Baidu translation and iFLYTEK translation in China have a high error rate of 266 and 301, respectively, which is much higher than Google translation. A model of machine translation evaluation based on the neural network algorithm is proposed to better solve the disadvantages of traditional English machine translation. The results show that the machine translation system based on the neural network algorithm can further optimize the problems existing in machine translation, such as insufficient use of information and large scale of model parameters, and further improve the performance of neural network machine translation.

A human evaluation of English-Slovak machine translation

ABSTRACT The paper aims to obtain an error profile for machine translation (MT) from English into Slovak. We present an adjusted framework for MT evaluation, which is based on Vanko's categorical framework, but reflects machine translation peculiarities of synthetic and/or inflectional languages. Based on the framework, we analyse the errors generated by Google Translate and identify the most frequent categories of errors occurring in machine translation when translating newspaper articles from English into Slovak. While we have seen research on widely-spoken languages, such as English or other major official EU languages, little is known about Slovak, which is also an official EU language. This paper provides the first human MT evaluation study of English-Slovak machine translation using professional translators for a more detailed depiction of translation quality. Our research has revealed that the highest numbers of errors occurred in the sphere of lexical semantics, as well as in syntactic-semantic correlativeness, both being closely related. Additionally, based on the results of the Cochran Q test, we show how individual MT errors located in the examined categories differ in co-incidence and in how they impact translation quality.

Evaluation of Machine Translation in English-Chinese Automatic Subtitling of TED Talks

With technology becoming an essential competitive factor for subtitle translation services, the quality of machine subtitle translation in the current era deserves further examination. This paper examines the Chinese translation of 20 Ted Talks subtitled by an automatic subtitling software under the frame of the FAR model. This paper examines if machine subtitle translation has semantic, contextual, grammatical or spelling errors, idiomatic errors, syncopation and inappropriate synchronization, as well as wrong reading speed and paragraph length. It also emphasizes the need to broaden new perspectives on the quality of machine subtitle translation.

Semantics based English-Arabic machine translation evaluation

Some classic machine translation (MT) Evaluation methods, such as the bilingual evaluation understudy score (BLEU), have notably underperformed in evaluating machine translations for morphologically rich languages like Arabic. However, the recent remarkable advancements in the domain of word vectors and sentence vectors have opened up new research avenues for low-resource languages. This paper proposes a novel linguistic-based evaluation method for English-translated sentences in Arabic. The proposed approach includes penalties based on length, positions, and context-based schemes such as part-of-speech tagging (POS) and multilingual sentenceBERT (SBERT) models for machine translation evaluation. The proposed technique is tested using pearson correlation as a performance evaluation parameter and compared with state-of-the-art techniques. The experimental results demonstrate that the proposed model evidently outperforms other MT evaluation methods such as BLEU.

Evaluation of Semantic Answer Similarity Metrics

There are several issues with the existing general machine translation or natural language generation evaluation metrics, and question-answering (QA) systems are indifferent in that context. To build robust QA systems, we need the ability to have equivalently robust evaluation systems to verify whether model predictions to questions are similar to ground-truth annotations. The ability to compare similarity based on semantics as opposed to pure string overlap is important to compare models fairly and to indicate more realistic acceptance criteria in real-life applications. We build upon the first to our knowledge paper that uses transformer-based model metrics to assess semantic answer similarity and achieve higher correlations to human judgement in the case of no lexical overlap. We propose cross-encoder augmented bi-encoder and BERTScore models for semantic answer similarity, trained on a new dataset consisting of name pairs of US-American public figures. As far as we are concerned, we provide the first dataset of co-referent name string pairs along with their similarities, which can be used for training.

An English Translation Quality Evaluation Model Integrating Knowledge Transfer and Wireless Network

Machine translation is to automatically translate one natural language text into another. It is one of the important applications of natural language processing. The research of machine translation is closely related to machine translation evaluation technology. Therefore, with the progress of machine translation technology in recent years, researchers are also committed to designing more reasonable evaluation methods. Manual evaluation is costly, time-consuming, and subjective, so it has been difficult to provide practical help for the development of machine translation systems. Finding a feasible automatic machine translation evaluation method has been a hot spot in machine translation evaluation research in recent years. Based on this, this paper proposes an English translation quality evaluation model integrating knowledge transfer and wireless network. Firstly, wireless network technology is used to coordinate multitasks, and then knowledge transfer optimization technology is used to construct an English translation quality evaluation model. In this paper, simulation experiments are designed to verify the effectiveness of the model. The experimental results show that the accuracy of the evaluation method based on fusion technology is significantly improved compared with the existing evaluation methods.

Automatic assessment of spoken-language interpreting based on machine-translation evaluation metrics

Abstract Automated metrics for machine translation (MT) such as BLEU are customarily used because they are quick to compute and sufficiently valid to be useful in MT assessment. Whereas the instantaneity and reliability of such metrics are made possible by automatic computation based on predetermined algorithms, their validity is primarily dependent on a strong correlation with human assessments. Despite the popularity of such metrics in MT, little research has been conducted to explore their usefulness in the automatic assessment of human translation or interpreting. In the present study, we therefore seek to provide an initial insight into the way MT metrics would function in assessing spoken-language interpreting by human interpreters. Specifically, we selected five representative metrics – BLEU, NIST, METEOR, TER and BERT – to evaluate 56 bidirectional consecutive English–Chinese interpretations produced by 28 student interpreters of varying abilities. We correlated the automated metric scores with the scores assigned by different types of raters using different scoring methods (i.e., multiple assessment scenarios). The major finding is that BLEU, NIST, and METEOR had moderate-to-strong correlations with the human-assigned scores across the assessment scenarios, especially for the English-to-Chinese direction. Finally, we discuss the possibility and caveats of using MT metrics in assessing human interpreting.

An empirical evaluation of phrase-based statistical machine translation for Indonesia slang-word translator

<span>The use of slang (non-standard language), especially in social media, is increasing. It causes reducing the level of understanding when communicating because not everyone understands slang (non-standard language). The purpose of this work is to develop a slang-word translator. The other objective is to find the minimum number of sentences and </span><span>BiLingual Evaluation Understudy</span><span> (BLEU) score used as a benchmark to determine that the translation is understandable. The approach used in this project is a Phrase-based statistical machine translation (PBSMT) approach, suitable for low resource language, with a dataset of 100,000 sentences taken from the comments column of several online political news portals. The comments are then manually translated to produce a parallel corpus of non-standard language-standard language. The sample sentences are taken from the dataset then distributed using questionnaires to obtain the human understanding level regarding the translation result. The result of the implementation is a BLEU score of 64 and the minimum number of sentences to have an understandable machine translation is 500. The conclusion drawn from the distributed questionnaires is that humans can understand the sentences produced by the translation machine.</span>

Machine vs. Radiologist-Based Translations of RadLex: Implications for Multi-language Report Interoperability.

The purpose of this study was to evaluate the feasibility of translation of RadLex lexicon from English to German performed by Google Translate, using the RadLex ontology as ground truth. The same comparison was also performed for German to English translations. We determined the concordance rate of the Google Translate-rendered translations (for both English to German and German to English translations) to the official German RadLex (translations provided by the German Radiological Society) and English RadLex terms via character-by-character concordance analysis (string matching). Specific term characteristics of term character count and word count were compared between concordant and discordant translations using t-tests. Google Translate-rendered translations originally considered incongruent (2482 English terms and 2500 German terms) were then reviewed by German and English-speaking radiologists to further evaluate clinical utility. Overall success rates of both methods were calculated by adding the percentage of terms marked correct by string comparison to the percentage marked correct during manual review extrapolated to the terms that had been initially marked incorrect during string analysis. 64,632 English and 47,425 German RadLex terms were analyzed. 3507 (5.4%) of the Google Translate-rendered English to German translations were concordant with the official German RadLex terms when evaluated via character-by-character concordance. 3288 (6.9%) of the Google Translate-rendered German to English translations matched the corresponding English RadLex terms. Human review of a random sample of non-concordant machine translations revealed that 95.5% of such English to German translations were understandable, whereas 43.9% of such German to English translations were understandable. Combining both string matching and human review resulted in an overall Google Translate success rate of 95.7% for English to German translations and 47.8% for German to English translations. For certain radiologic text translation tasks, Google Translate may be a useful tool for translating multi-language radiology reports into a common language for natural language processing and subsequent labeling of datasets for machine learning. Indeed, string matching analysis alone is an incomplete method for evaluating machine translation. However, when human review of automated translation is also incorporated, measured performance improves. Additional evaluation using longer text samples and full imaging reports is needed. An apparent discordance between English to German versus German to English translation suggests that the direction of translation affects accuracy.

Procjena kvalitete strojnog prijevoda govora: studija slučaja aplikacije ILA

Machine translation (MT) is becoming qualitatively more successful and quantitatively more productive at an unprecedented pace. It is becoming a widespread solution to the challenges of a constantly rising demand for quick and affordable translations of both text and speech, causing disruption and adjustments of the translation practice and profession, but at the same time making multilingual communication easier than ever before. This paper focuses on the speech-to-speech (S2S) translation app Instant Language Assistant (ILA), which brings together the state-of-the-art translation technology: automatic speech recognition, machine translation and text-to-speech synthesis, and allows for MT-mediated multilingual communication. The aim of the paper is to assess the quality of translations of conversational language produced by the S2S translation app ILA for en-de and en-hr language pairs. The research includes several levels of translation quality analysis: human translation quality assessment by translation experts using the Fluency/Adequacy Metrics, light-post editing, and automated MT evaluation (BLEU). Moreover, the translation output is assessed with respect to language pairs to get an insight into whether they affect the MT output quality and how. The results show a relatively high quality of translations produced by the S2S translation app ILA across all assessment models and a correlation between human and automated assessment results.

A comprehensive understanding of popular machine translation evaluation metrics

A comprehensive understanding of popular machine translation evaluation metrics

The FEMTI guidelines for contextual MT evaluation: principles and resources

A large number of evaluation metrics exist for machine translation (MT) systems, but depending on the intended context of use of such a system, not all metrics are equally relevant. Based on the ISO/IEC 9126 and 14598 standards for software evaluation, the Framework for the Evaluation of Machine Translation in ISLE (FEMTI) provides guidelines for the selection of quality characteristics to be evaluated depending on the expected task, users, and input characteristics of an MT system. This approach to contextual evaluation was implemented as a web-based application which helps its users design evaluation plans. In addition, FEMTI offers experts in evaluation the possibility to enter and share their knowledge using a dedicated web-based tool, tested in several evaluation exercises.

The FEMTI guidelines for contextual MT evaluation: principles and resources

A large number of evaluation metrics exist for machine translation (MT) systems, but depending on the intended context of use of such a system, not all metrics are equally relevant. Based on the ISO/IEC 9126 and 14598 standards for software evaluation, the Framework for the Evaluation of Machine Translation in ISLE (FEMTI) provides guidelines for the selection of quality characteristics to be evaluated depending on the expected task, users, and input characteristics of an MT system. This approach to contextual evaluation was implemented as a web-based application which helps its users design evaluation plans. In addition, FEMTI offers experts in evaluation the possibility to enter and share their knowledge using a dedicated web-based tool, tested in several evaluation exercises.

AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages

Machine Translation (MT) depends upon the MT evaluation for comparing several MT systems and gives a measure of their efficiency in terms of translation sentences. Consequently, MT evaluation has become an integral part of the MT procedure and it has led to the development of several MT evaluation metrics which can automatically assess the MT systems. Lexical metrics like BLEU have been mostly used in MT evaluation. However, these metrics poorly represent lexical relationships and impose strict identity matching, leading to less correlation with human evaluation for morphologically rich languages. To overcome the limitations posed by the BLEU evaluation metric for morphologically rich languages, we propose a MT evaluation score called AdaBLEU that is a modified BLEU evaluation score. The proposed score considers the lexical and syntactical properties for any language including the morphologically rich languages. It considers the Parts-of-Speech tags and Dependency Parsing tags along with the BLEU score of sentences. Our modification to the BLEU score does not require multiple reference sentences for evaluation. The evaluation of the performance of AdaBLEU has been conducted by comparing our proposed metric’s performance with several other evaluation metrices on different test datasets for different morphological languages. Experimental results show an improved performance in the case of the proposed score.

Can automated machine translation evaluation metrics be used to assess students’ interpretation in the language learning classroom?

The use of translation and interpreting (T&I) in the language learning classroom is commonplace, serving various pedagogical and assessment purposes. Previous utilization of T&I exercises is driven largely by their potential to enhance language learning, whereas the latest trend has begun to underscore T&I as a crucial skill to be acquired as part of transcultural competence for language learners and future language users. Despite their growing popularity and utility in the language learning classroom, assessing T&I is time-consuming, labor-intensive and cognitively taxing for human raters (e.g., language teachers), primarily because T&I assessment entails meticulous evaluation of informational equivalence between the source-language message and target-language renditions. One possible solution is to rely on automated quality metrics that are originally developed to evaluate machine translation (MT). In the current study, we investigated the viability of using four automated MT evaluation metrics, BLEU, NIST, METEOR and TER, to assess human interpretation. Essentially, we correlated the automated metric scores with the human-assigned scores (i.e., the criterion measure) from multiple assessment scenarios to examine the degree of machine-human parity. Overall, we observed fairly strong metric-human correlations for BLEU (Pearson’s r = 0.670), NIST (r = 0.673) and METEOR (r = 0.882), especially when the metric computation was conducted on the sentence level rather than the text level. We discussed these emerging findings and others in relation to the feasibility of operationalizing MT metrics to evaluate students’ interpretation in the language learning classroom. Supplemental data for this article is available online at https://doi.org/10.1080/09588221.2021.1968915 .

Cadlaws – An English–French Parallel Corpus of Legally Equivalent Documents

This article presents Cadlaws, a new English–French corpus built from Canadian legal documents, and describes the corpus construction process and preliminary statistics obtained from it. The corpus contains over 16 million words in each language and includes unique features since it is composed of documents that are legally equivalent in both languages but not the result of a translation. The corpus is built upon enactments co-drafted by two jurists to ensure legal equality of each version and to re­flect the concepts, terms and institutions of two legal traditions. In this article the corpus definition as a parallel corpus instead of a comparable one is also discussed. Cadlaws has been pre-processed for machine translation and baseline Bilingual Evaluation Understudy (bleu), a score for comparing a candidate translation of text to a gold-standard translation of a neural machine translation system. To the best of our knowledge, this is the largest parallel corpus of texts which convey the same meaning in this language pair and is freely available for non-commercial use.

The Quality of Machine Translation Assessment On Gender Markers Lingual Units

Machine Translation (MT) is one of the most advanced and elaborate research fields within Translation Technology, the quality of MT output has always been a great concern, and MT evaluation is a popular research topic. This research aims to assess the quality translation on the gender markers lingual unit of the Arabic short story "عَبْدُ اللهِ وَالْعُصْفُوْرُ" which is translated to English and Indonesian using Machine Translation. The research was qualitatively-based method. The subject is gender markers lingual unit that taken from the Arabic short story "عَبْدُ اللهِ وَالْعُصْفُوْرُ". The key instrument of this research is human instrument. Additional instruments used to support this research consisted tables of the lingual units of gender markers and table of rating scales based on Nababans' theory (2012). The research findings showed that this analysis has discovered were 72 lingual units of gender markers in the short story. Further, the dominant type was personal pronoun. Based on the results, it can be concluded that the google translate translation on the gender markers lingual units have the high quality on the accuracy, acceptance, and readability level.

Bridging the gap between machine translation output and images in multimodal documents

The aim of this article is to report on recent findings concerning the use of Google Translate outputs in multimodal contexts. Development and evaluation of machine translation often focus on verbal mode, but accounts by the area on the exploration text-image relations in multimodal documents translated automatically are rare. Thus, this work seeks to describe just what are such relations and how to describe them, organized in two parts: firstly, by exploring the problem through an interdisciplinary interface, involving Machine Translation and Multimodality to analyze some examples from the Wikihow website; secondly, by reporting on recent investigation on suitable tools and methods to properly annotate these issues from within a long-term purpose to assemble a corpus. Finally, this article provides a discussion on the findings, including some limitations and perspectives for future research.