Evaluation Of Translation Quality Research Articles

Evaluation of English–Slovak Neural and Statistical Machine Translation

This study is focused on the comparison of phrase-based statistical machine translation (SMT) systems and neural machine translation (NMT) systems using automatic metrics for translation quality evaluation for the language pair of English and Slovak. As the statistical approach is the predecessor of neural machine translation, it was assumed that the neural network approach would generate results with a better quality. An experiment was performed using residuals to compare the scores of automatic metrics of the accuracy (BLEU_n) of the statistical machine translation with those of the neural machine translation. The results showed that the assumption of better neural machine translation quality regardless of the system used was confirmed. There were statistically significant differences between the SMT and NMT in favor of the NMT based on all BLEU_n scores. The neural machine translation achieved a better quality of translation of journalistic texts from English into Slovak, regardless of if it was a system trained on general texts, such as Google Translate, or specific ones, such as the European Commission’s (EC’s) tool, which was trained on a specific-domain.

Automatic evaluation of the quality of machine translation of a scientific text: the results of a five-year-long experiment

We report on various approaches to automatic evaluation of machine translation quality and describe three widely used methods. These methods, i.e. methods based on string matching and n-gram models, make it possible to compare the quality of machine translation to reference translation. We employ modern metrics for automatic evaluation of machine translation quality such as BLEU, F-measure, and TER to compare translations made by Google and PROMT neural machine translation systems with translations obtained 5 years ago, when statistical machine translation and rule-based machine translation algorithms were employed by Google and PROMT, respectively, as the main translation algorithms [6]. The evaluation of the translation quality of candidate texts generated by Google and PROMT with reference translation using an automatic translation evaluation program reveal significant qualitative changes as compared with the results obtained 5 years ago, which indicate a dramatic improvement in the work of the above-mentioned online translation systems. Ways to improve the quality of machine translation are discussed. It is shown that modern systems of automatic evaluation of translation quality allow errors made by machine translation systems to be identified and systematized, which will enable the improvement of the quality of translation by these systems in the future.

Post-Editing Oriented Human Quality Evaluation of Neural Machine Translation in Translator Training: A Study on Perceived Difficulties and Benefits

The aim of this article is to investigate translation trainees’ perceived difficulties and benefits of a post-editing oriented neural machine translation (NMT) error annotation and quality evaluation task which was carried out for the language pair English-Turkish and for two separate text types and domains, i.e., environmental blogs and movie/TV reviews, within the scope of an MA course on translation quality standards. The data to be analyzed were collected through a semi-structured questionnaire which was given to the trainees attending the course after the completion of the task. The questionnaire was prepared with the aim of understanding perceived difficulties and benefits of the task. Analysis of the answers revealed that most of the trainees were in the opinion that the task was difficult. Majority of the trainees also believed that the task was beneficial and enabled them to feel empowered to make decisions on human translation quality evaluation of machine translation (MT) and to carry out error annotation and post-editing activities in the future. According to a significant number of trainees, error annotation facilitated their post-editing process and reduced the effort in post-editing. Enhanced understanding of MT error annotation and enhanced ability to perform post-editing were the significant benefits stated by the trainees. The difficulties were associated with being introduced and assigned to perform tasks they were not familiar with. Yet, as displayed in the answers to the questionnaire, a considerable majority of the trainees were positive about the learning experience. The results have also shown that integration of MT-related activities into translator training with a focus on the empowerment of the human translator has its difficulties and benefits also for translation trainers. While the difficulties for the trainers concern the decisions on the design, implementation, and planning of the task and the responsibility to carry out the task in constant interaction and collaboration with the trainees, the benefits are the sense of fulfillment and enrichment brought by positive feedback from the trainees and the discovery of the fact that the so-called ‘teaching experience’ becomes a ‘learning experience’ for trainers as well as trainees.

Как и какой перевод (не) оценивают компьютеры

The article discusses modern metrics for evaluating the quality of translation used in the development and tuning of MT systems, in machine translation competitions, and in evaluating the performance of some other NLP systems. The authors describe the criteria for evaluating the quality of translation and some methods of expert (human) evaluation. The article also reveals the mechanisms of automatic metrics (such as BLEU, TER, METEOR, BERTScore, COMET), their features, advantages and disadvantages. The authors emphasize the importance of BERTScore and COMET metrics and explain the popularity of some traditional metrics (e.g., BLEU). Modern metrics for the evaluation of translation quality give distorted results when the text contains numerous expressions with indirect meanings: poetic tropes, metaphors, metonymy, humor, or riddles. Communication with indirect meanings is linked with a human ability to think in contradictions. They are a source of insight and were used by Donald Davidson to describe the mechanism of a metaphor. However, communication with indirect meanings is still difficult to computerize. That is why the metric-based evaluation of professional literary translations shows poor results. Further development of metrics should use computer processing of contradictions, possibly with the help of inconsistent logics: paracomplete, paraconsistent and dialethic.

ĐÁNH GIÁ CHẤT LƯỢNG DỊCH TRONG ĐÀO TẠO DỊCH THUẬT BẬC ĐẠI HỌC

The article is a conceptual research with synthesis and analysis of theories and practices of translation quality assessment (TQA) in translation pedagogy in higher education institutions in the world. The research aims at providing a comprehensive picture with relevant basic issues in TQA; supplying criteria in the holistic assessment and analytic assessment, and classifications of translation errors in evaluation of translation quality in assignments and tests in translation courses; and scoring systems. The importance of TQA is emphasized and finding an answer to the open question of the possibility to build a common model in TQA for translation training at tertiary level is pinpointed. The article is expected to help teachers of translation in Vietnam to grasp a more realistic view of TQA so that they can make appropriate decisions and choose effective TQA solutions in their teaching context at tertiary level.

Reconfiguring Universities for the Future: E-learning Resources and Instructional Designs in Training Translation

Background: This study attempts to understand the importance of applying E-learning resources through evaluating the importance of said-resources in the differences appearing in the translation quality of texts trained to the students through various educational references. Objectives: To achieve the purpose, the study was conducted at Islamic Azad University-Tehran Science and Research branch during the second semester of academic year 2018 - 19. Methods: The evaluation of translation quality of the students who trained translation via traditional references and E-learning resource tools was conducted at the textual level according to the Sager’s general classification for three levels of gravity of errors considering distortion of sense, omission, and minor errors using one way ANCOVA to determine a statistically significant difference in the learner’s outcome through controlling the potential effect of translation pretest. Results: For both traditional and E-learning references in translation training, the scorings of the two translation raters were consistent since the intraclass correlation (α) was 0.933 (> 0.70). The two groups of the study were significantly different from one another via obtaining the mean of 15.51 in the E-learning resources group of the participants in comparison with the mean obtained in the traditional references group which was 13.75 in the early observation of the descriptive statistics. Besides, the P value of 0.001 (< 0.050) retrieved from the result of One-way ANCOVA, represented the significant effect of training translation via applying E-learning in comparison to the traditional referenced training. The significance value of .001, which was smaller than 0.050, indicated the fact that our groups were significantly different. The proof to the claim was given strength from the results of the post hoc test for the dependent variable of the study (translation posttest), in which the means of 15.547 (for the control group) and 16.073 (for the experimental group) in the multiple measurements of P value (0.001) was calculated based on the 95% of confidence interval and standard error of 0.175. Accordingly, the translation training via E-learning resources made a statistically significant difference in the learning outcome of the students, the quality of students’ translation, while controlling the effect of the pretest. Also, the effect size of 0.244 revealed the strength of independent variable as an appropriate numerical value for indicating the efficacy of the treatment-training translation via applying E-learning resources. Conclusions: The consistency of quality assurance in translation could be achievable via E-learning resources; however it is not possible to consider all the resources as the appropriate material within all educational contexts with various instructional designs. Thus, the use of E-learning resources, as the necessary factor in reconfiguring the universities for the future through updated instructional designs should be preserved by the teachers and students regarding the necessary adaptations required in the local contexts of training translation. The research implications are basically referring the stake holders in developing instructional designs to consider the importance of E-learning resources in the training procedures applied by the instructors to provide various courses including translation. However, it is necessary for the policy makers to pave the grounds to settle the required platforms for the use of such resources within educational contexts concerning their applicability in the local context.

Issues of Culture Specific Item Translation in Subtitling

Abstract Culture specific items (CSI) have received the attention of many researchers who claim that CSI translation is always problematic due to cultural interrelation. Rendering culture specific items in an audiovisual format is currently one of the most topical issues in translation research. The paper employs the taxonomy designed by Pedersen (2011) for culture specific item translation from English into Lithuanian analyzed in the subtitles of the Australian TV reality show My Kitchen Rules . The results indicate that not all culture specific items are appropriately rendered into the target language, which is a significant factor in the evaluation of translation quality.

A Quantitative Analysis of Discourse Phenomena in Machine Translation

State-of-the-art Machine Translation (MT) systems translate documents by considering isolated sentences, disregarding information beyond sentence level. As a result, machine-translated documents often contain problems related to discourse coherence and cohesion. Recently, some initiatives in the evaluation and quality estimation of MT outputs have attempted to detect discourse problems in order to assess the quality of these machine translations. However, a quantitative analysis of discourse phenomena in MT outputs is still needed in order to better understand the phenomena and identify possible solutions or ways to improve evaluation. This paper aims to answer the following questions: What is the impact of discourse phenomena on MT quality? Can we capture and measure quantitatively any issues related to discourse in MT outputs? In order to answer these questions, we present a quantitative analysis of several discourse phenomena and correlate the resulting figures with scores from automatic translation quality evaluation metrics. We show that figures related to discourse phenomena present a higher correlation with quality scores than the baseline counts widely used for quality estimation of MT.

The role of online translation tools in language education

The aim of the pilot study is to draw attention to the importance of online translation tools as a contemporary didactic tool in the language and translation educational process. In the paper the following hypotheses were tested: (i) use of free online translation tools can contribute to the development of ICT competences ; (ii) use of modern translation tools contribute to the consistency and quality ; (iii) free online translation tools for under-resourced languages can be used as pedagogical tools. Two studies were conducted using the text of EU legislation (acquis communautaire). In the first study, evaluation of the German-Slovenian translation quality of legislation was performed between two groups: one group using traditional reference works and the other group using online electronic translation tools (the Slovenian multilingual terminology database and the bilingual corpus). In both translations, mistakes were analyzed and a paired samples t-test was conducted. In the second study, machine translation of English-Croatian legislative text using the Google Translate online translation service was analyzed. The human evaluation of the translation quality was calculated, and Fleiss’ kappa metric used to assess inter-rater reliability of agreement. The evaluation was enriched by analysis of different types of errors.

Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics

In recent years translation quality evaluation has emerged as a major, and at times contentious, topic. Despite a focus on systematizing the evaluation of translation quality, the industry landscape is still highly fragmented, in part because different kinds of translation projects require very different evaluations of quality. In addition, human and machine translation (MT) quality evaluation methods have been fundamentally different in kind, preventing comparison of MT with human translation. The lack of clarity has contributed to an environment in which requesters of translation often cannot be certain whether or not a translation meets their needs or the needs of end users and where providers are unclear about what requesters need and want. In response the EU-funded QTLaunchPad project has developer the Multidimensional Quality Metrics (MQM) framework, an open an extensible system for declaring and describing translation quality metrics using a shared vocabulary of “issue types”.

Dynamic Quality Framework: quantifying and benchmarking quality

Translation quality is one of the key topics in the translation industry today. In 2011, TAUS developed the Dynamic Quality Framework (DQF) in an attempt to standardize translation quality evaluation. Quality in DQF is considered dynamic since today’s translation quality requirements change depending on content type, purpose and audience. DQF contains a rich knowledge base, resources on quality evaluation and a number of tools to profile and evaluate translated content. As of this year, DQF is freely available for academics and can be accessed on the TAUS Evaluate platform.

Quality Assurance in Legal Translation: Evaluating Process, Competence and Product in the Pursuit of Adequacy

AbstractBuilding on a functionalist framework for decision-making in legal translation, a holistic approach to quality is presented in order to respond to the specificities of this field and overcome the shortcomings of general models of translation quality evaluation. The proposed approach connects legal, contextual, macrotextual and microtextual variables for the definition of the translation adequacy strategy, which guides problem-solving and the rest of the translationprocess. The same parameters remain traceable between the translation brief and the translationproductboth in pre-delivery (self-)revision and in post-delivery assessment. They are the yardstick for identifying predictable evaluative criteria andcompetencerequirements for translators and quality controllers. The implications of the approach on quality assessment (including training contexts) and quality management practices are also discussed. Overall, the model illustrates the potential benefits of enhancing predictability and reducing subjectivity on the basis of specific legal translation methodologies. It supports the need for legal translation expertise in quality evaluation and the relevance of Legal Translation Studies to raising standards in professional practice.

Evaluating Female Translators

This paper is an attempt to guide the teachers about how the evaluation process should be and it highlights the effectiveness and suitability of adopting Hurtado's method of evaluation on female translators. This method was applied to the correction of female students' translations of the final exam containing different texts to be translated in both directions between English and Arabic. The exam was applied to 43 respondents. The hypothesis regarding the suitability and effectiveness of using Hurtado's method and the possibility to improve the quality of the evaluating the students' translations in future based on this method has been verified. This study concluded that this method was found out to be reasonable to give impartial translation quality evaluation for the students' translations.

Further evidence for a functionalist approach to translation quality evaluation

Colina (2008)proposes a componential-functionalist approach to translation quality evaluation and reports on the results of a pilot test of a tool designed according to that approach. The results show good inter-rater reliability and justify further testing. The current article presents an experiment designed to test the approach and tool. Data was collected during two rounds of testing. A total of 30 raters, consisting of Spanish, Chinese and Russian translators and teachers, were asked to rate 4–5 translated texts (depending on the language). Results show that the tool exhibits good inter-rater reliability for all language groups and texts except Russian and suggest that the low reliability of the Russian raters’ scores is unrelated to the tool itself. The findings are in line with those ofColina (2008).

Kryteria jakości w ewaluacji przekładu ustnego

Translation quality evaluation has been in the focus of interest of researchers for quite a long time, although empirical investigations were started only two decades ago. The general assumption is that the source language message should be ideally manifested in the target language, thus effective communication between the listeners (users) and the speaker is the priority of translation/interpretation. Effective communication is enabled by means of good quality interpretation. In the attempt aimed at quality description, apart from subjective impressions resulting from our perception of the features that good translation/interpretation and effective communication should have, we cannot ignore three basic factors, i.e., the interpreter (as the text author/ producer), the interpretation process and product which is the result of this process and involvement as well as competence on the part of the interpreter/translator. Quality criteria are largely fuzzy; their conceptual limits are not distinctive and, in the opinion of research subjects, are interrelated. Therefore, the specification of quality criteria that do not raise any doubts both on the part of users (e.g. listeners) and on the part of interpreters becomes problematic. The paradox of quality may emerge if the interpreter’s role is, on the one hand, to retain absolute fidelity to the original text and his/ her neutrality (or invisibility), and on the other to be a cultural mediator who controls communication, facilitates it and corrects the text to make it accurate, clear and coherent.

Translation Quality Evaluation

Following a review of existing approaches to translation quality evaluation, this paper describes a proposal for evaluation that addresses some of the deficiencies found in these models. The proposed approach is referred to as componential because it evaluates components of quality separately, and functionalist, because evaluation is carried out relative to the function specified for the translated text. In order to obtain some empirical evidence for the functionalist/componential approach, a tool was developed and pilot-tested for inter-rater reliability. In addition, the research project sought to obtain some data on qualifications of raters/users and their performance using the tool. Forty raters were asked to use the tool to rate three translated texts. The texts selected for evaluation consisted of reader-oriented health education materials. Raters were bilinguals, professional translators and language teachers. Some basic training was provided. Data was collected by means of the tool and a questionnaire. Results indicate good inter rater reliability for the tool; teachers’ and translators’ ratings were more alike than those of bilinguals; bilinguals were found to rate higher and faster than the other groups. The results provide support for further research and testing of this tool and offer evidence in favour of the approach proposed.

Methodological remarks on the study of translation and translating

Disciplinary identity depends on how the disciplinary research object is conceived of. If translation studies is a discipline studying translation and translating, it is natural that it can define its identity at the intersection between translation and translating. This intersection is translation process. The need for an epistemological description of translation process arises form the need to make the process describable or to find a possibility of its optimum description. To do this translation studies needs an elementary model of translation process that would ensure the comparability of different translations and describability of translation culture. New interest in the evaluation of translation quality and in the linguistic nature of translation brings along the necessity to reconceptualize different trends in translation studies. Translation studies is on the verge of a new self-description, and the language and attitudes of this self-description should derive from a systemic understanding of the research object of translation studies.