Post-Editing Oriented Human Quality Evaluation of Neural Machine Translation in Translator Training: A Study on Perceived Difficulties and Benefits

Abstract

The aim of this article is to investigate translation trainees’ perceived difficulties and benefits of a post-editing oriented neural machine translation (NMT) error annotation and quality evaluation task which was carried out for the language pair English-Turkish and for two separate text types and domains, i.e., environmental blogs and movie/TV reviews, within the scope of an MA course on translation quality standards. The data to be analyzed were collected through a semi-structured questionnaire which was given to the trainees attending the course after the completion of the task. The questionnaire was prepared with the aim of understanding perceived difficulties and benefits of the task. Analysis of the answers revealed that most of the trainees were in the opinion that the task was difficult. Majority of the trainees also believed that the task was beneficial and enabled them to feel empowered to make decisions on human translation quality evaluation of machine translation (MT) and to carry out error annotation and post-editing activities in the future. According to a significant number of trainees, error annotation facilitated their post-editing process and reduced the effort in post-editing. Enhanced understanding of MT error annotation and enhanced ability to perform post-editing were the significant benefits stated by the trainees. The difficulties were associated with being introduced and assigned to perform tasks they were not familiar with. Yet, as displayed in the answers to the questionnaire, a considerable majority of the trainees were positive about the learning experience. The results have also shown that integration of MT-related activities into translator training with a focus on the empowerment of the human translator has its difficulties and benefits also for translation trainers. While the difficulties for the trainers concern the decisions on the design, implementation, and planning of the task and the responsibility to carry out the task in constant interaction and collaboration with the trainees, the benefits are the sense of fulfillment and enrichment brought by positive feedback from the trainees and the discovery of the fact that the so-called ‘teaching experience’ becomes a ‘learning experience’ for trainers as well as trainees.