Analysing the Impact of Interactive Machine Translation on Post-editing Effort

Abstract

The combination of temporal, technical and cognitive effort has been proposed as metrics to evaluate the feasibility of post-editing on machine-translation (MT) output (Krings, 2001). In this study, we investigate the impact of interactive machine translation on the post-editing effort required to post-edit two specialized texts under experimental conditions and correlate it with Translation Edit Rate (TER) scores. Using the CasMaCat workbench as a post-editing tool in conjunction with a Tobii T60 eye tracker, process data were collected from 16 participants with some training on postediting. They were asked to carry out post-editing tasks under two different conditions: i) traditional post-editing (MT) and ii) interactive post-editing (IMT). In the IMT condition, as the user types, the MT system suggests alternative target translations which the post-editor can interactively accept or overwrite, whereas in the traditional MT condition no aids are provided to the user while editing the raw MT output. Temporal effort is measured by the total time spent to complete the task whereas technical effort is measured by the number of keystrokes and mouse events performed by each participant. In turn, cognitive effort is measured by fixation duration and the number of eye fixations (fixation count) in each task. Results show that IMT post-editing had significantly lower fixation duration and fewer fixation counts in comparison to traditional post-editing.