How do measures of cognitive effort relate to each other? A multivariate analysis of post-editing process data

Lucas Nunes Vieira

doi:10.1007/s10590-016-9188-5

Abstract

There has been growing interest of late in the cognitive effort required by post-editing of machine translation. Compared to number of editing operations, cognitive (or mental) effort is frequently considered a more decisive indicator of the overall effort expended by post-editors. Estimating cognitive effort is not straightforward, however. Previous studies often triangulate different measures to obtain a consensus, but little post-editing research to date has attempted to show how measures of cognitive effort relate to each other in a multivariate analysis. This paper addresses this by presenting an exploratory comparison of cognitive measures based on eye tracking, pauses, editing time, and subjective ratings collected in a post-editing task carried out by professional and non-professional participants. All measures correlated with each other, but a principal components analysis showed that the measures cluster together in different ways. In particular, measures that increase with task time alone behaved differently from the others, with higher mutual associations and higher reliability. Regarding differences between professional and non-professional participants, it was observed that subjective ratings were overall more strongly associated with objective measures in the case of professionals. Surprising findings from previous research based on pause ratio are discussed. The paper argues that a pause typology will benefit the study of pause lengths and cognitive effort in post-editing.

Highlights

The potential benefits of post-editing machine translation (MT) output, as opposed to translating source texts from scratch, are largely uncontroversial in the context of non-literary translation
The present study investigates the behaviour of seven such measures: eye fixation count, average fixation duration, editing seconds per word, subjective ratings, pause ratio, pause-to-word ratio, and average pause ratio
Seconds per word (SPW), pause-to-word ratio (PWR) and fixations per word (FPW) have stronger correlations between themselves when compared to the other measures

Summary

Introduction

The potential benefits of post-editing machine translation (MT) output, as opposed to translating source texts from scratch, are largely uncontroversial in the context of non-literary translation (cf. Green et al 2013; Plitt and Masselot 2010). MT quality and the effort required by post-editing can be influenced by a number of factors, such as genre/domain and source-text features that are problematic for MT (cf Bernth and Gdaniec 2001; Calude 2004). There has been growing interest in measuring the effort required by post-editing for the purpose of examining the feasibility of this practice and for empirically identifying characteristics of the source text or MT output that can be used as effort predictors There have been studies aimed at identifying the extent to which different individual profiles affect post-editing effort. A positive attitude to MT is often found to be a factor in post-editing performance (e.g. de Almeida 2013; Mitchell 2015). Moorkens and O’Brien (2015) observed that attitudes tend to be more negative in the case of professionals

Objectives

Methods

Results

Discussion

Conclusion