Guiding automatic MT evaluation by means of linguistic features

Abstract

Machine translation (MT) has become increasingly important and popular in the past decade, leading to the development of MT evaluation metrics aiming at automatically assessing MT output. Most of these metrics use reference translations to compare systems output, therefore, they should not only detect MT errors but also be able to identify correct equivalent expressions so as not to penalize them when those are not displayed in the reference translations. With the aim of improving MT evaluation metrics a study has been carried out of a wide panorama of linguistic features and their implications. For that purpose a Spanish and an English corpora containing hypothesis and reference translations have been analysed from a linguistic point of view, so that common errors can be detected and positive equivalencies highlighted. This article focuses on this qualitative analysis describing the linguistic phenomena that should be considered when developing an automatic MT evaluation metric. The results of this analysis have been used to develop an automatic MT evaluation metric that takes into account different dimensions of language. A brief review of the metric and its evaluation are also provided.