LeBLEU: N-gram-based Translation Evaluation Score for Morphologically Complex Languages

Sami Virpioja,Stig-Arne Grönroos

doi:10.18653/v1/w15-3052

Abstract

This paper describes the LeBLEU evaluation score for machine translation, submitted to WMT15 Metrics Shared Task. LeBLEU extends the popular BLEU score to consider fuzzy matches between word n-grams. While there are several variants of BLEU that allow to non-exact matches between words either by character-based distance measures or morphological preprocessing, none of them use fuzzy comparison between longer chunks of text. The results on WMT data sets show that fuzzy n-gram matching improves correlations to human evaluation especially for highly compounding languages.

Highlights

The quality of machine translation has improved to the level that the translation hypotheses are useful starting points for human translators for almost any language pair
Common evaluation measures are Word Error Rate (WER) and Letter Error Rate (LER) that are based on the Levenshtein edit distance (Levenshtein, 1966)
Segmentlevel results of Workshops on Statistical Machine Translation (WMT) 2013 are dominated by single submission, SIMPBLEU-RECALL by Song et al (2013)

Summary

Introduction

The quality of machine translation has improved to the level that the translation hypotheses are useful starting points for human translators for almost any language pair. If we compare the raw translation hypothesis and its post-edited version, an appropriate edit distance measure should correlate to the edit time. Implementing such a measure is far from trivial. LER is more reasonable measure than WER for morphologically complex languages, in which the same word can occur in many inflected and derived forms (Creutz et al, 2007). Both give too high penalty for the variations in word ordering, which are frequent in translations. For languages in which the grammatical roles are marked by morphology and not the word order, there may be many more options

Methods

Results

Conclusion