AdaBLEU: A Modified BLEU Score for Morphologically Rich Languages

Abstract

Machine Translation (MT) depends upon the MT evaluation for comparing several MT systems and gives a measure of their efficiency in terms of translation sentences. Consequently, MT evaluation has become an integral part of the MT procedure and it has led to the development of several MT evaluation metrics which can automatically assess the MT systems. Lexical metrics like BLEU have been mostly used in MT evaluation. However, these metrics poorly represent lexical relationships and impose strict identity matching, leading to less correlation with human evaluation for morphologically rich languages. To overcome the limitations posed by the BLEU evaluation metric for morphologically rich languages, we propose a MT evaluation score called AdaBLEU that is a modified BLEU evaluation score. The proposed score considers the lexical and syntactical properties for any language including the morphologically rich languages. It considers the Parts-of-Speech tags and Dependency Parsing tags along with the BLEU score of sentences. Our modification to the BLEU score does not require multiple reference sentences for evaluation. The evaluation of the performance of AdaBLEU has been conducted by comparing our proposed metric’s performance with several other evaluation metrices on different test datasets for different morphological languages. Experimental results show an improved performance in the case of the proposed score.