A case study on the evaluation of Korean-English legal translations by generic and custom neural machine translation engines

Abstract

This paper addresses the quality of Korean-English legal translation outputs generated by three neural machine translation engines, namely NMT1, NMT2 and NMT3. NMT1 and NMT2 are widely available multilingual generic machine translation engines, whereas NMT3 is a custom machine translation engine for legal translation. In both automatic and human evaluations of the three engines’ English translations of Korean statutes, the custom engine outperformed the other two NMTs. BLEU and METEOR scores revealed that NMT3’s output was more similar to human reference translation. Human evaluation results confirmed NMT3’s strong performance, outperforming the generic engines in accuracy and fluency while underperforming in terminology. However, further analysis revealed that NMT3 was superior in handling terms comprising compound nouns and sentences over 25 words. Although the custom engine’s performance still requires substantial human involvement for quality translation, it showed a potential for the legal domain, demonstrating human parity in short sentences. The growing demand for speedy translation under budget constraints calls for further research on technology-based translation tools and human-computer interaction for legal translation.