Near Human-Level Performance in Grammatical Error Correction with Hybrid Machine Translation

Roman Grundkiewicz,Marcin Junczys-Dowmunt

doi:10.18653/v1/n18-2046

Abstract

We combine two of the most popular approaches to automated Grammatical Error Correction (GEC): GEC based on Statistical Machine Translation (SMT) and GEC based on Neural Machine Translation (NMT). The hybrid system achieves new state-of-the-art results on the CoNLL-2014 and JFLEG benchmarks. This GEC system preserves the accuracy of SMT output and, at the same time, generates more fluent sentences as it typical for NMT. Our analysis shows that the created systems are closer to reaching human-level performance than any other GEC system reported so far.

Highlights

Statistical Machine Translation (SMT) M2 Neural Machine Translation (NMT) HybridR&RJ’D16&GT’h1i6s work YS&cBh.’&16aJl.i’&17aTlh.’i1s7work Y&Calh.’.1&7NgT’h1i7s work Figure 1: Comparison of SMT, NMT and hybrid Grammatical Error Correction (GEC) systems on the CoNLL-2014 test set (M2).Currently, the most effective GEC systems are based on phrase-based statistical machine translation (Rozovskaya and Roth, 2016; JunczysDowmunt and Grundkiewicz, 2016; Chollampatt and Ng, 2017)
Systems that rely on neural machine translation (Yuan and Briscoe, 2016; Xie et al, 2016; Schmaltz et al, 2017; Ji et al, 2017) are not yet able to achieve as high performance as SMT systems according to automatic evaluation metrics
It has been shown that the neural approach can produce more fluent output, which might be desirable by human evaluators (Napoles et al, 2017). We combine both MT flavors within a hybrid GEC system. Such a GEC system preserves the accuracy of SMT output and at the same time generates more fluent sentences achieving new state-of-the-art results on two different benchmarks: the annotationbased CoNLL-2014 and the fluency-based JFLEG benchmark

Summary

Introduction

R&RJ’D16&GT’h1i6s work YS&cBh.’&16aJl.i’&17aTlh.’i1s7work Y&Calh.’.1&7NgT’h1i7s work Figure 1: Comparison of SMT, NMT and hybrid GEC systems on the CoNLL-2014 test set (M2). Systems that rely on neural machine translation (Yuan and Briscoe, 2016; Xie et al, 2016; Schmaltz et al, 2017; Ji et al, 2017) are not yet able to achieve as high performance as SMT systems according to automatic evaluation metrics (see Table 1 for comparison on the CoNLL-2014 test set). We combine both MT flavors within a hybrid GEC system. Such a GEC system preserves the accuracy of SMT output and at the same time generates more fluent sentences achieving new state-of-the-art results on two different benchmarks: the annotationbased CoNLL-2014 and the fluency-based JFLEG benchmark. We compare the performance with human annotations and identify issues with current state-of-the-art systems (§ 6)

Data and preprocessing

SMT systems

Results

NMT systems

52 Avg 50

Hybrid SMT-NMT systems

Analysis and future work