Abstract

Previously, neural methods in grammatical error correction (GEC) did not reach state-of-the-art results compared to phrase-based statistical machine translation (SMT) baselines. We demonstrate parallels between neural GEC and low-resource neural MT and successfully adapt several methods from low-resource MT to neural GEC. We further establish guidelines for trustable results in neural GEC and propose a set of model-independent methods for neural GEC that can be easily applied in most GEC settings. Proposed methods include adding source-side noise, domain-adaptation techniques, a GEC-specific training-objective, transfer learning with monolingual data, and ensembling of independently trained GEC models and language models. The combined effects of these methods result in better than state-of-the-art neural GEC models that outperform previously best neural GEC systems by more than 10% M² on the CoNLL-2014 benchmark and 5.9% on the JFLEG test set. Non-neural state-of-the-art systems are outperformed by more than 2% on the CoNLL-2014 benchmark and by 4% on JFLEG.

Highlights

  • Most successful approaches to automated grammatical error correction (GEC) are based on methods from statistical machine translation (SMT), especially the phrase-based variant

  • If we look at recent MT work with this in mind, we find one area where phrased-based SMT dominates over neural machine translation (NMT): low-resource machine translation

  • Current state-of-the-art GEC systems based on SMT, all include large-scale indomain language models either following the steps outlined in Junczys-Dowmunt and Grundkiewicz (2016) or directly re-using their domain-adapted Common-Crawl language model

Read more

Summary

Introduction

Most successful approaches to automated grammatical error correction (GEC) are based on methods from statistical machine translation (SMT), especially the phrase-based variant. The Cambridge Learner Corpus (CLC) by Nicholls (2003) — probably the best resource in this list — is non-public and we would strongly discourage reporting results that include it as training data as this makes comparisons difficult Current state-of-the-art GEC systems based on SMT, all include large-scale indomain language models either following the steps outlined in Junczys-Dowmunt and Grundkiewicz (2016) or directly re-using their domain-adapted Common-Crawl language model It seems that the current state of neural methods in GEC reflects the behavior for NMT systems trained on smaller data sets. We recommend a model-independent toolbox for neural GEC

A trustable baseline for neural GEC
Training and test data
Preprocessing and sub-words
Model and training procedure
Optimizer instability
Ensembling of independent models
Adaptations for GEC
Source-word dropout as corruption
Domain adaptation
Error adaptation
Tied embeddings
Edit-weighted MLE objective
Transfer learning for GEC
Pre-training embeddings
Pre-training decoder parameters
Results for transfer learning
Ensembling with language models
Deeper NMT models
Architectures
Training settings
Pre-training deep models
Results
A standard tool set for neural GEC
Full Text
Published version (Free)

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call