Modeling Second Language Acquisition with pre-trained neural language models

Abstract

Prediction of language mistakes is a task introduced by Duolingo as part of the Second Language Acquisition Modeling topic that aims to learn from the history of mistakes to improve the experience of language learners. Using transfer learning by means of pre-trained language models, we propose a framework that can learn the actual mistakes distribution according to which faraway words of a sentence have a higher chance to produce errors. To adapt the information provided by the pre-trained language models, more approaches based on feature extraction or fine-tuning were tried. However, according to our experiments, integrating these two options in a stack-and-finetune approach seems to be more appropriate for our task. Regarding the comparison of language models in terms of model distillation, we notice that distillation does not affect the effectiveness while significantly reducing the training time. We conclude that the model complexity should be adjusted to the specifics of the analyzed problem and the distillation is an efficient option for low complexity corpora without considerably affecting the overall performance.