Mastering Overdetection and Underdetection in Learner-Answer Processing

Abstract

This paper presents a "didactic triangulation" strategy to cope with the problem of reliability of NLP applications for computer-assisted language learning (CALL) systems. It is based on the implementation of basic but well mastered NLP techniques and puts the emphasis on an adapted gearing between computable linguistic clues and didactic features of the evaluated activities. We claim that a correct balance between false positives (i.e., false error detection) and false negatives (i.e., undetected errors) is not only an outcome of NLP techniques, but also of an appropriate didactic integration of what NLP can do well--and what it cannot do well. Based on this approach, ExoGen is a prototype for generating activities such as gap-fill exercises. It integrates a module for error detection and description which checks learners' answers against expected ones. Through the analysis of graphic, orthographic, and morphosyntactic differences, it is able to diagnose problems like spelling errors, lexical mix-ups, agreement errors, conjugation errors, and so on. The first evaluation of ExoGen outputs, based on the FRIDA learner corpus, has yielded very promising results, paving the way for the development of an efficient and general model adaptable to a wide variety of activities.