Predicting Students Disengaged Behaviors in an Online Meaning-Generation Task

Abstract

In an intelligent tutoring system (ITS), it can be useful to know when a student has disengaged from a task and might benefit from a particular intervention. However, predicting disengagement on a trial-by-trial basis is a challenging problem, particularly in complex cognitive domains. In the present work, data-driven methods were used to address two aspects of this problem: identification of predictive features at the single-trial level, and selection of accurate and robust models. Experiment data were collected in a middle-school classroom using a vocabulary training ITS. On each trial, the ITS presented a low-frequency (Tier 2 or frontier) word and prompted students to type in the word's meaning. Single-trial measures—including the orthographic and semantic accuracy of each response, and context-sensitive measures such as interaction patterns across trials—were computed throughout the task. There were two key findings. First, as expected, different features predicted when a student was likely to be more engaged (e.g., high semantic accuracy) or less engaged (e.g., repetition of same or similar words across consecutive trials). Second, there was added value in representing context-sensitive information, which captures patterns of performance over time, as well as trial-specific information. These findings provide useful insights into effective methods for representing and modeling temporal patterns of student engagement in an ITS, especially those related to language learning. Such models may be useful in the design and implementation of adaptive tutors in complex cognitive domains like language learning.