Personalized task difficulty adaptation based on reinforcement learning

Abstract

Traditionally, the task difficulty level is often determined by domain experts based on some hand-crafted rules. However, with the adoption of Massive Open Online Courses (MOOCs), it has become harder to manually personalize task difficulty as the system designers are faced with a very large question bank and a user base of individuals with diverse backgrounds and ability levels. This research focuses on developing a data-driven method to adaptively adjust difficulty levels in order to maintain a target user performance level over a series of tasks whose difficulty level is highly variable among different individuals. Specifically, the issue of difficulty adaptation was formulated as a reinforcement learning problem. To ensure responsiveness of the interactive systems, a novel bootstrapped policy gradient (BPG) framework was developed, which can incorporate prior knowledge of difficulty ranking into policy gradient to enhance sample efficiency. To obtain high-quality prior information on difficulty ranking, a clustering-based approach was proposed which can learn a personalized difficulty ranking to capture users’ individual differences. To evaluate the effectiveness of the difficulty adaptation method, we focused on a visual memory training problem with a large question bank and a diverse user base. Specifically, the proposed algorithms were combined and applied to a real-world application consisting of an online visual-spatial memory recall game and were shown to outperform the traditional rule-based adaptation approach in adapting to the slow players while achieving comparable performance in adapting to the fast players.