Connecting Model-Based and Model-Free Control With Emotion Modulation in Learning Systems

Abstract

This article proposes a novel decision-making framework that bridges a gap between model-based (MB) and model-free (MF) control processes through only adjusting the planning horizon. Specifically, the output policy is obtained by solving a model predictive control problem with a locally optimal state value as terminal constraints. When the planning horizon decreases to zero, the MB control will transform into the MF control smoothly. Meanwhile, inspired by the neural mechanism of emotion modulation on decision-making, we build a biologically plausible computational model of emotion processing. This model can generate an uncertainty-related emotional response on the basis of the state prediction error and reward prediction error, and then dynamically modulates the planning horizon in the tasks. The simulation results demonstrate that the proposed decision-making framework can produce better policies than traditional methods. Emotion modulation can shift the MB and MF control well to improve the learning efficiency and the speed of decision-making.