An extended reinforcement learning model of basal ganglia to understand the contributions of serotonin and dopamine in risk-based decision making, reward prediction, and punishment learning.

Pragathi P Balasubramani,V Srinivasa Chakravarthy,Ahmed A Moustafa,Balaraman Ravindran

doi:10.3389/fncom.2014.00047

Pragathi P Balasubramani, V Srinivasa Chakravarthy + Show 2 more

Open Access

https://doi.org/10.3389/fncom.2014.00047

Copy DOI

Abstract

Although empirical and neural studies show that serotonin (5HT) plays many functional roles in the brain, prior computational models mostly focus on its role in behavioral inhibition. In this study, we present a model of risk based decision making in a modified Reinforcement Learning (RL)-framework. The model depicts the roles of dopamine (DA) and serotonin (5HT) in Basal Ganglia (BG). In this model, the DA signal is represented by the temporal difference error (δ), while the 5HT signal is represented by a parameter (α) that controls risk prediction error. This formulation that accommodates both 5HT and DA reconciles some of the diverse roles of 5HT particularly in connection with the BG system. We apply the model to different experimental paradigms used to study the role of 5HT: (1) Risk-sensitive decision making, where 5HT controls risk assessment, (2) Temporal reward prediction, where 5HT controls time-scale of reward prediction, and (3) Reward/Punishment sensitivity, in which the punishment prediction error depends on 5HT levels. Thus the proposed integrated RL model reconciles several existing theories of 5HT and DA in the BG.

Highlights

Monoamine neuromodulators such as dopamine, serotonin, norepinephrine and acetylcholine are hailed to be the most promising neural messengers to ensure healthy adaptation to our uncertain environments
The semblance between the temporal difference error (TD) error signal and DA signal served as a starting point of an extensive theoretical and experimental effort to apply concepts of Reinforcement Learning (RL) to understand the functions of the Basal Ganglia (BG) (Schultz et al, 1997; Sutton and Barto, 1998; Joel et al, 2002; Chakravarthy et al, 2010)
RESULTS we apply the model of 5HT and DA in BG (Section Methods) to explain several risk-based decision making phenomena pertaining to BG function

Summary

Introduction

Monoamine neuromodulators such as dopamine, serotonin, norepinephrine and acetylcholine are hailed to be the most promising neural messengers to ensure healthy adaptation to our uncertain environments. The semblance between the TD error signal and DA signal served as a starting point of an extensive theoretical and experimental effort to apply concepts of RL to understand the functions of the Basal Ganglia (BG) (Schultz et al, 1997; Sutton and Barto, 1998; Joel et al, 2002; Chakravarthy et al, 2010). This led to the emergence of a framework for understanding the BG functions in which the DA signal played a crucial role. Deficiency of such a neuromodulator (DA) leads to symptoms observed in neurodegenerative disorders like Parkinson’s Disease (Bertler and Rosengren, 1966; Goetz et al, 2001)

Methods

Results

Conclusion