Computation Of Reward Prediction Error Research Articles

Neuronal implementation of the temporal difference learning algorithm in the midbrain dopaminergic system.

The temporal difference learning (TDL) algorithm has been essential to conceptualizing the role of dopamine in reinforcement learning (RL). Despite its theoretical importance, it remains unknown whether a neuronal implementation of this algorithm exists in the brain. Here, we provide an interpretation of the recently described signaling properties of ventral tegmental area (VTA) GABAergic neurons and show that a circuitry of these neurons implements the TDL algorithm. Specifically, we identified the neuronal mechanism of three key components of the TDL model: a sustained state value signal encoded by an afferent input to the VTA, a temporal differentiation circuit formed by two types of VTA GABAergic neurons the combined output of which computes momentary reward prediction (RP) as the derivative of the state value, and the computation of reward prediction errors (RPEs) in dopamine neurons utilizing the output of the differentiation circuit. Using computational methods, we also show that this mechanism is optimally adapted to the biophysics of RPE signaling in dopamine neurons, mechanistically links the emergence of conditioned reinforcement to RP, and can naturally account for the temporal discounting of reinforcement. Elucidating the implementation of the TDL algorithm may further the investigation of RL in biological and artificial systems.

Striatal dopamine supports reward expectation and learning: A simultaneous PET/fMRI study

Converging evidence from both human neuroimaging and animal studies has supported a model of mesolimbic processing underlying reward learning behaviors, based on the computation of reward prediction errors. However, competing evidence supports human dopamine signaling in the basal ganglia as also contributing to the generation of higher order learning heuristics. Here, we present data from a large (N = 81, 18–30yo), multi-modal neuroimaging study using simultaneously acquired task fMRI, affording temporal resolution of reward system function, and PET imaging with [11C]Raclopride (RAC), assessing striatal dopamine (DA) D2/3 receptor binding, during performance of a probabilistic reward learning task. Both fMRI activation and PET DA measures showed ventral striatum involvement for signaling rewards. However, greater DA release was uniquely associated with learning strategies (i.e., learning rates) that were more task-optimal within the best fitting reinforcement learning model. This DA response was associated with BOLD activation of a network of regions including anterior cingulate cortex, medial prefrontal cortex, thalamus and posterior parietal cortex, primarily during expectation, rather than prediction error, task epochs. Together, these data provide novel, human in vivo evidence that striatal dopaminergic signaling interacts with a network of cortical regions to generate task-optimal learning strategies, rather than representing reward outcomes in isolation.

The nucleus accumbens and inhibition in the ventral tegmental area play a causal role in the Kamin blocking effect.

The discovery of Kamin blocking led to the idea that associative learning occurs only when there is a mismatch between actual and predicted outcomes, or prediction error. The neural substrates involved in regulating this prediction error during behavioral learning are still not fully elucidated. We investigated in rats the role of the ventral tegmental area and the nucleus accumbens in Kamin blocking. Our blocking paradigm involved three phases: appetitive classical conditioning of a lever cue, conditioning of a compound of the lever cue plus an auditory cue, and testing response to the auditory cue in extinction. We found that disruption of inhibition in the ventral tegmental area by bicuculline, or designer receptor mediated inactivation of the nucleus accumbens, during compound cue conditioning, attenuated Kamin blocking. These results suggest that inhibition in the ventral tegmental area and inhibitory output from the nucleus accumbens are necessary for blocking and make behaviorally significant contributions to the computation of reward prediction error. In addition, we found that inactivating the neurons in the nucleus accumbens during classical conditioning of the lever cue also attenuated blocking, without affecting classical conditioning of the lever. This indicates that learning in the nucleus accumbens is necessary for blocking and reward estimation. Our results reveal a causal role for nucleus accumbens modulated inhibitory inputs to the ventral tegmental area in the blocking effect and suggest that they contribute to computation of reward prediction error during associative learning.

Aberrant reward prediction error during Pavlovian appetitive learning in alexithymia.

Extensive literature shows that alexithymia, a subclinical trait defined by difficulties in identifying and describing feelings, is characterized by multifaceted impairments in processing emotional stimuli. Nevertheless, its underlying mechanisms remain elusive. Here, we hypothesize that alexithymia may be characterized by an alteration in learning the emotional value of encountered stimuli and test this by assessing differences between individuals with low (LA) and high (HA) levels of alexithymia in the computation of reward prediction errors (RPEs) during Pavlovian appetitive conditioning. As a marker of RPE, the amplitude of the feedback-related negativity (FRN) event-related potential was assessed while participants were presented with two conditioned stimuli (CS) associated with expected or unexpected feedback, indicating delivery of reward or no-reward. No-reward (vs reward) feedback elicited the FRN both in LA and HA. However, unexpected (vs expected) feedback enhanced the FRN in LA but not in HA, indicating impaired computation of RPE in HA. Thus, although HA show preserved sensitivity to rewards, they cannot use this response to update the value of CS that predict them. This impairment may hinder the construction of internal representations of emotional stimuli, leaving individuals with alexithymia unable to effectively recognize, respond and regulate their response to emotional stimuli.

Working Memory Load Strengthens Reward Prediction Errors.

Reinforcement learning (RL) in simple instrumental tasks is usually modeled as a monolithic process in which reward prediction errors (RPEs) are used to update expected values of choice options. This modeling ignores the different contributions of different memory and decision-making systems thought to contribute even to simple learning. In an fMRI experiment, we investigated how working memory (WM) and incremental RL processes interact to guide human learning. WM load was manipulated by varying the number of stimuli to be learned across blocks. Behavioral results and computational modeling confirmed that learning was best explained as a mixture of two mechanisms: a fast, capacity-limited, and delay-sensitive WM process together with slower RL. Model-based analysis of fMRI data showed that striatum and lateral prefrontal cortex were sensitive to RPE, as shown previously, but, critically, these signals were reduced when the learning problem was within capacity of WM. The degree of this neural interaction related to individual differences in the use of WM to guide behavioral learning. These results indicate that the two systems do not process information independently, but rather interact during learning.SIGNIFICANCE STATEMENT Reinforcement learning (RL) theory has been remarkably productive at improving our understanding of instrumental learning as well as dopaminergic and striatal network function across many mammalian species. However, this neural network is only one contributor to human learning and other mechanisms such as prefrontal cortex working memory also play a key role. Our results also show that these other players interact with the dopaminergic RL system, interfering with its key computation of reward prediction errors.

Stimulus-dependent adjustment of reward prediction error in the midbrain.

Previous reports have described that neural activities in midbrain dopamine areas are sensitive to unexpected reward delivery and omission. These activities are correlated with reward prediction error in reinforcement learning models, the difference between predicted reward values and the obtained reward outcome. These findings suggest that the reward prediction error signal in the brain updates reward prediction through stimulus–reward experiences. It remains unknown, however, how sensory processing of reward-predicting stimuli contributes to the computation of reward prediction error. To elucidate this issue, we examined the relation between stimulus discriminability of the reward-predicting stimuli and the reward prediction error signal in the brain using functional magnetic resonance imaging (fMRI). Before main experiments, subjects learned an association between the orientation of a perceptually salient (high-contrast) Gabor patch and a juice reward. The subjects were then presented with lower-contrast Gabor patch stimuli to predict a reward. We calculated the correlation between fMRI signals and reward prediction error in two reinforcement learning models: a model including the modulation of reward prediction by stimulus discriminability and a model excluding this modulation. Results showed that fMRI signals in the midbrain are more highly correlated with reward prediction error in the model that includes stimulus discriminability than in the model that excludes stimulus discriminability. No regions showed higher correlation with the model that excludes stimulus discriminability. Moreover, results show that the difference in correlation between the two models was significant from the first session of the experiment, suggesting that the reward computation in the midbrain was modulated based on stimulus discriminability before learning a new contingency between perceptually ambiguous stimuli and a reward. These results suggest that the human reward system can incorporate the level of the stimulus discriminability flexibly into reward computations by modulating previously acquired reward values for a typical stimulus.

Modeling of reward-regulated learning impairment in alchohol addiction

Event Abstract Back to Event Modeling of reward-regulated learning impairment in alchohol addiction Sinem Balta Beylergil1*, Anne Beck2, Florian Schlagenhauf2, Michael Rapp2, Klaus Obermayer1 and Andreas Heinz2 1 Bernstein Center for Computational Neuroscience, Germany 2 Department of Psychiatry and Psychotherapy, Charité Universitätsmedizin, Germany Alcohol addiction is characterized by compulsive and uncontrolled consumption of alcohol despite its adverse consequences. It is known from previous studies that alcohol-dependent individuals have difficulties in integrating reinforcements to guide future behavior. Main causes of these difficulties might be abnormal computation of reward prediction errors (PEs) and inability to update the value of behavioral options. In this study the aim is modeling the performances of 37 alcohol-dependent patients and 41 healthy controls in a two-alternative forced choice probabilistic reversal task. The subjects first have to learn which of the two choices is more rewarding, and then flexibly switch their choices when reward-contingencies change. We hypothesize that behavioral data of experimental and control groups will have distinct computational models showing discrepancy between their learning patterns. The choice behavior trace of each individual is modeled using Rescorla-Wagner reinforcement theory. The model is optimized with respect to maximum log-likelihood estimate (MLE) such that minimum MLE ensures the best fit of the model to the data. Individual-specific model fitting parameters are derived from the best fitting model in order to show experimental group members might have difficulties in adapting dynamic reward contingencies. Expected values and PEs are generated using these model parameters and convolved with haemodynamic response function (HRF) to generate regressors used in parametrical fMRI analysis. This combined neuroimaging and computational modeling approach lets us demonstrate reward-regulated learning impairment in alcohol-dependent subjects compared to healthy controls. Funding: Deutsche Forschungsgemeinschaft (DFG). Keywords: Alchohol addiction, decision-making Conference: XI International Conference on Cognitive Neuroscience (ICON XI), Palma, Mallorca, Spain, 25 Sep - 29 Sep, 2011. Presentation Type: Poster Presentation Topic: Poster Sessions: Decision Making, Reward Processing & Response Selection Citation: Balta Beylergil S, Beck A, Schlagenhauf F, Rapp M, Obermayer K and Heinz A (2011). Modeling of reward-regulated learning impairment in alchohol addiction. Conference Abstract: XI International Conference on Cognitive Neuroscience (ICON XI). doi: 10.3389/conf.fnhum.2011.207.00367 Copyright: The abstracts in this collection have not been subject to any Frontiers peer review or checks, and are not endorsed by Frontiers. They are made available through the Frontiers publishing platform as a service to conference organizers and presenters. The copyright in the individual abstracts is owned by the author of each abstract or his/her employer unless otherwise stated. Each abstract, as well as the collection of abstracts, are published under a Creative Commons CC-BY 4.0 (attribution) licence (https://creativecommons.org/licenses/by/4.0/) and may thus be reproduced, translated, adapted and be the subject of derivative works provided the authors and Frontiers are attributed. For Frontiers’ terms and conditions please see https://www.frontiersin.org/legal/terms-and-conditions. Received: 23 Nov 2011; Published Online: 28 Nov 2011. * Correspondence: Dr. Sinem Balta Beylergil, Bernstein Center for Computational Neuroscience, Berlin, Germany, sinembalta@gmail.com Login Required This action requires you to be registered with Frontiers and logged in. To register or login click here. Abstract Info Abstract The Authors in Frontiers Sinem Balta Beylergil Anne Beck Florian Schlagenhauf Michael Rapp Klaus Obermayer Andreas Heinz Google Sinem Balta Beylergil Anne Beck Florian Schlagenhauf Michael Rapp Klaus Obermayer Andreas Heinz Google Scholar Sinem Balta Beylergil Anne Beck Florian Schlagenhauf Michael Rapp Klaus Obermayer Andreas Heinz PubMed Sinem Balta Beylergil Anne Beck Florian Schlagenhauf Michael Rapp Klaus Obermayer Andreas Heinz Related Article in Frontiers Google Scholar PubMed Abstract Close Back to top Javascript is disabled. Please enable Javascript in your browser settings in order to see all the content on this page.

Different Pedunculopontine Tegmental Neurons Signal Predicted and Actual Task Rewards

The dopamine system has been implicated in guiding behavior based on rewards. The pedunculopontine tegmental nucleus (PPTN) of the brainstem receives afferent inputs from reward-related structures, including the cerebral cortices and the basal ganglia, and in turn provides strong excitatory projections to dopamine neurons. This anatomical evidence predicts that PPTN neurons may carry reward information. To elucidate the functional role of the PPTN in reward-seeking behavior, we recorded single PPTN neurons while monkeys performed a visually guided saccade task in which the predicted reward value was informed by the shape of the fixation target. Two distinct groups of neurons, fixation target (FT) and reward delivery (RD) neurons, carried reward information. The activity of FT neurons persisted between FT onset and reward delivery, with the level of activity associated with the magnitude of the expected reward. RD neurons discharged phasically after reward delivery, with the levels of activity associated with the actual reward. These results suggest that separate populations of PPTN neurons signal predicted and actual reward values, both of which are necessary for the computation of reward prediction error as represented by dopamine neurons.

Can the apparent adaptation of dopamine neurons’ mismatch sensitivities be reconciled with their computation of reward prediction errors?

According to modern reinforcement learning theories, midbrain dopamine (DA) neurons are part of an adaptive system within which learned expectations filter reward-related signals to enable computation of reward prediction errors (RPEs). Recent electrophysiological data on DA neuron responses to probabilistic reward schedules inspired the idea that DA neurons might be adapting their mismatch sensitivities to reflect variances of expected rewards. Taken literally as a mathematical hypothesis, this idea contradicts reinforcement learning theory, and most computational models of basal ganglia learning. Here, we report a qualitative mathematical derivation of the implications of a generic class of circuit models for learning to compute RPEs. This analysis and concordant circuit simulations, both of which predict DA neuron responses on probabilistic schedules, support a reinterpretation of the electrophysiological data that is fully compatible with the examined class of RPE models. This reinterpretation implies a novel and readily testable prediction.

Reward Prediction Error Computation in the Pedunculopontine Tegmental Nucleus Neurons

In this article, we address the role of neuronal activity in the pathways of the brainstem-midbrain circuit in reward and the basis for believing that this circuit provides advantages over previous reinforcement learning theory. Several lines of evidence support the reward-based learning theory proposing that midbrain dopamine (DA) neurons send a teaching signal (the reward prediction error signal) to control synaptic plasticity of the projection area. However, the underlying mechanism of where and how the reward prediction error signal is computed still remains unclear. Since the pedunculopontine tegmental nucleus (PPTN) in the brainstem is one of the strongest excitatory input sources to DA neurons, we hypothesized that the PPTN may play an important role in activating DA neurons and reinforcement learning by relaying necessary signals for reward prediction error computation to DA neurons. To investigate the involvement of the PPTN neurons in computation of reward prediction error, we used a visually guided saccade task (VGST) during recording of neuronal activity in monkeys. Here, we predict that PPTN neurons may relay the excitatory component of tonic reward prediction and phasic primary reward signals, and derive a new computational theory of the reward prediction error in DA neurons.

Reward Prediction Error Computation in the Pedunculopontine Tegmental Nucleus Neurons

In this article, we address the role of neuronal activity in the pathways of the brainstem-midbrain circuit in reward and the basis for believing that this circuit provides advantages over previous reinforcement learning theory. Several lines of evidence support the reward-based learning theory proposing that midbrain dopamine (DA) neurons send a teaching signal (the reward prediction error signal) to control synaptic plasticity of the projection area. However, the underlying mechanism of where and how the reward prediction error signal is computed still remains unclear. Since the pedunculopontine tegmental nucleus (PPTN) in the brainstem is one of the strongest excitatory input sources to DA neurons, we hypothesized that the PPTN may play an important role in activating DA neurons and reinforcement learning by relaying necessary signals for reward prediction error computation to DA neurons. To investigate the involvement of the PPTN neurons in computation of reward prediction error, we used a visually guided saccade task (VGST) during recording of neuronal activity in monkeys. Here, we predict that PPTN neurons may relay the excitatory component of tonic reward prediction and phasic primary reward signals, and derive a new computational theory of the reward prediction error in DA neurons.