Negative Reward Prediction Error Research Articles

Principal components analysis of reward prediction errors in a reinforcement learning task

Models of reinforcement learning represent reward and punishment in terms of reward prediction errors (RPEs), quantitative signed terms describing the degree to which outcomes are better than expected (positive RPEs) or worse (negative RPEs). An electrophysiological component known as feedback related negativity (FRN) occurs at frontocentral sites 240–340ms after feedback on whether a reward or punishment is obtained, and has been claimed to neurally encode an RPE. An outstanding question however, is whether the FRN is sensitive to the size of both positive RPEs and negative RPEs. Previous attempts to answer this question have examined the simple effects of RPE size for positive RPEs and negative RPEs separately. However, this methodology can be compromised by overlap from components coding for unsigned prediction error size, or “salience”, which are sensitive to the absolute size of a prediction error but not its valence. In our study, positive and negative RPEs were parametrically modulated using both reward likelihood and magnitude, with principal components analysis used to separate out overlying components. This revealed a single RPE encoding component responsive to the size of positive RPEs, peaking at ~330ms, and occupying the delta frequency band. Other components responsive to unsigned prediction error size were shown, but no component sensitive to negative RPE size was found.

Impaired Neural Response to Negative Prediction Errors in Cocaine Addiction

Learning can be guided by unexpected success or failure, signaled via dopaminergic positive reward prediction error (+RPE) and negative reward-prediction error (-RPE) signals, respectively. Despite conflicting empirical evidence, RPE signaling is thought to be impaired in drug addiction. To resolve this outstanding question, we studied as a measure of RPE the feedback negativity (FN) that is sensitive to both reward and the violation of expectation. We examined FN in 25 healthy controls; 25 individuals with cocaine-use disorder (CUD) who tested positive for cocaine on the study day (CUD+), indicating cocaine use within the past 72 h; and in 25 individuals with CUD who tested negative for cocaine (CUD-). EEG was acquired while the participants performed a gambling task predicting whether they would win or lose money on each trial given three known win probabilities (25, 50, or 75%). FN was scored for the period in each trial when the actual outcome (win or loss) was revealed. A significant interaction between prediction, outcome, and group revealed that controls showed increased FN to unpredicted compared with predicted wins (i.e., intact +RPE) and decreased FN to unpredicted compared with predicted losses (i.e., intact -RPE). However, neither CUD subgroup showed FN modulation to loss (i.e., impaired -RPE), and unlike CUD+ individuals, CUD- individuals also did not show FN modulation to win (i.e., impaired +RPE). Thus, using FN, the current study directly documents -RPE deficits in CUD individuals. The mechanisms underlying -RPE signaling impairments in addiction may contribute to the disadvantageous nature of excessive drug use, which can persist despite repeated unfavorable life experiences (e.g., frequent incarcerations).

Do learning rates adapt to the distribution of rewards?

Studies of reinforcement learning have shown that humans learn differently in response to positive and negative reward prediction errors, a phenomenon that can be captured computationally by positing asymmetric learning rates. This asymmetry, motivated by neurobiological and cognitive considerations, has been invoked to explain learning differences across the lifespan as well as a range of psychiatric disorders. Recent theoretical work, motivated by normative considerations, has hypothesized that the learning rate asymmetry should be modulated by the distribution of rewards across the available options. In particular, the learning rate for negative prediction errors should be higher than the learning rate for positive prediction errors when the average reward rate is high, and this relationship should reverse when the reward rate is low. We tested this hypothesis in a series of experiments. Contrary to the theoretical predictions, we found that the asymmetry was largely insensitive to the average reward rate; instead, the dominant pattern was a higher learning rate for negative than for positive prediction errors, possibly reflecting risk aversion.

Cognitive flexibility in adolescence: Neural and behavioral mechanisms of reward prediction error processing in adaptive decision making during development

Adolescence is associated with quickly changing environmental demands which require excellent adaptive skills and high cognitive flexibility. Feedback-guided adaptive learning and cognitive flexibility are driven by reward prediction error (RPE) signals, which indicate the accuracy of expectations and can be estimated using computational models. Despite the importance of cognitive flexibility during adolescence, only little is known about how RPE processing in cognitive flexibility deviates between adolescence and adulthood.In this study, we investigated the developmental aspects of cognitive flexibility by means of computational models and functional magnetic resonance imaging (fMRI). We compared the neural and behavioral correlates of cognitive flexibility in healthy adolescents (12–16years) to adults performing a probabilistic reversal learning task. Using a modified risk-sensitive reinforcement learning model, we found that adolescents learned faster from negative RPEs than adults. The fMRI analysis revealed that within the RPE network, the adolescents had a significantly altered RPE-response in the anterior insula. This effect seemed to be mainly driven by increased responses to negative prediction errors.In summary, our findings indicate that decision making in adolescence goes beyond merely increased reward-seeking behavior and provides a developmental perspective to the behavioral and neural mechanisms underlying cognitive flexibility in the context of reinforcement learning.

The hypothetical cost-conflict monitor: is it a possible trigger for conflict-driven control mechanisms in the human brain?

The hypothetical cost-conflict monitor: is it a possible trigger for conflict-driven control mechanisms in the human brain?

Mediofrontal event-related potentials in response to positive, negative and unsigned prediction errors

Reinforcement learning models make use of reward prediction errors (RPEs), the difference between an expected and obtained reward. There is evidence that the brain computes RPEs, but an outstanding question is whether positive RPEs (“better than expected”) and negative RPEs (“worse than expected”) are represented in a single integrated system. An electrophysiological component, feedback related negativity, has been claimed to encode an RPE but its relative sensitivity to the utility of positive and negative RPEs remains unclear. This study explored the question by varying the utility of positive and negative RPEs in a design that controlled for other closely related properties of feedback and could distinguish utility from salience. It revealed a mediofrontal sensitivity to utility, for positive RPEs at 275–310ms and for negative RPEs at 310–390ms. These effects were preceded and succeeded by a response consistent with an unsigned prediction error, or “salience” coding.

Striatal dopamine ramping may indicate flexible reinforcement learning with forgetting in the cortico-basal ganglia circuits.

It has been suggested that the midbrain dopamine (DA) neurons, receiving inputs from the cortico-basal ganglia (CBG) circuits and the brainstem, compute reward prediction error (RPE), the difference between reward obtained or expected to be obtained and reward that had been expected to be obtained. These reward expectations are suggested to be stored in the CBG synapses and updated according to RPE through synaptic plasticity, which is induced by released DA. These together constitute the “DA=RPE” hypothesis, which describes the mutual interaction between DA and the CBG circuits and serves as the primary working hypothesis in studying reward learning and value-based decision-making. However, recent work has revealed a new type of DA signal that appears not to represent RPE. Specifically, it has been found in a reward-associated maze task that striatal DA concentration primarily shows a gradual increase toward the goal. We explored whether such ramping DA could be explained by extending the “DA=RPE” hypothesis by taking into account biological properties of the CBG circuits. In particular, we examined effects of possible time-dependent decay of DA-dependent plastic changes of synaptic strengths by incorporating decay of learned values into the RPE-based reinforcement learning model and simulating reward learning tasks. We then found that incorporation of such a decay dramatically changes the model's behavior, causing gradual ramping of RPE. Moreover, we further incorporated magnitude-dependence of the rate of decay, which could potentially be in accord with some past observations, and found that near-sigmoidal ramping of RPE, resembling the observed DA ramping, could then occur. Given that synaptic decay can be useful for flexibly reversing and updating the learned reward associations, especially in case the baseline DA is low and encoding of negative RPE by DA is limited, the observed DA ramping would be indicative of the operation of such flexible reward learning.

Phasic Dopamine Release in the Rat Nucleus Accumbens Symmetrically Encodes a Reward Prediction Error Term

Making predictions about the rewards associated with environmental stimuli and updating those predictions through feedback is an essential aspect of adaptive behavior. Theorists have argued that dopamine encodes a reward prediction error (RPE) signal that is used in such a reinforcement learning process. Recent work with fMRI has demonstrated that the BOLD signal in dopaminergic target areas meets both necessary and sufficient conditions of an axiomatic model of the RPE hypothesis. However, there has been no direct evidence that dopamine release itself also meets necessary and sufficient criteria for encoding an RPE signal. Further, the fact that dopamine neurons have low tonic firing rates that yield a limited dynamic range for encoding negative RPEs has led to significant debate about whether positive and negative prediction errors are encoded on a similar scale. To address both of these issues, we used fast-scan cyclic voltammetry to measure reward-evoked dopamine release at carbon fiber electrodes chronically implanted in the nucleus accumbens core of rats trained on a probabilistic decision-making task. We demonstrate that dopamine concentrations transmit a bidirectional RPE signal with symmetrical encoding of positive and negative RPEs. Our findings strengthen the case that changes in dopamine concentration alone are sufficient to encode the full range of RPEs necessary for reinforcement learning.

Prolonged dopamine signalling in striatum signals proximity and value of distant rewards

Predictions about future rewarding events have a powerful influence on behaviour. The phasic spike activity of dopamine-containing neurons, and corresponding dopamine transients in the striatum, are thought to underlie these predictions, encoding positive and negative reward prediction errors1–5. Many behaviours, however, are directed toward distant goals, for which transient signals might fail to provide sustained drive. Here we report a novel, extended mode of reward-predictive dopamine signalling in the striatum that emerged as rats moved toward distant goals. These dopamine signals, which were detected with fast-scan cyclic voltammetry (FSCV), gradually increased or--in rare instances--decreased as the animals navigated mazes to reach remote rewards, rather than having phasic or steady tonic profiles. These dopamine increases (ramps) scaled flexibly with both the distance and size of the rewards. During learning, these dopamine signals exhibited spatial preferences for goals in different locations and readily changed in magnitude to reflect changing values of the distant rewards. Such prolonged dopamine signalling could provide sustained motivational drive, a control mechanism that may be important for normal behaviour and that can be impaired in a range of neurologic and neuropsychiatric disorders.

The Feedback-Related Negativity Signals Salience Prediction Errors, Not Reward Prediction Errors

Modulations of the feedback-related negativity (FRN) event-related potential (ERP) have been suggested as a potential biomarker in psychopathology. A dominant theory about this signal contends that it reflects the operation of the neural system underlying reinforcement learning in humans. The theory suggests that this frontocentral negative deflection in the ERP 230-270 ms after the delivery of a probabilistic reward expresses a prediction error signal derived from midbrain dopaminergic projections to the anterior cingulate cortex. We tested this theory by investigating whether FRN will also be observed for an inherently aversive outcome: physical pain. In another session, the outcome was monetary reward instead of pain. As predicted, unexpected reward omissions (a negative reward prediction error) yielded a more negative deflection relative to unexpected reward delivery. Surprisingly, unexpected pain omission (a positive reward prediction error) also yielded a negative deflection relative to unexpected pain delivery. Our data challenge the theory by showing that the FRN expresses aversive prediction errors with the same sign as reward prediction errors. Both FRNs were spatiotemporally and functionally equivalent. We suggest that FRN expresses salience prediction errors rather than reward prediction errors.

Stimulus-Response-Outcome Coding in the Pigeon Nidopallium Caudolaterale

A prerequisite for adaptive goal-directed behavior is that animals constantly evaluate action outcomes and relate them to both their antecedent behavior and to stimuli predictive of reward or non-reward. Here, we investigate whether single neurons in the avian nidopallium caudolaterale (NCL), a multimodal associative forebrain structure and a presumed analogue of mammalian prefrontal cortex, represent information useful for goal-directed behavior. We subjected pigeons to a go-nogo task, in which responding to one visual stimulus (S+) was partially reinforced, responding to another stimulus (S–) was punished, and responding to test stimuli from the same physical dimension (spatial frequency) was inconsequential. The birds responded most intensely to S+, and their response rates decreased monotonically as stimuli became progressively dissimilar to S+; thereby, response rates provided a behavioral index of reward expectancy. We found that many NCL neurons' responses were modulated in the stimulus discrimination phase, the outcome phase, or both. A substantial fraction of neurons increased firing for cues predicting non-reward or decreased firing for cues predicting reward. Interestingly, the same neurons also responded when reward was expected but not delivered, and could thus provide a negative reward prediction error or, alternatively, signal negative value. In addition, many cells showed motor-related response modulation. In summary, NCL neurons represent information about the reward value of specific stimuli, instrumental actions as well as action outcomes, and therefore provide signals useful for adaptive behavior in dynamically changing environments.

Modulation of feedback-related negativity during trial-and-error exploration and encoding of behavioral shifts

The feedback-related negativity (FRN) is a mid-frontal event-related potential (ERP) recorded in various cognitive tasks and associated with the onset of sensory feedback signaling decision outcome. Some properties of the FRN are still debated, notably its sensitivity to positive and negative reward prediction error (RPE)—i.e., the discrepancy between the expectation and the actual occurrence of a particular feedback,—and its role in triggering the post-feedback adjustment. In the present study we tested whether the FRN is modulated by both positive and negative RPE. We also tested whether an instruction cue indicating the need for behavioral adjustment elicited the FRN. We asked 12 human subjects to perform a problem-solving task where they had to search by trial and error which of five visual targets, presented on a screen, was associated with a correct feedback. After exploration and discovery of the correct target, subjects could repeat their correct choice until the onset of a visual signal to change (SC) indicative of a new search. Analyses showed that the FRN was modulated by both negative and positive prediction error (RPE). Finally, we found that the SC elicited an FRN-like potential on the frontal midline electrodes that was not modulated by the probability of that event. Collectively, these results suggest the FRN may reflect a mechanism that evaluates any event (outcome, instruction cue) signaling the need to engage adaptive actions.

Diverse sources of reward value signals in the basal ganglia nuclei transmitted to the lateral habenula in the monkey

The lateral habenula (LHb) plays an important role in motivational decision making. Neurons in the primate LHb signal negative ‘reward prediction errors’ and inhibit midbrain dopamine (DA) neurons. These negative reward prediction error signals in the LHb are, at least partly, provided by a distinct group of neurons in the border region of the globus pallidus internal segment (GPb). However, it is still unclear whether other basal ganglia nuclei provide the LHb with reward signals, either through the GPb or through different circuits. As a first step to answer this question, we electrically stimulated various parts of the basal ganglia and monitored the neural activity in the LHb in the awake monkey. First, we found that low intensity stimulations in the GPb and the internal segment of the globus pallidus (GPi) evoked a short latency (5 ms) excitatory response in LHb neurons. Second, LHb neurons were inhibited by stimulations in the ventral pallidum (VP). These results suggest that reward-related signals are transmitted to the LHb mainly through excitatory connections from the GPb and inhibitory connections from the VP. Finally, excitations or inhibitions are induced in LHb neurons from diverse but patchy regions in the striatum. These effects have considerably longer latencies, suggesting that they may be mediated by the GPb or the VP. The patchy nature of the stimulation effect raises the possibility that the striosomes are the source of reward-related signals transmitted to the LHb.

Encoding of Both Positive and Negative Reward Prediction Errors by Neurons of the Primate Lateral Prefrontal Cortex and Caudate Nucleus

Learning can be motivated by unanticipated success or unexpected failure. The former encourages us to repeat an action or activity, whereas the latter leads us to find an alternative strategy. Understanding the neural representation of these unexpected events is therefore critical to elucidate learning-related circuits. We examined the activity of neurons in the lateral prefrontal cortex (PFC) and caudate nucleus of monkeys as they performed a trial-and-error learning task. Unexpected outcomes were widely represented in both structures, and neurons driven by unexpectedly negative outcomes were as frequent as those activated by unexpectedly positive outcomes. Moreover, both positive and negative reward prediction errors (RPEs) were represented primarily by increases in firing rate, unlike the manner in which dopamine neurons have been observed to reflect these values. Interestingly, positive RPEs tended to appear with shorter latency than negative RPEs, perhaps reflecting the mechanism of their generation. Last, in the PFC but not the caudate, trial-by-trial variations in outcome-related activity were linked to the animals' subsequent behavioral decisions. More broadly, the robustness of RPE signaling by these neurons suggests that actor-critic models of reinforcement learning in which the PFC and particularly the caudate are considered primarily to be "actors" rather than "critics," should be reconsidered to include a prominent evaluative role for these structures.

Negative Reward Signals from the Lateral Habenula to Dopamine Neurons Are Mediated by Rostromedial Tegmental Nucleus in Primates

Lateral habenula (LHb) neurons signal negative "reward-prediction errors" and inhibit midbrain dopamine (DA) neurons. Yet LHb neurons are largely glutamatergic, indicating that this inhibition may occur through an intermediate structure. Recent studies in rats have suggested a candidate for this role, the GABAergic rostromedial tegmental nucleus (RMTg), but this neural pathway has not yet been tested directly. We now show using electrophysiology and anatomic tracing that (1) the monkey has an inhibitory structure similar to the rat RMTg; (2) RMTg neurons receive excitatory input from the LHb, exhibit negative reward-prediction errors, and send axonal projections near DA soma; and (3) stimulating this structure inhibits DA neurons. Surprisingly, some RMTg neurons responded to reward cues earlier than the LHb, and carry "state-value" signals not found in DA neurons. Thus, our data suggest that the RMTg translates LHb reward-prediction errors (negative) into DA reward-prediction errors (positive), while transmitting additional motivational signals to non-DA networks.

Activation of dorsal raphe serotonin neurons underlies waiting for delayed rewards.

The serotonergic system plays a key role in the control of impulsive behaviors. Forebrain serotonin depletion leads to premature actions and steepens discounting of delayed rewards. However, there has been no direct evidence for serotonin neuron activity in relation to actions for delayed rewards. Here we show that serotonin neurons increase their tonic firing while rats wait for food and water rewards and conditioned reinforcement tones. The rate of tonic firing during the delay period was significantly higher for rewards than for tones, for which rats could not wait as long. When the delay was extended, tonic firing persisted until reward or tone delivery. When rats gave up waiting because of extended delay or reward omission, serotonin neuron firing dropped preceding the exit from reward sites. Serotonin neurons did not show significant response when an expected reward was omitted, which was predicted by the theory that serotonin signals negative reward prediction errors. These results suggest that increased serotonin neuron firing facilitates a rat's waiting behavior in prospect of forthcoming rewards and that higher serotonin activation enables longer waiting.

Chemosensory Learning in the Cortex

Taste is a primary reinforcer. Olfactory–taste and visual–taste association learning takes place in the primate including human orbitofrontal cortex to build representations of flavor. Rapid reversal of this learning can occur using a rule-based learning system that can be reset when an expected taste or flavor reward is not obtained, that is by negative reward prediction error, to which a population of neurons in the orbitofrontal cortex responds. The representation in the orbitofrontal cortex but not the primary taste or olfactory cortex is of the reward value of the visual/olfactory/taste input as shown by devaluation experiments in which food is fed to satiety, and by correlations of the activations with subjective pleasantness ratings in humans. Sensory-specific satiety for taste, olfactory, visual, and oral somatosensory inputs produced by feeding a particular food to satiety is implemented it is proposed by medium-term synaptic adaptation in the orbitofrontal cortex. Cognitive factors, including word-level descriptions, modulate the representation of the reward value of food in the orbitofrontal cortex, and this effect is learned it is proposed by associative modification of top-down synapses onto neurons activated by bottom-up taste and olfactory inputs when both are active in the orbitofrontal cortex. A similar associative synaptic learning process is proposed to be part of the mechanism for the top-down attentional control to the reward value vs. the sensory properties such as intensity of taste and olfactory inputs in the orbitofrontal cortex, as part of a biased activation theory of selective attention.

The Impact of Disappointment in Decision Making: Inter-Individual Differences and Electrical Neuroimaging

Disappointment, the emotion experienced when faced to reward prediction errors (RPEs), considerably impacts decision making (DM). Individuals tend to modify their behavior in an often unpredictable way just to avoid experiencing negative emotions. Despite its importance, disappointment remains much less studied than regret and its impact on upcoming decisions largely unexplored. Here, we adapted the Trust Game to effectively elicit, quantify, and isolate disappointment by relying on the formal definition provided by Bell's in economics. We evaluated the effects of experienced disappointment and elation on future cooperation and trust as well as the rationality and utility of the different behavioral and neural mechanisms used to cope with disappointment. All participants in our game trusted less and particularly expected less from unknown opponents as a result of disappointing outcomes in the previous trial but not necessarily after elation indicating that behavioral consequences of positive and negative RPEs are not the same. A large variance in the tolerance to disappointment was observed across subjects, with some participants needing only a small disappointment to impulsively bias their subsequent decisions. As revealed by high-density EEG recordings the most tolerant individuals – who thought twice before making a decision and earned more money – relied on different neural generators to contend with neutral and unexpected outcomes. This study thus provides some support to the idea that different neural systems underlie reflexive and reflective decisions within the same individuals as predicted by the dual-system theory of social judgment and DM.

Manipulation of feedback expectancy and valence induces negative and positive reward prediction error signals manifest in event‐related brain potentials

The feedback-related negativity (FRN) has been hypothesized to be most sensitive to unexpected negative feedback. The present study investigated feedback expectancy and valence using a probabilistic gambling paradigm where subjects encountered expected or unexpected positive and negative feedback outcomes. In line with previous studies, FRN amplitude reflected a negative reward prediction error, but to a minor extent also a positive reward prediction error. Moreover, the P300 amplitude was largest after unexpected feedback, irrespective of valence. We propose to interpret the FRN in terms of a reinforcement learning signal which is detecting mismatch between internal and external representations indexed by the ACC to extract motivationally salient outcomes.

MEASURING BELIEFS AND REWARDS: A NEUROECONOMIC APPROACH.

The neurotransmitter dopamine is central to the emerging discipline of neuroeconomics; it is hypothesized to encode the difference between expected and realized rewards and thereby to mediate belief formation and choice. We develop the first formal test of this theory of dopaminergic function, based on a recent axiomatization by Caplin and Dean [2008A]. These tests are satisfied by neural activity in the nucleus accumbens, an area rich in dopamine receptors. We find evidence for separate positive and negative reward prediction error signals, suggesting that behavioral asymmetries in response to losses and gains may parallel asymmetries in nucleus accumbens activity.