Reward Learning Task Research Articles

Expectancy-related changes in firing of dopamine neurons depend on orbitofrontal cortex

The orbitofrontal cortex has been hypothesized to carry information regarding the value of expected rewards. Such information is essential for associative learning, which relies on comparisons between expected and obtained reward for generating instructive error signals. These error signals are thought to be conveyed by dopamine neurons. To test whether orbitofrontal cortex contributes to these error signals, we recorded from dopamine neurons in orbitofrontal-lesioned rats performing a reward learning task. Lesions caused marked changes in dopaminergic error signaling. However, the effect of lesions was not consistent with a simple loss of information regarding expected value. Instead, without orbitofrontal input, dopaminergic error signals failed to reflect internal information about the impending response that distinguished externally similar states leading to differently valued future rewards. These results are consistent with current conceptualizations of orbitofrontal cortex as supporting model-based behavior and suggest an unexpected role for this information in dopaminergic error signaling.

Corticotropin-Releasing Hormone Receptor Type 1 (CRHR1) Genetic Variation and Stress Interact to Influence Reward Learning

Stress is a general risk factor for psychopathology, but the mechanisms underlying this relationship remain largely unknown. Animal studies and limited human research suggest that stress can induce anhedonic behavior. Moreover, emerging data indicate that genetic variation within the corticotropin-releasing hormone type 1 receptor gene (CRHR1) at rs12938031 may promote psychopathology, particularly in the context of stress. Using an intermediate phenotypic neurogenetics approach, we assessed how stress and CRHR1 genetic variation (rs12938031) influence reward learning, an important component of anhedonia. Psychiatrically healthy female participants (n = 75) completed a probabilistic reward learning task during stress and no-stress conditions while 128-channel event-related potentials were recorded. Fifty-six participants were also genotyped across CRHR1. Response bias, an individual's ability to modulate behavior as a function of reward, was the primary behavioral variable of interest. The feedback-related positivity (FRP) in response to reward feedback was used as a neural index of reward learning. Relative to the no-stress condition, acute stress was associated with blunted response bias as well as a smaller and delayed FRP (indicative of disrupted reward learning) and reduced anterior cingulate and orbitofrontal cortex activation to reward. Critically, rs12938031 interacted with stress to influence reward learning: both behaviorally and neurally, A homozygotes showed stress-induced reward learning abnormalities. These findings indicate that acute, uncontrollable stressors reduce participants' ability to modulate behavior as a function of reward, and that such effects are modulated by CRHR1 genotype. Homozygosity for the A allele at rs12938031 may increase risk for psychopathology via stress-induced reward learning deficits.

Tryptophan depletion disinhibits punishment but not reward prediction: implications for resilience

RationaleWe have previously shown that tryptophan depletion enhances punishment but not reward prediction (Cools et al. in Neuropsychopharmacology 33:2291–2299, 2008b). This provided evidence for a valence-specific role of serotonin (which declines under depleted tryptophan) in aversive processing. Recent theoretical (Dayan and Huys in PLoS Comput Biol 4:e4, 2008) and experimental (Crockett et al. in J Neurosci 29:11993–11999, 2009) approaches have, however, further specified this role by showing that serotonin is critical for punishment-induced inhibition.ObjectivesWe sought to examine the role of serotonin in punishment-induced inhibition. We also examined the impact of induced mood on this effect to assess whether effects of tryptophan depletion on affective inhibition are moderated by mood.MethodsHealthy females consumed a balanced amino acid mixture with (N = 20) or without (N = 21) the serotonin precursor tryptophan. Each subject completed either negative or neutral mood induction. All subjects completed the reward and punishment reversal learning task adopted in the previous study.ResultsWe demonstrate a punishment prediction impairment in individuals who consumed tryptophan which was absent in individuals who were depleted of tryptophan. This effect was impervious to mood state.ConclusionsOur results suggest that serotonin promotes the inhibition of responses to punishing outcomes. This may lead to reduced punishment prediction accuracy in the presence of tryptophan and may contribute to resilience to affective disorders. Reduction of serotonin via tryptophan depletion then removes this inhibition. As such, we highlight a mechanism by which reduced serotonin can contribute to disorders of impulsivity and compulsivity as well as disorders of emotion.

Expected value and prediction error abnormalities in depression and schizophrenia

The dopamine system has been linked to anhedonia in depression and both the positive and negative symptoms of schizophrenia, but it remains unclear how dopamine dysfunction could mechanistically relate to observed symptoms. There is considerable evidence that phasic dopamine signals encode prediction error (differences between expected and actual outcomes), with reinforcement learning theories being based on prediction error-mediated learning of associations. It has been hypothesized that abnormal encoding of neural prediction error signals could underlie anhedonia in depression and negative symptoms in schizophrenia by disrupting learning and blunting the salience of rewarding events, and contribute to psychotic symptoms by promoting aberrant perceptions and the formation of delusions. To test this, we used model based functional magnetic resonance imaging and an instrumental reward-learning task to investigate the neural correlates of prediction errors and expected-reward values in patients with depression (n=15), patients with schizophrenia (n=14) and healthy controls (n=17). Both patient groups exhibited abnormalities in neural prediction errors, but the spatial pattern of abnormality differed, with the degree of abnormality correlating with syndrome severity. Specifically, reduced prediction errors in the striatum and midbrain were found in depression, with the extent of signal reduction in the bilateral caudate, nucleus accumbens and midbrain correlating with increased anhedonia severity. In schizophrenia, reduced prediction error signals were observed in the caudate, thalamus, insula and amygdala-hippocampal complex, with a trend for reduced prediction errors in the midbrain, and the degree of blunting in the encoding of prediction errors in the insula, amygdala-hippocampal complex and midbrain correlating with increased severity of psychotic symptoms. Schizophrenia was also associated with disruption in the encoding of expected-reward values in the bilateral amygdala-hippocampal complex and parahippocampal gyrus, with the degree of disruption correlating with psychotic symptom severity. Neural signal abnormalities did not correlate with negative symptom severity in schizophrenia. These findings support the suggestion that a disruption in the encoding of prediction error signals contributes to anhedonia symptoms in depression. In schizophrenia, the findings support the postulate of an abnormality in error-dependent updating of inferences and beliefs driving psychotic symptoms. Phasic dopamine abnormalities in depression and schizophrenia are suggested by our observation of prediction error abnormalities in dopamine-rich brain areas, given the evidence for dopamine encoding prediction errors. The findings are consistent with proposals that psychiatric syndromes reflect different disorders of neural valuation and incentive salience formation, which helps bridge the gap between biological and phenomenological levels of understanding.

Social and monetary reward learning engage overlapping neural substrates

Learning to make choices that yield rewarding outcomes requires the computation of three distinct signals: stimulus values that are used to guide choices at the time of decision making, experienced utility signals that are used to evaluate the outcomes of those decisions and prediction errors that are used to update the values assigned to stimuli during reward learning. Here we investigated whether monetary and social rewards involve overlapping neural substrates during these computations. Subjects engaged in two probabilistic reward learning tasks that were identical except that rewards were either social (pictures of smiling or angry people) or monetary (gaining or losing money). We found substantial overlap between the two types of rewards for all components of the learning process: a common area of ventromedial prefrontal cortex (vmPFC) correlated with stimulus value at the time of choice and another common area of vmPFC correlated with reward magnitude and common areas in the striatum correlated with prediction errors. Taken together, the findings support the hypothesis that shared anatomical substrates are involved in the computation of both monetary and social rewards.

Deficits in sustaining reward responses in subsyndromal and syndromal major depression

Preliminary findings suggest a reduction in capacity to sustain reward responses in major depression. However, relatively little is known about the stability of reward learning over time and the effect of stress on reward responses in depressed individuals. This study aimed to evaluate sustained behaviour to maximize reward in the context of known reinforcement contingencies and to evaluate the extent to which stress influences such behaviour in clinically depressed patients (n=43), subsyndromally depressed individuals (n=43), and healthy controls (n=44). A probabilistic reward learning task with contingencies known to participants was used to evaluate the change of reward response over time in both 'stress' and 'non-stress' conditions. Stress was induced by salient negative feedback during the task performance. Questionnaires capturing subjective affect were also administered to all participants after completion of the task. Response bias to the stimulus signaling greater reward decreased significantly over time in both subsyndromally and clinically depressed participants, but not in healthy controls. Healthy controls demonstrated a trend of dysfunctional reward processing under the stress condition. Moreover, in the stress condition, the deficit in sustaining behaviour to maximize reward was associated with subjective rating of pleasure in participants with either subsyndromal depression or major depression. These findings suggest that individuals with depression have difficulty sustaining behaviour during a known reinforcement schedule. Participants with anhedonic symptoms are even less likely to sustain behaviour to maximize reward under stress.

Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study

Animal studies have found that the phasic activity of dopamine neurons during reward-related learning resembles a “prediction error” (PE) signal derived from a class of computational models called reinforcement learning (RL). An apparently similar signal can be measured using fMRI in the human striatum, a primary dopaminergic target. However, the fMRI signal does not measure dopamine per se, and therefore further evidence is needed to determine if these signals are related to each other. Parkinson's disease (PD) involves the neurodegeneration of the dopamine system and is accompanied by deficits in reward-related decision-making tasks. In the current study we used a computational RL model to assess striatal error signals in PD patients performing an RL task during fMRI scanning. Results show that error signals were preserved in ventral striatum of PD patients, but impaired in dorsolateral striatum, relative to healthy controls, a pattern reflecting the known selective anatomical degeneration of dopamine nuclei in PD. These findings support the notion that PE signals measured in the human striatum by the BOLD signal may reflect phasic DA activity. These results also provide evidence for a deficiency in PE signaling in the dorsolateral striatum of PD patients that may offer an explanation for their deficits observed in other reward learning tasks.

Neuroelectric Signatures of Reward Learning and Decision-Making in the Human Nucleus Accumbens

Learning that certain actions lead to risky rewards is critical for biological, social, and economic survival, but the precise neural mechanisms of such reward-guided learning remain unclear. Here, we show that the human nucleus accumbens plays a key role in learning about risks by representing reward value. We recorded electrophysiological activity directly from the nucleus accumbens of five patients undergoing deep brain stimulation for treatment of refractory major depression. Patients engaged in a simple reward-learning task in which they first learned stimulus-outcome associations (learning task), and then were able to choose from among the learned stimuli (choosing task). During the learning task, nucleus accumbens activity reflected potential and received reward values both during the cue stimulus and during the feedback. During the choosing task, there was no nucleus accumbens activity during the cue stimulus, but feedback-related activity was pronounced and similar to that during the learning task. This pattern of results is inconsistent with a prediction error response. Finally, analyses of cross-correlations between the accumbens and simultaneous recordings of medial frontal cortex suggest a dynamic interaction between these structures. The high spatial and temporal resolution of these recordings provides novel insights into the timing of activity in the human nucleus accumbens, its functions during reward-guided learning and decision-making, and its interactions with medial frontal cortex.

S.07.07 Seven days of citalopram reduces sensitivity to negative appraisal in a probabilistic learning task

Previous literature established a link between major depressive disorder (MDD) and altered reward processing as well as between empathy and (observational) reward learning. The aim of the present study was to assess the effects of MDD on the electrophysiological correlates – the feedback-related negativity (FRN) and the P300 – of active and observational reward processing and to relate them to trait cognitive and affective empathy. Eighteen patients with MDD and 16 healthy controls performed an active and an observational probabilistic reward-learning task while event- related potentials were recorded. Also, participants were assessed with regard to self-reported cognitive and affective trait empathy. Relative to healthy controls, patients with MDD showed overall impaired learning and attenuated FRN amplitudes, irrespective of feedback valence and learning type (active vs. observational), but comparable P300 amplitudes. In the patient group, but not in controls, higher trait perspective taking scores were significantly correlated with reduced FRN amplitudes. The pattern of results suggests impaired prediction error processing and a negative effect of higher trait empathy on feedback-based learning in patients with MDD.

Abnormal temporal difference reward-learning signals in major depression

Anhedonia is a core symptom of major depressive disorder (MDD), long thought to be associated with reduced dopaminergic function. However, most antidepressants do not act directly on the dopamine system and all antidepressants have a delayed full therapeutic effect. Recently, it has been proposed that antidepressants fail to alter dopamine function in antidepressant unresponsive MDD. There is compelling evidence that dopamine neurons code a specific phasic (short duration) reward-learning signal, described by temporal difference (TD) theory. There is no current evidence for other neurons coding a TD reward-learning signal, although such evidence may be found in time. The neuronal substrates of the TD signal were not explored in this study. Phasic signals are believed to have quite different properties to tonic (long duration) signals. No studies have investigated phasic reward-learning signals in MDD. Therefore, adults with MDD receiving long-term antidepressant medication, and comparison controls both unmedicated and acutely medicated with the antidepressant citalopram, were scanned using fMRI during a reward-learning task. Three hypotheses were tested: first, patients with MDD have blunted TD reward-learning signals; second, controls given an antidepressant acutely have blunted TD reward-learning signals; third, the extent of alteration in TD signals in major depression correlates with illness severity ratings. The results supported the hypotheses. Patients with MDD had significantly reduced reward-learning signals in many non-brainstem regions: ventral striatum (VS), rostral and dorsal anterior cingulate, retrosplenial cortex (RC), midbrain and hippocampus. However, the TD signal was increased in the brainstem of patients. As predicted, acute antidepressant administration to controls was associated with a blunted TD signal, and the brainstem TD signal was not increased by acute citalopram administration. In a number of regions, the magnitude of the abnormal signals in MDD correlated with illness severity ratings. The findings highlight the importance of phasic reward-learning signals, and are consistent with the hypothesis that antidepressants fail to normalize reward-learning function in antidepressant-unresponsive MDD. Whilst there is evidence that some antidepressants acutely suppress dopamine function, the long-term action of virtually all antidepressants is enhanced dopamine agonist responsiveness. This distinction might help to elucidate the delayed action of antidepressants. Finally, analogous to recent work in schizophrenia, the finding of abnormal phasic reward-learning signals in MDD implies that an integrated understanding of symptoms and treatment mechanisms is possible, spanning physiology, phenomenology and pharmacology.

Motivation, reward, and Parkinson’s disease: influence of dopatherapy

“Orbitofrontal” and “cingulate” striatofrontal loops and the mesolimbic dopaminergic system that modulates their function have been implicated in motivation and sensitivity to reinforcement in animals. Parkinson’s disease (PD) provides a model to assess their implications in humans. The aims of the study were to investigate motivation and sensitivity to reinforcement in non-demented and -depressed PD patients and to evaluate the influence of dopaminergic therapy by comparing patients in “on” (with l-Dopa) and “off” (without l-Dopa) states. Twenty-three PD patients were compared, in both the “on” and “off” states, to 28 controls, using: (1) an Apathy Scale; (2) Stimulus–Reward Learning, Reversal, and Extinction tasks; and (3) a Gambling task. PD patients were found: (1) mildly apathetic; (2) impaired on Stimulus–Reward Learning and Reversal, but not on Extinction; and (3) able to progress in the Gambling task during the first, but not the second assessment. There was no significant correlation between these various deficits. l-Dopa treatment clearly improved motivation, but had more limited and contrasting effects on other variables, decreasing the number of omission errors in Reversal, but increasing the number of perseveration errors in Extinction. These results suggest: (1) an implication of striatofrontal loops in human motivation and explicit and implicit sensitivity to reinforcement; (2) a positive influence of l-Dopa treatment on the subjective evaluation of motivation, but contrasting effects on reward sensitivity.

Chronic methanesulfonyl fluoride enhances one-trial per day reward learning in aged rats

Aged (24-months-old) rats were treated chronically with methanesulfonyl fluoride (MSF), an acetylcholinesterase (AChE) inhibitor with selectivity for central nervous system AChE, or with injection vehicle alone. Twelve 0.22 mg/kg IP injections were given over 4 weeks. MSF rats showed significantly greater speed and accuracy on a 1 trial/day discriminative reward learning task. The chronic MSF treatment resulted in a 56% decrease in brain AChE activity but no discemable locomotor side effects and no liver damage as indicated by aspartate transferase activity.

Galanin attenuates retention of one-trial reward learning

Galanin is a neuropetide that coexists with acetylcholine in the septohippocampal pathway. Galanin appears to have a negative modulating influence on cholinergic transmission, suggesting that it might interfere with memory formation on a one-trial discriminative reward learning task. The apparatus was a starburst maze with five radiating alleys, one an ascending baited alley. The subjects were 38 two to three month old Sprague-Dawley rats cannulated in the body of the lateral ventricles and deprived to 80% of initial weight. Ten rats were infused i.c.v. over six mins. with 8 μg galanin in 24 μl saline and 10 with saline alone. Twenty mins. after completion of infusion, each rat was placed in the maze and observed under “blind” conditions for number of errors (blind alleys entered) and latency to reach reward. Each rat's speed score was 100 sec./latency. One day later, each rat was retested in the maze. Each rat's retention scores were its decrease in errors and increase in speed between the single training trial and the retention trial. Galanin infused rats showed significantly less retention by both measures. In a second experiment, either the same dose of galanin or saline alone was infused 20 mins. before the retention trial. There was no significant effect, suggesting that galanin may interfere with memory formation rather than memory retrieval or task performance.

Methanesulfonyl fluoride enhances one-trial reward learning in mid-aged rats

In previous studies, 18-month-old rats have shown no significant retention 24 hours after the single acquisition trial in a one-trial discriminative reward learning task. In the present study, ten 18-month-old rats pretreated with 0.5 mg/kg MSF IP showed significantly better retention in terms of speed and errors than eleven 18-month-old rats pretreated with injection vehicle alone. However, twelve two-three-month-old rats pretreated with the same dose of MSF failed to show better retention than eleven two-three-month-old rats pretreated with vehicle alone.