Reward-related Signals Research Articles

355 Learning Related Power Changes in Caudate Nucleus

Abstract INTRODUCTION Associative learning is the process whereby a connection is formed between a sensory cue and an outcome resulting from a behavioral response. This process allows us to learn to adapt and optimally respond to a changing environment. Primate research has demonstrated that the caudate nucleus is involved in associative learning and contains information encoding whether a response was correct or incorrect. Our objective was to determine whether correlates of learning are present in the human caudate nucleus and to differentiate between learning and reward related signaling. METHODS Five subjects who underwent depth electrode placement for seizure localization for medically refractory epilepsy were included in our study. Two behavioral tasks were performed while intracranial local field potentials were recorded from the implanted electrodes. A learning task required subjects to learn an association between a series of presented images and a button press. A gambling task required subjects to place a wager on the outcome of a simulated card game. We computed power in caudate electrodes and compared power during the feedback epoch of the task between correct and incorrect trials for the learning task and between winning and losing trials for the gambling task. RESULTS >There was a significant increase in beta (15-30Hz) power during the feedback epoch of the learning task, with significant differences between beta power following correct versus incorrect responses. Conversely, no difference in beta power was seen during the feedback epoch of the gambling task between winning and losing trials. CONCLUSION Changes in beta power were seen in the caudate nucleus that differed between correct and incorrect trials in a learning task. No correlate was seen in a gambling task, suggesting that this signal is related specifically to learning rather than to reward.

Cortical neurons multiplex reward-related signals along with sensory and motor information

Rewards are known to influence neural activity associated with both motor preparation and execution. This influence can be exerted directly upon the primary motor (M1) and somatosensory (S1) cortical areas via the projections from reward-sensitive dopaminergic neurons of the midbrain ventral tegmental areas. However, the neurophysiological manifestation of reward-related signals in M1 and S1 are not well understood. Particularly, it is unclear how the neurons in these cortical areas multiplex their traditional functions related to the control of spatial and temporal characteristics of movements with the representation of rewards. To clarify this issue, we trained rhesus monkeys to perform a center-out task in which arm movement direction, reward timing, and magnitude were manipulated independently. Activity of several hundred cortical neurons was simultaneously recorded using chronically implanted microelectrode arrays. Many neurons (9-27%) in both M1 and S1 exhibited activity related to reward anticipation. Additionally, neurons in these areas responded to a mismatch between the reward amount given to the monkeys and the amount they expected: A lower-than-expected reward caused a transient increase in firing rate in 60-80% of the total neuronal sample, whereas a larger-than-expected reward resulted in a decreased firing rate in 20-35% of the neurons. Moreover, responses of M1 and S1 neurons to reward omission depended on the direction of movements that led to those rewards. These observations suggest that sensorimotor cortical neurons corepresent rewards and movement-related activity, presumably to enable reward-based learning.

Influence of behavioral state on auditory processing in the marmoset midbrain

Recent work shows that neurons in the inferior colliculus (IC) undergo receptive field changes during auditory behavior in ferrets (Slee & David, 2015. J. Neurosci.). In this study, we tested for similar effects in the marmoset monkey, a highly vocal primate species. We trained two marmosets to detect a pure tone target embedded in a background of random spectral shape (RSS) distractor stimuli. We recorded single-unit activity in the central nucleus of the IC (ICC). Neural responses to targets and distractors were compared between conditions when the marmoset performed the detection task or listened passively. When target frequency was near the neuronal best frequency (BF), responses to distractors were suppressed in about half of the neurons during behavior. Target responses were modulated in some neurons but suppression or enhancement was equally likely. We also measured effects of task engagement in non-central divisions of the IC (NCIC). There, target responses were strongly enhanced during behavior in about half of neurons in this region (median change = 70%). We also found a subset of NCIC neurons that responded with increased firing following the target sound, possibly encoding a reward-related signal. This study replicates our previous finding that distractor responses are suppressed during auditory behavior in the ferret IC. In addition, we found an area in the NCIC with large enhancement of target responses during behavior, suggesting that task engagement produces distinct effects across subdivisions of the IC.

Reward-based contextual learning supported by anterior cingulate cortex.

The anterior cingulate cortex (ACC) is commonly associated with cognitive control and decision making, but its specific function is highly debated. To explore a recent theory that the ACC learns the reward values of task contexts (Holroyd & McClure in Psychological Review, 122, 54-83, 2015; Holroyd & Yeung in Trends in Cognitive Sciences, 16, 122-128, 2012), we recorded the event-related brain potentials (ERPs) from participants as they played a novel gambling task. The participants were first required to select from among three games in one "virtual casino," and subsequently they were required to select from among three different games in a different virtual casino; unbeknownst to them, the payoffs for the games were higher in one casino than in the other. Analysis of the reward positivity, an ERP component believed to reflect reward-related signals carried to the ACC by the midbrain dopamine system, revealed that the ACC is sensitive to differences in the reward values associated with both the casinos and the games inside the casinos, indicating that participants learned the values of the contexts in which rewards were delivered. These results highlight the importance of the ACC in learning the reward values of task contexts in order to guide action selection.

Reversing the Atypical Valuation of Drug and Nondrug Rewards in Smokers Using Multimodal Neuroimaging

BackgroundChronic substance use can disrupt the reward function of the anterior cingulate cortex (ACC), biasing the ACC to favor goal-directed behaviors that converge on drug use. Here we used multimodal neuroimaging methods to ask whether modulating reward-related signaling in the ACC can reverse the atypical valuation of nondrug and drug rewards in abstinent smokers. MethodsWe first recorded functional magnetic resonance imaging data from 20 moderately dependent cigarette smokers (mean age = 25 years; no history of neuropsychiatric disorders), following an overnight period of abstinence, to identify regions of the left dorsal lateral prefrontal cortex associated with the anticipation of drug-related rewards (cigarette puff). Next, we recorded the reward positivity—an electrophysiological signal believed to index sensitivity of the ACC to rewards—while participants engaged in two feedback tasks to gain either monetary or cigarette rewards. Lastly, guided by functional magnetic resonance imaging data, a robotic arm positioned a repetitive transcranial magnetic stimulation coil over a subject-specific dorsal lateral prefrontal cortex target, and 50 repetitive transcranial magnetic stimulation pulses were delivered at 10 Hz (excitatory stimulation) immediately before each block of 10 trials of the money condition and at 1 Hz (inhibitory stimulation) before each block of 10 trials of the cigarette condition. ResultsOur findings show that abstained smokers exhibited a heightened reward positivity to cigarette rewards relative to monetary rewards, and by applying excitatory or inhibitory repetitive transcranial magnetic stimulation to a subject-specific frontal-cingulate reward pathway, this pattern of results was reversed. ConclusionsBy modulating how the brain links value to drug and nondrug rewards, novel brain-based treatments may finally be on the horizon.

Cortical neurons multiplex reward-related signals along with sensory and motor information

Cortical neurons multiplex reward-related signals along with sensory and motor information

Multiple Dopamine Signals and Their Contributions to Reinforcement Learning

Midbrain dopamine neurons are activated by reward and sensory cue that predicts reward. Their responses resemble reward prediction error that indicates the discrepancy between obtained and expected reward values, which has been thought to play an important role as a teaching signal in reinforcement learning. Indeed, pharmacological blockade of dopamine transmission interferes with reinforcement learning. Recent studies reported, however, that not all dopamine neurons transmit the reward-related signal. They found that a subset of dopamine neurons transmits signals related to non-rewarding, salient experiences such as aversive stimulations and cognitively demanding events. How these signals contribute to animal behavior is not yet well understood. This article reviews recent findings on dopamine signals related to rewarding and non-rewarding experiences, and discusses their contributions to reinforcement learning.

Intrinsic monitoring of learning success facilitates memory encoding via the activation of the SN/VTA-Hippocampal loop.

Humans constantly learn in the absence of explicit rewards. However, the neurobiological mechanisms supporting this type of internally-guided learning (without explicit feedback) are still unclear. Here, participants who completed a task in which no external reward/feedback was provided, exhibited enhanced fMRI-signals within the dopaminergic midbrain, hippocampus, and ventral striatum (the SN/VTA-Hippocampal loop) when successfully grasping the meaning of new-words. Importantly, new-words that were better remembered showed increased activation and enhanced functional connectivity between the midbrain, hippocampus, and ventral striatum. Moreover, enhanced emotion-related physiological measures and subjective pleasantness ratings during encoding were associated with remembered new-words after 24 hr. Furthermore, increased subjective pleasantness ratings were also related to new-words remembered after seven days. These results suggest that intrinsic-potentially reward-related-signals, triggered by self-monitoring of correct performance, can promote the storage of new information into long-term memory through the activation of the SN/VTA-Hippocampal loop, possibly via dopaminergic modulation of the midbrain.

A preliminary study of longitudinal neuroadaptation associated with recovery from addiction

BackgroundFew studies have explored longitudinal change in event-related brain responses during early recovery from addiction. Moreover, existing findings yield evidence of both increased and decreased signaling within reward and control centers over time. The current study explored reward- and control-related signals in a risky decision-making task and specifically investigated parametric modulations of the BOLD signal, rather than signal magnitude alone. It was hypothesized that risk-related signals during decision-making and outcome evaluation would reflect recovery and that change in specific signals would correspond with improved treatment outcomes. MethodsTwenty-one substance dependent individuals were recruited upon enrollment in community-based substance use treatment programs, wherein they received treatment-as-usual. Participants completed functional neuroimaging assessments at baseline and 3-month follow-up while performing the Balloon Analogue Risk Task (BART). Risk- and reward-sensitive signals were identified using parametric modulators. Substance use was tracked throughout the 3-month study interval using the timeline follow-back procedure. ResultsLongitudinal contrasts of parametric modulators suggested improved formation of risk-informed outcome expectations at follow-up. Specifically, a greater response to high risk (low-likelihood) positive feedback was identified in caudal anterior cingulate cortex (ACC) and a greater response to low risk (low-likelihood) negative feedback was identified in caudal ACC and inferior frontal gyrus. In addition, attenuation of a ventromedial prefrontal cortex (vmPFC) “reward-seeking” signal (i.e., increasing response with greater reward) during risky decisions at follow-up was associated with less substance use during the study interval. ConclusionsChanges in risk- and reward-related signaling in ACC/vmPFC appear to reflect recovery and may support sobriety.

It\u2019s not what you expect: feedback negativity is independent of reward expectation and affective responsivity in a non-probabilistic task

ERP studies commonly utilize gambling-based reinforcement tasks to elicit feedback negativity (FN) responses. This study used a pattern learning task in order to limit gambling-related fallacious reasoning and possible affective responses to gambling, while investigating relationships between the FN components between high and low reward expectation conditions. Eighteen undergraduates completed measures of reinforcement sensitivity, trait and state affect, and psychophysiological recording. The pattern learning task elicited a FN component for both high and low win expectancy conditions, which was found to be independent of reward expectation and showed little relationship with task and personality variables. We also observed a P3 component, which showed sensitivity to outcome expectancy variation and relationships to measures of anxiety, appetitive motivation, and cortical asymmetry, although these varied by electrode location and expectancy condition. Findings suggest that the FN reflected a binary reward-related signal, with little relationship to reward expectation found in previous studies, in the absence of positive affective responses.

Retroactive modulation of spike timing-dependent plasticity by dopamine.

Most reinforcement learning models assume that the reward signal arrives after the activity that led to the reward, placing constraints on the possible underlying cellular mechanisms. Here we show that dopamine, a positive reinforcement signal, can retroactively convert hippocampal timing-dependent synaptic depression into potentiation. This effect requires functional NMDA receptors and is mediated in part through the activation of the cAMP/PKA cascade. Collectively, our results support the idea that reward-related signaling can act on a pre-established synaptic eligibility trace, thereby associating specific experiences with behaviorally distant, rewarding outcomes. This finding identifies a biologically plausible mechanism for solving the 'distal reward problem'.

Sensing Positive versus Negative Reward Signals through Adenylyl Cyclase-Coupled GPCRs in Direct and Indirect Pathway Striatal Medium Spiny Neurons.

Dopamine transients are considered to carry reward-related signal in reinforcement learning. An increase in dopamine concentration is associated with an unexpected reward or salient stimuli, whereas a decrease is produced by omission of an expected reward. Often dopamine transients are accompanied by other neuromodulatory signals, such as acetylcholine and adenosine. We highlight the importance of interaction between acetylcholine, dopamine, and adenosine signals via adenylyl-cyclase coupled GPCRs in shaping the dopamine-dependent cAMP/PKA signaling in striatal neurons. Specifically, a dopamine peak and an acetylcholine dip must interact, via D1 and M4 receptor, and a dopamine dip must interact with adenosine tone, via D2 and A2a receptor, in direct and indirect pathway neurons, respectively, to have any significant downstream PKA activation.

Orbitofrontal cortex mediates the differential impact of signaled-reward probability on discrimination accuracy.

Orbitofrontal cortex (OFC) function is critical to decision making and behavior based on the value of expected outcomes. While some of the roles the OFC plays in value computations and behavior have been identified, the role of the OFC in modulating cognitive resources based on reward expectancy has not been explored. Here we assessed the involvement of OFC in the interaction between motivation and attention. We tested mice in a sustained-attention task in which explicitly signaling the probability of reward differentially modulates discrimination accuracy. Using pharmacogenetic methods, we generated mice in which neuronal activity in the OFC could be transiently and reversibly inhibited during performance of our signaled-probability task. We found that inhibiting OFC neuronal activity abolished the ability of reward-associated cues to differentially impact accuracy of sustained-attention performance. This failure to modulate attention occurred despite evidence that mice still processed the differential value of the reward-associated cues. These data indicate that OFC function is critical for the ability of a reward-related signal to impact other cognitive and decision-making processes and begin to delineate the neural circuitry involved in the interaction between motivation and attention.

No unified reward prediction error in local field potentials from the human nucleus accumbens: evidence from epilepsy patients.

Functional magnetic resonance imaging (fMRI), cyclic voltammetry, and single-unit electrophysiology studies suggest that signals measured in the nucleus accumbens (Nacc) during value-based decision making represent reward prediction errors (RPEs), the difference between actual and predicted rewards. Here, we studied the precise temporal and spectral pattern of reward-related signals in the human Nacc. We recorded local field potentials (LFPs) from the Nacc of six epilepsy patients during an economic decision-making task. On each trial, patients decided whether to accept or reject a gamble with equal probabilities of a monetary gain or loss. The behavior of four patients was consistent with choices being guided by value expectations. Expected value signals before outcome onset were observed in three of those patients, at varying latencies and with nonoverlapping spectral patterns. Signals after outcome onset were correlated with RPE regressors in all subjects. However, further analysis revealed that these signals were better explained as outcome valence rather than RPE signals, with gamble gains and losses differing in the power of beta oscillations and in evoked response amplitudes. Taken together, our results do not support the idea that postsynaptic potentials in the Nacc represent a RPE that unifies outcome magnitude and prior value expectation. We discuss the generalizability of our findings to healthy individuals and the relation of our results to measurements of RPE signals obtained from the Nacc with other methods.

Reward Acts on the pFC to Enhance Distractor Resistance of Working Memory Representations

Working memory and reward processing are often thought to be separate, unrelated processes. However, most daily activities involve integrating these two types of information, and the two processes rarely, if ever, occur in isolation. Here, we show that working memory and reward interact in a task-dependent manner and that this task-dependent interaction involves modulation of the pFC by the ventral striatum. Specifically, BOLD signal during gains relative to losses in the ventral striatum and pFC was associated not only with enhanced distractor resistance but also with impairment in the ability to update working memory representations. Furthermore, the effect of reward on working memory was accompanied by differential coupling between the ventral striatum and ignore-related regions in the pFC. Together, these data demonstrate that reward-related signals modulate the balance between cognitive stability and cognitive flexibility by altering functional coupling between the ventral striatum and the pFC.

A functional difference in information processing between orbitofrontal cortex and ventral striatum during decision-making behaviour

Both orbitofrontal cortex (OFC) and ventral striatum (vStr) have been identified as key structures that represent information about value in decision-making tasks. However, the dynamics of how this information is processed are not yet understood. We recorded ensembles of cells from OFC and vStr in rats engaged in the spatial adjusting delay-discounting task, a decision-making task that involves a trade-off between delay to and magnitude of reward. Ventral striatal neural activity signalled information about reward before the rat's decision, whereas such reward-related signals were absent in OFC until after the animal had committed to its decision. These data support models in which vStr is directly involved in action selection, but OFC processes decision-related information afterwards that can be used to compare the predicted and actual consequences of behaviour.

Pedunculopontine tegmental nucleus neurons provide reward, sensorimotor, and alerting signals to midbrain dopamine neurons

Dopamine (DA) neurons in the midbrain are crucial for motivational control of behavior. However, recent studies suggest that signals transmitted by DA neurons are heterogeneous. This may reflect a wide range of inputs to DA neurons, but which signals are provided by which brain areas is still unclear. Here we focused on the pedunculopontine tegmental nucleus (PPTg) in macaque monkeys and characterized its inputs to DA neurons. Since the PPTg projects to many brain areas, it is crucial to identify PPTg neurons that project to DA neuron areas. For this purpose we used antidromic activation technique by electrically stimulating three locations (medial, central, lateral) in the substantia nigra pars compacta (SNc). We found SNc-projecting neurons mainly in the PPTg, and some in the cuneiform nucleus. Electrical stimulation in the SNc-projecting PPTg regions induced a burst of spikes in presumed DA neurons, suggesting that the PPTg–DA (SNc) connection is excitatory. Behavioral tasks and clinical tests showed that the SNc-projecting PPTg neurons encoded reward, sensorimotor and arousal/alerting signals. Importantly, reward-related PPTg neurons tended to project to the medial and central SNc, whereas sensorimotor/arousal/alerting-related PPTg neurons tended to project to the lateral SNc. Most reward-related signals were positively biased: excitation and inhibition when a better and worse reward was expected, respectively. These PPTg neurons tended to retain the reward value signal until after a reward outcome, representing ‘value state’; this was different from DA neurons which show phasic signals representing ‘value change’. Our data, together with previous studies, suggest that PPTg neurons send positive reward-related signals mainly to the medial–central SNc where DA neurons encode motivational values, and sensorimotor/arousal signals to the lateral SNc where DA neurons encode motivational salience.

Modulation of neural activity by reward in medial intraparietal cortex is sensitive to temporal sequence of reward.

To restore movements to paralyzed patients, neural prosthetic systems must accurately decode patients' intentions from neural signals. Despite significant advancements, current systems are unable to restore complex movements. Decoding reward-related signals from the medial intraparietal area (MIP) could enhance prosthetic performance. However, the dynamics of reward sensitivity in MIP is not known. Furthermore, reward-related modulation in premotor areas has been attributed to behavioral confounds. Here we investigated the stability of reward encoding in MIP by assessing the effect of reward history on reward sensitivity. We recorded from neurons in MIP while monkeys performed a delayed-reach task under two reward schedules. In the variable schedule, an equal number of small- and large-rewards trials were randomly interleaved. In the constant schedule, one reward size was delivered for a block of trials. The memory period firing rate of most neurons in response to identical rewards varied according to schedule. Using systems identification tools, we attributed the schedule sensitivity to the dependence of neural activity on the history of reward. We did not find schedule-dependent behavioral changes, suggesting that reward modulates neural activity in MIP. Neural discrimination between rewards was less in the variable than in the constant schedule, degrading our ability to decode reach target and reward simultaneously. The effect of schedule was mitigated by adding Haar wavelet coefficients to the decoding model. This raises the possibility of multiple encoding schemes at different timescales and reinforces the potential utility of reward information for prosthetic performance.

Diminished Role for Dopamine D1 Receptors in Cocaine Addiction?

Dopamine D1-like receptors are directly related to the generation of reward-related signals in the brain. In drug naive animals, selective stimulation of D1 receptors is sufficient to support acquisition and maintenance of self-administration behavior (1). Blockade of D1 receptors strongly attenuates the rewarding effects of psychostimulants drugs like cocaine and amphetamine, and D1 receptor knockout mice fail to acquire cocaine self-administration (1). D1 receptors are positively coupled to stimulatory G proteins, adenylate cyclase, and can facilitate excitatory input to D1-expressing neurons in striatal reward regions. Hence, drug naive rodents will self-administer direct optogenetic excitation of D1-containing striatal neurons, essentially bypassing D1 receptor effects (2). In contrast to D1 receptors, selective stimulation of D2-like receptors fails to engender primary rewarding effects in drug naive rodents, but can facilitate the conditioned rewarding effects of environmental stimuli associated with prior drug self-administration (1, 3).

Attentional modulation by reward and punishment cues in relation to depressive symptoms

Background and objectivesResearch indicates that individuals at-risk for depression are characterized by high sensitivity to loss and reduced sensitivity to reward. Moreover, it has been shown that attentional bias plays an important role in depression vulnerability. The current study aimed to examine the interplay between these risk factors for depression by examining the development of attentional bias toward reward and loss signals in dysphoric participants (individuals with elevated levels of depressive symptoms). MethodsShapes were conditioned to reward and loss and subsequently presented in a dot probe task in a sample of dysphoric and nondysphoric participants. ResultsNondysphoric individuals oriented towards reward-related signals whereas dysphoric individuals failed to develop a reward-related attentional bias. This attentional effect was observed in the absence of group differences in motivational factors. No group differences were found for attentional bias for loss-related signals, despite the fact that dysphoric individuals performed worse in response to losing. LimitationsThe current sample is not clinical thus generalization to clinical depression is not warranted. ConclusionsWe argue that impaired early attentional processing of rewards are an important cognitive risk factor for anhedonic symptoms in persons with dysphoria.