Reward Prediction Error Signal Research Articles

Mild exogenous inflammation blunts neural signatures of bounded evidence accumulation and reward prediction error processing in healthy male participants

BackgroundAltered neural haemodynamic activity during decision making and learning has been linked to the effects of inflammation on mood and motivated behaviours. So far, it has been reported that blunted mesolimbic dopamine reward signals are associated with inflammation-induced anhedonia and apathy. Nonetheless, it is still unclear whether inflammation impacts neural activity underpinning decision dynamics. The process of decision making involves integration of noisy evidence from the environment until a critical threshold of evidence is reached. There is growing empirical evidence that such process, which is usually referred to as bounded accumulation of decision evidence, is affected in the context of mental illness. MethodsIn a randomised, placebo-controlled, crossover study, 19 healthy male participants were allocated to placebo and typhoid vaccination. Three to four hours post-injection, participants performed a probabilistic reversal-learning task during functional magnetic resonance imaging. To capture the hidden neurocognitive operations underpinning decision-making, we devised a hybrid sequential sampling and reinforcement learning computational model. We conducted whole brain analyses informed by the modelling results to investigate the effects of inflammation on the efficiency of decision dynamics and reward learning. ResultsWe found that during the decision phase of the task, typhoid vaccination attenuated neural signatures of bounded evidence accumulation in the dorsomedial prefrontal cortex, only for decisions requiring short integration time. Consistent with prior work, we showed that, in the outcome phase, mild acute inflammation blunted the reward prediction error in the bilateral ventral striatum and amygdala. ConclusionsOur study extends current insights into the effects of inflammation on the neural mechanisms of decision making and shows that exogenous inflammation alters neural activity indexing efficiency of evidence integration, as a function of choice discriminability. Moreover, we replicate previous findings that inflammation blunts striatal reward prediction error signals.

Glutamate inputs send prediction error of reward, but not negative value of aversive stimuli, to dopamine neurons

Midbrain dopamine neurons are thought to signal reward prediction errors (RPEs), but the mechanisms underlying RPE computation, particularly the contributions of different neurotransmitters, remain poorly understood. Here, we used a genetically encoded glutamate sensor to examine the pattern of glutamate inputs to dopamine neurons in mice. We found that glutamate inputs exhibit virtually all of the characteristics of RPE rather than conveying a specific component of RPE computation, such as reward or expectation. Notably, whereas glutamate inputs were transiently inhibited by reward omission, they were excited by aversive stimuli. Opioid analgesics altered dopamine negative responses to aversive stimuli into more positive responses, whereas excitatory responses of glutamate inputs remained unchanged. Our findings uncover previously unknown synaptic mechanisms underlying RPE computations; dopamine responses are shaped by both synergistic and competitive interactions between glutamatergic and GABAergic inputs to dopamine neurons depending on valences, with competitive interactions playing a role in responses to aversive stimuli.

Dopamine-independent effect of rewards on choices through hidden-state inference

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world’s state, not the dopaminergic RPEs they produce.

Striatal response to negative feedback in a stop signal task operates as a multi-value learning signal.

We examined error-driven learning in fMRI activity of 217 subjects in a stop signal task to obtain a more robust characterization of the relation between behavioral measures of learning and corresponding neural learning signals than previously possible. The stop signal task is a two-alternative forced choice in which participants respond to an arrow by pressing a left or right button but must inhibit that response on 1 in 7 trials when cued by an auditory "stop signal." We examined post-error learning by comparing brain activity (BOLD signal) and behavioral responses on trials preceded by successful (correct stop) vs. failed (failed stop) inhibition. There was strong evidence of greater bilateral striatal activity in the period immediately following correct (vs. failed) stop trials (most evident in the putamen; peak MNI coordinates [-26 8 -2], 430 voxels, p < 0.001; [24 14 0], 527 voxels, p < 0.001). We measured median activity in the bilateral striatal cluster following every failed stop and correct stop trial and correlated it with learning signals for (a) probability and (b) latency of the stop signal. In a mixed-effects model predicting activity 5-10 s after the stop signal, both reaction time (RT) change (B = -0.05, t = 3.0, χ2 = 11.3, p < 0.001) and probability of stop trial change (B = 1.53, t = 6.0, χ2 = 43.0, p < 0.001) had significant within-subjects effects on median activity. In a similar mixed model predicting activity 1-5 s after the stop signal, only probability of stop trial change was predictive. A mixed-effects model indicates the striatal activity might be a learning signal that encodes reaction time change and the current expected probability of a stop trial occuring. This extends existing evidence that the striatum encodes a reward prediction error signal for learning within the stop signal task, and demonstrates for the first time that this signal seems to encode both change in stop signal probability and in stop signal delay.

Striatal Dopamine Signals and Reward Learning.

We are constantly bombarded by sensory information and constantly making decisions on how to act. In order to optimally adapt behavior, we must judge which sequences of sensory inputs and actions lead to successful outcomes in specific circumstances. Neuronal circuits of the basal ganglia have been strongly implicated in action selection, as well as the learning and execution of goal-directed behaviors, with accumulating evidence supporting the hypothesis that midbrain dopamine neurons might encode a reward signal useful for learning. Here, we review evidence suggesting that midbrain dopaminergic neurons signal reward prediction error, driving synaptic plasticity in the striatum underlying learning. We focus on phasic increases in action potential firing of midbrain dopamine neurons in response to unexpected rewards. These dopamine neurons prominently innervate the dorsal and ventral striatum. In the striatum, the released dopamine binds to dopamine receptors, where it regulates the plasticity of glutamatergic synapses. The increase of striatal dopamine accompanying an unexpected reward activates dopamine type 1 receptors (D1Rs) initiating a signaling cascade that promotes long-term potentiation of recently active glutamatergic input onto striatonigral neurons. Sensorimotor-evoked glutamatergic input, which is active immediately before reward delivery will thus be strengthened onto neurons in the striatum expressing D1Rs. In turn, these neurons cause disinhibition of brainstem motor centers and disinhibition of the motor thalamus, thus promoting motor output to reinforce rewarded stimulus-action outcomes. Although many details of the hypothesis need further investigation, altogether, it seems likely that dopamine signals in the striatum might underlie important aspects of goal-directed reward-based learning.

Reward prediction error in learning-related behaviors.

Learning is a complex process, during which our opinions and decisions are easily changed due to unexpected information. But the neural mechanism underlying revision and correction during the learning process remains unclear. For decades, prediction error has been regarded as the core of changes to perception in learning, even driving the learning progress. In this article, we reviewed the concept of reward prediction error, and the encoding mechanism of dopaminergic neurons and the related neural circuities. We also discussed the relationship between reward prediction error and learning-related behaviors, including reversal learning. We then demonstrated the evidence of reward prediction error signals in several neurological diseases, including Parkinson's disease and addiction. These observations may help to better understand the regulatory mechanism of reward prediction error in learning-related behaviors.

Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, "three-factor" corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.

The behavioral signature of stepwise learning strategy in male rats and its neural correlate in the basal forebrain

Studies of associative learning have commonly focused on how rewarding outcomes are predicted by either sensory stimuli or animals’ actions. However, in many learning scenarios, reward delivery requires the occurrence of both sensory stimuli and animals’ actions in a specific order, in the form of behavioral sequences. How such behavioral sequences are learned is much less understood. Here we provide behavioral and neurophysiological evidence to show that behavioral sequences are learned using a stepwise strategy. In male rats learning a new association, learning started from the behavioral event closest to the reward and sequentially incorporated earlier events. This led to the sequential refinement of reward-seeking behaviors, which was characterized by the stepwise elimination of ineffective and non-rewarded behavioral sequences. At the neuronal level, this stepwise learning process was mirrored by the sequential emergence of basal forebrain neuronal responses toward each event, which quantitatively conveyed a reward prediction error signal and promoted reward-seeking behaviors. Together, these behavioral and neural signatures revealed how behavioral sequences were learned in discrete steps and when each learning step took place.

Neural dissociation between reward and salience prediction errors through the lens of optimistic bias.

The question of how the brain represents reward prediction errors is central to reinforcement learning and adaptive, goal-directed behavior. Previous studies have revealed prediction error representations in multiple electrophysiological signatures, but it remains elusive whether these electrophysiological correlates underlying prediction errors are sensitive to valence (in a signed form) or to salience (in an unsigned form). One possible reason concerns the loose correspondence between objective probability and subjective prediction resulting from the optimistic bias, that is, the tendency to overestimate the likelihood of encountering positive future events. In the present electroencephalography (EEG) study, we approached this question by directly measuring participants' idiosyncratic, trial-to-trial prediction errors elicited by subjective and objective probabilities across two experiments. We adopted monetary gain and loss feedback in Experiment 1 and positive and negative feedback as communicated by the same zero-value feedback in Experiment 2. We provided electrophysiological evidence in time and time-frequency domains supporting both reward and salience prediction error signals. Moreover, we showed that these electrophysiological signatures were highly flexible and sensitive to an optimistic bias and various forms of salience. Our findings shed new light on multiple presentations of prediction error in the human brain, which differ in format and functional role.

Involvement of cortical input to the rostromedial tegmental nucleus in aversion to foot shock.

The rostromedial tegmental nucleus (RMTg) encodes negative reward prediction error (RPE) and plays an important role in guiding behavioral responding to aversive stimuli. Previous research has focused on regulation of RMTg activity by the lateral habenula despite studies revealing RMTg afferents from other regions including the frontal cortex. The current study provides a detailed anatomical and functional analysis of cortical input to the RMTg of male rats. Retrograde tracing uncovered dense cortical input to the RMTg spanning the medial prefrontal cortex, the orbitofrontal cortex and anterior insular cortex. Afferents were most dense in the dorsomedial subregion of the PFC (dmPFC), an area that is also implicated in both RPE signaling and aversive responding. RMTg-projecting dmPFC neurons originate in layer V, are glutamatergic, and collateralize to select brain regions. In-situ mRNA hybridization revealed that neurons in this circuit are predominantly D1 receptor-expressing with a high degree of D2 receptor colocalization. Consistent with cFos induction in this neural circuit during exposure to foot shock and shock-predictive cues, optogenetic stimulation of dmPFC terminals in the RMTg drove avoidance. Lastly, acute slice electrophysiology and morphological studies revealed that exposure to repeated foot shock resulted in significant physiological and structural changes consistent with a loss of top-down modulation of RMTg-mediated signaling. Altogether, these data reveal the presence of a prominent cortico-subcortical projection involved in adaptive behavioral responding to aversive stimuli such as foot shock and provide a foundation for future work aimed at exploring alterations in circuit function in diseases characterized by deficits in cognitive control over reward and aversion.

Impulse control disorder in Parkinson's disease is associated with abnormal frontal value signalling.

Dopaminergic medication is well established to boost reward- versus punishment-based learning in Parkinson's disease. However, there is tremendous variability in dopaminergic medication effects across different individuals, with some patients exhibiting much greater cognitive sensitivity to medication than others. We aimed to unravel the mechanisms underlying this individual variability in a large heterogeneous sample of early-stage patients with Parkinson's disease as a function of comorbid neuropsychiatric symptomatology, in particular impulse control disorders and depression. One hundred and ninety-nine patients with Parkinson's disease (138 ON medication and 61 OFF medication) and 59 healthy controls were scanned with functional MRI while they performed an established probabilistic instrumental learning task. Reinforcement learning model-based analyses revealed medication group differences in learning from gains versus losses, but only in patients with impulse control disorders. Furthermore, expected-value related brain signalling in the ventromedial prefrontal cortex was increased in patients with impulse control disorders ON medication compared with those OFF medication, while striatal reward prediction error signalling remained unaltered. These data substantiate the hypothesis that dopamine's effects on reinforcement learning in Parkinson's disease vary with individual differences in comorbid impulse control disorder and suggest they reflect deficient computation of value in medial frontal cortex, rather than deficient reward prediction error signalling in striatum. See Michael Browning (https://doi.org/10.1093/brain/awad248) for a scientific commentary on this article.

Anticipation of Appetitive Operant Action Induces Sustained Dopamine Release in the Nucleus Accumbens.

The mesolimbic dopamine system is implicated in signaling reward-related information as well as in actions that generate rewarding outcomes. These implications are commonly investigated in either pavlovian or operant reinforcement paradigms, where only the latter requires instrumental action. To parse contributions of reward- and action-related information to dopamine signals, we directly compared the two paradigms: male rats underwent either pavlovian or operant conditioning while dopamine release was measured in the nucleus accumbens, a brain region central for processing this information. Task conditions were identical with the exception of the operant-lever response requirement. Rats in both groups released the same quantity of dopamine at the onset of the reward-predictive cue. However, only the operant-conditioning group showed a subsequent, sustained plateau in dopamine concentration throughout the entire 5 s cue presentation (preceding the required action). This dopamine ramp was unaffected by probabilistic reward delivery, occurred exclusively before operant actions, and was not related to task performance or task acquisition as it persisted throughout the 2 week daily behavioral training. Instead, the ramp flexibly increased in duration with longer cue presentation, seemingly modulating the initial cue-onset-triggered dopamine release, that is, the reward prediction error (RPE) signal, as both signal amplitude and sustainment diminished when reward timing was made more predictable. Thus, our findings suggest that RPE and action components of dopamine release can be differentiated temporally into phasic and ramping/sustained signals, respectively, where the latter depends on the former and presumably reflects the anticipation or incentivization of appetitive action, conceptually akin to motivation.SIGNIFICANCE STATEMENT It is unclear whether the components of dopamine signals that are related to reward-associated information and reward-driven approach behavior can be separated. Most studies investigating the dopamine system use either pavlovian or operant conditioning, which both involve the delivery of reward and necessitate appetitive approach behavior. Thus, used exclusively, neither paradigm can disentangle the contributions of these components to dopamine release. However, by combining both paradigms in the same study, we find that anticipation of a reward-driven operant action induces a modulation of reward-prediction-associated dopamine release, producing so-called dopamine ramps. Therefore, our findings provide new insight into dopamine ramps and suggest that dopamine signals integrate reward and appetitive action in a temporally distinguishable, yet dependent, manner.

Beta oscillations in monkey striatum encode reward prediction error signals.

Reward prediction error (RPE) signals are crucial for reinforcement learning and decision making as they quantify the mismatch between predicted and obtained rewards. RPE signals are encoded in the neural activity of multiple brain areas, such as midbrain dopaminergic neurons, prefrontal cortex and striatum. However, it remains unclear how these signals are expressed through anatomically and functionally distinct subregions of the striatum. In the current study, we examined to which extent RPE signals are represented across different striatal regions. To do so, we recorded local field potentials (LFPs) in sensorimotor, associative, and limbic striatal territories of two male rhesus monkeys performing a free-choice probabilistic learning task. The trial-by-trial evolution of RPE during task performance was estimated using a reinforcement learning model fitted on monkeys' choice behavior. Overall, we found that changes in beta-band oscillations (15-35 Hz), after the outcome of the animal's choice, are consistent with RPE encoding. Moreover, we provide evidence that the signals related to RPE are more strongly represented in the ventral (limbic) than dorsal (sensorimotor and associative) part of the striatum. To conclude, our results suggest a relationship between striatal beta oscillations and the evaluation of outcomes based on RPE signals and highlight a major contribution of the ventral striatum to the updating of learning processes.Significance StatementReward prediction error (RPE) signals are crucial for reinforcement learning and decision making as they quantify the mismatch between predicted and obtained rewards. Current models suggest that RPE signals are encoded in the neural activity of multiple brain areas, including the midbrain dopaminergic neurons, prefrontal cortex and striatum. However, it remains elusive whether RPEs recruit anatomically and functionally distinct subregions of the striatum. Our study provides evidence that RPE-related modulations in LFP power are dominant in the striatum. In particular, they are stronger in the rostro-ventral rather than the caudo-dorsal striatum. Our findings contribute to a better understanding of the role of striatal territories in reward-based learning and may be relevant for neuropsychiatric and neurological diseases that affect striatal circuits.

Adaptive coding of pain prediction error in the anterior insula.

Understanding the mechanisms behind the influence of expectation and context on pain perception is crucial for improving analgesic treatments. Prediction error (PE) signals how much a noxious stimulus deviates from expectation and is therefore crucial for our understanding of pain perception. It is thought that the brain engages in 'adaptive coding' of pain PE, such that sensitivity to unexpected outcomes is modulated by contextual information. While there is behavioural evidence that pain is coded adaptively, and evidence that reward PE signals are coded adaptively, controversy remains regarding the underlying neural mechanism of adaptively-coded pain PEs. A cued-pain task was performed by 19 healthy adults while undergoing FMRI scanning. BOLD responses to the task were tested using an axiomatic approach to identify areas that may code pain PE adaptively. The left dorsal anterior insula demonstrated a pattern of response consistent with adaptively-coded pain PE. Signals from this area were sensitive to both predicted pain magnitudes on the instigation of expectation, and the unexpectedness of pain delivery. Crucially however, the response at pain delivery was consistent with the local context of the pain stimulation, rather than the absolute magnitude of delivered pain, a pattern suggestive of an adaptively-coded PE signal. The study advances our understanding of the neural basis of pain prediction. Alongside existing evidence that the periaqueductal grey codes pain PE and the posterior insula codes pain magnitude, the results highlight a distinct contribution of the left dorsal anterior insula in the processing of pain. Although there is behavioural evidence that pain is coded adaptively, the neural mechanisms serving this process are not well understood. This study used functional MRI to provide the first evidence that the left dorsal anterior insula, an area associated with aversive learning, responds to pain in a manner consistent with the adaptive coding of pain prediction error. This study aids our understanding of the neural basis of subjective pain representation, and thus can contribute to the advancement of analgesic treatments.

Do the negative symptoms of schizophrenia reflect reduced responsiveness to reward? Examination using a reward prediction error (RPE) task.

A leading theory of the negative symptoms of schizophrenia is that they reflect reduced responsiveness to rewarding stimuli. This proposal has been linked to abnormal (reduced) dopamine function in the disorder, because phasic release of dopamine is known to code for reward prediction error (RPE). Nevertheless, few functional imaging studies have examined if patients with negative symptoms show reduced RPE-associated activations. Matched groups of DSM-5 schizophrenia patients with high negative symptom scores (HNS, N = 27) or absent negative symptoms (ANS, N = 27) and healthy controls (HC, N = 30) underwent fMRI scanning while they performed a probabilistic monetary reward task designed to generate a measure of RPE. In the HC, whole-brain analysis revealed that RPE was positively associated with activation in the ventral striatum, the putamen, and areas of the lateral prefrontal cortex and orbitofrontal cortex, among other regions. Group comparison revealed no activation differences between the healthy controls and the ANS patients. However, compared to the ANS patients, the HNS patients showed regions of significantly reduced activation in the left ventrolateral and dorsolateral prefrontal cortex, and in the right lingual and fusiform gyrus. HNS and ANS patients showed no activation differences in ventral striatal or midbrain regions-of-interest (ROIs), but the HNS patients showed reduced activation in a left orbitofrontal cortex ROI. The findings do not suggest that a generalized reduction of RPE signalling underlies negative symptoms. Instead, they point to a more circumscribed dysfunction in the lateral frontal and possibly the orbitofrontal cortex.

Distinct cortico-striatal compartments drive competition between adaptive and automatized behavior.

Cortical and basal ganglia circuits play a crucial role in the formation of goal-directed and habitual behaviors. In this study, we investigate the cortico-striatal circuitry involved in learning and the role of this circuitry in the emergence of inflexible behaviors such as those observed in addiction. Specifically, we develop a computational model of cortico-striatal interactions that performs concurrent goal-directed and habit learning. The model accomplishes this by distinguishing learning processes in the dorsomedial striatum (DMS) that rely on reward prediction error signals as distinct from the dorsolateral striatum (DLS) where learning is supported by salience signals. These striatal subregions each operate on unique cortical input: the DMS receives input from the prefrontal cortex (PFC) which represents outcomes, and the DLS receives input from the premotor cortex which determines action selection. Following an initial learning of a two-alternative forced choice task, we subjected the model to reversal learning, reward devaluation, and learning a punished outcome. Behavior driven by stimulus-response associations in the DLS resisted goal-directed learning of new reward feedback rules despite devaluation or punishment, indicating the expression of habit. We repeated these simulations after the impairment of executive control, which was implemented as poor outcome representation in the PFC. The degraded executive control reduced the efficacy of goal-directed learning, and stimulus-response associations in the DLS were even more resistant to the learning of new reward feedback rules. In summary, this model describes how circuits of the dorsal striatum are dynamically engaged to control behavior and how the impairment of executive control by the PFC enhances inflexible behavior.

Dopamine error signal to actively cope with lack of expected reward

To obtain more of a particular uncertain reward, animals must learn to actively overcome the lack of reward and adjust behavior to obtain it again. The neural mechanisms underlying such coping with reward omission remain unclear. Here, we developed a task in rats to monitor active behavioral switch toward the next reward after no reward. We found that some dopamine neurons in the ventral tegmental area exhibited increased responses to unexpected reward omission and decreased responses to unexpected reward, following the opposite responses of the well-known dopamine neurons that signal reward prediction error (RPE). The dopamine increase reflected in the nucleus accumbens correlated with behavioral adjustment to actively overcome unexpected no reward. We propose that these responses signal error to actively cope with lack of expected reward. The dopamine error signal thus cooperates with the RPE signal, enabling adaptive and robust pursuit of uncertain reward to ultimately obtain more reward.

Blunted Expected Reward Value Signals in Binge Alcohol Drinkers.

Alcohol-related morbidities and mortality are highly prevalent, increasing the burden to societies and health systems with 3 million deaths globally each year in young adults directly attributable to alcohol. Cue-induced alcohol craving has been formulated as a type of aberrant associative learning, modeled using temporal difference theory with an expected reward value (ERV) linked to craving. Clinically, although harmful use of alcohol is associated with increased time spent obtaining and using alcohol, it is also associated with self-neglect. The latter implies that the motivational aspects of nonalcohol stimuli are blunted. Using an instrumental learning task with non-alcohol-related stimuli, here, we tested hypotheses that the encoding of cue signals (ERV) predicting reward delivery would be blunted in binge alcohol drinkers in both sexes. We also predicted that for the binge drinking group alone, ratings of problematic alcohol use would correlate with abnormal ERV signals consistent with between groups (i.e., binge drinkers vs controls) abnormalities. Our results support our hypotheses with the ERV (nonalcohol cue) signal blunted in binge drinkers and with the magnitude of the abnormality correlating with ratings of problematic alcohol use. This implies that consistent with hypotheses, the motivational aspects of non-alcohol-related stimuli are blunted in binge drinkers. A better understanding of the mechanisms of harmful alcohol use will, in time, facilitate the development of more effective interventions, which should aim to decrease the motivational value of alcohol and increase the motivational value of non-alcohol-related stimuli.SIGNIFICANCE STATEMENT Allostasis theory predicts specific abnormalities in brain function and subjective experiences that occur when people develop drug problems including addiction. Cue-induced alcohol craving has been formulated as a type of aberrant associative learning, modeled using temporal difference theory with ERV linked to craving. Here, we used an instrumental learning task with non-alcohol-associated stimuli to test hypotheses that the encoding of nonalcohol cue signals (ERV) and reward prediction error signals showed blunting in binge alcohol drinkers. We conclude that fMRI can be used to noninvasively test allostasis and associative learning theory predictions in binge drinkers.

Large-scale neural network computations and multivariate representations during approach-avoidance conflict decision-making

Many real-world situations require navigating decisions for both reward and threat. While there has been significant progress in understanding mechanisms of decision-making and mediating neurocircuitry separately for reward and threat, there is limited understanding of situations where reward and threat contingencies compete to create approach-avoidance conflict (AAC). Here, we leverage computational learning models, independent component analysis (ICA), and multivariate pattern analysis (MVPA) approaches to understand decision-making during a novel task that embeds concurrent reward and threat learning and manipulates congruency between reward and threat probabilities. Computational modeling supported a modified reinforcement learning model where participants integrated reward and threat value into a combined total value according to an individually varying policy parameter, which was highly predictive of decisions to approach reward vs avoid threat during trials where the highest reward option was also the highest threat option (i.e., approach-avoidance conflict). ICA analyses demonstrated unique roles for salience, frontoparietal, medial prefrontal, and inferior frontal networks in differential encoding of reward vs threat prediction error and value signals. The left frontoparietal network uniquely encoded degree of conflict between reward and threat value at the time of choice. MVPA demonstrated that delivery of reward and threat could accurately be decoded within salience and inferior frontal networks, respectively, and that decisions to approach reward vs avoid threat were predicted by the relative degree to which these reward vs threat representations were active at the time of choice. This latter result suggests that navigating AAC decisions involves generating mental representations for possible decision outcomes, and relative activation of these representations may bias subsequent decision-making towards approaching reward or avoiding threat accordingly.

Dopamine mediates the bidirectional update of interval timing.

The role of dopamine (DA) as a reward prediction error (RPE) signal in reinforcement learning (RL) tasks has been well-established over the past decades. Recent work has shown that the RPE interpretation can also account for the effects of DA on interval timing by controlling the speed of subjective time. According to this theory, the timing of the dopamine signal relative to reward delivery dictates whether subjective time speeds up or slows down: Early DA signals speed up subjective time and late signals slow it down. To test this bidirectional prediction, we reanalyzed measurements of dopaminergic neurons in the substantia nigra pars compacta of mice performing a self-timed movement task. Using the slope of ramping dopamine activity as a readout of subjective time speed, we found that trial-by-trial changes in the slope could be predicted from the timing of dopamine activity on the previous trial. This result provides a key piece of evidence supporting a unified computational theory of RL and interval timing. (PsycInfo Database Record (c) 2022 APA, all rights reserved).