Phasic Activity Of Dopamine Neurons Research Articles

Computational reinforcement learning, reward (and punishment), and dopamine in psychiatric disorders.

In the DSM-5, psychiatric diagnoses are made based on self-reported symptoms and clinician-identified signs. Though helpful in choosing potential interventions based on the available regimens, this conceptualization of psychiatric diseases can limit basic science investigation into their underlying causes. The reward prediction error (RPE) hypothesis of dopamine neuron function posits that phasic dopamine signals encode the difference between the rewards a person expects and experiences. The computational framework from which this hypothesis was derived, temporal difference reinforcement learning (TDRL), is largely focused on reward processing rather than punishment learning. Many psychiatric disorders are characterized by aberrant behaviors, expectations, reward processing, and hypothesized dopaminergic signaling, but also characterized by suffering and the inability to change one's behavior despite negative consequences. In this review, we provide an overview of the RPE theory of phasic dopamine neuron activity and review the gains that have been made through the use of computational reinforcement learning theory as a framework for understanding changes in reward processing. The relative dearth of explicit accounts of punishment learning in computational reinforcement learning theory and its application in neuroscience is highlighted as a significant gap in current computational psychiatric research. Four disorders comprise the main focus of this review: two disorders of traditionally hypothesized hyperdopaminergic function, addiction and schizophrenia, followed by two disorders of traditionally hypothesized hypodopaminergic function, depression and post-traumatic stress disorder (PTSD). Insights gained from a reward processing based reinforcement learning framework about underlying dopaminergic mechanisms and the role of punishment learning (when available) are explored in each disorder. Concluding remarks focus on the future directions required to characterize neuropsychiatric disorders with a hypothesized cause of underlying dopaminergic transmission.

Tamping Ramping: Algorithmic, Implementational, and Computational Explanations of Phasic Dopamine Signals in the Accumbens.

Substantial evidence suggests that the phasic activity of dopamine neurons represents reinforcement learning’s temporal difference prediction error. However, recent reports of ramp-like increases in dopamine concentration in the striatum when animals are about to act, or are about to reach rewards, appear to pose a challenge to established thinking. This is because the implied activity is persistently predictable by preceding stimuli, and so cannot arise as this sort of prediction error. Here, we explore three possible accounts of such ramping signals: (a) the resolution of uncertainty about the timing of action; (b) the direct influence of dopamine over mechanisms associated with making choices; and (c) a new model of discounted vigour. Collectively, these suggest that dopamine ramps may be explained, with only minor disturbance, by standard theoretical ideas, though urgent questions remain regarding their proximal cause. We suggest experimental approaches to disentangling which of the proposed mechanisms are responsible for dopamine ramps.

A Specific Component of the Evoked Potential Mirrors Phasic Dopamine Neuron Activity during Conditioning.

Midbrain dopamine (DA) neurons are thought to be a critical node in the circuitry that mediates reward learning. DA neurons receive diverse inputs from regions distributed throughout the neuraxis from frontal neocortex to the mesencephalon. While a great deal is known about changes in the activity of individual DA neurons during learning, much less is known about the functional changes in the microcircuits in which DA neurons are embedded. Here we used local field potentials recorded from the midbrain of behaving mice to show that the midbrain evoked potential (mEP) faithfully reflects the temporal and spatial structure of the phasic response of midbrain neuron populations during conditioning. By comparing the mEP to simultaneously recorded single units, we identified specific components of the mEP that corresponded to phasic DA and non-DA responses to salient stimuli. The DA component of the mEP emerged with the acquisition of a conditioned stimulus, was extinguished following changes in reinforcement contingency, and could be inhibited by pharmacological manipulations that attenuate the phasic responses of DA neurons. In contrast to single-unit recordings, the mEP permitted relatively dense sampling of the midbrain circuit during conditioning and thus could be used to reveal the spatiotemporal structure of multiple intermingled midbrain circuits. Finally, the mEP response was stable for months and thus provides a new approach to study long-term changes in the organization of ventral midbrain microcircuits during learning. Significance statement: Neurons that synthesize and release the neurotransmitter dopamine play a critical role in voluntary reward-seeking behavior. Much of our insight into the function of dopamine neurons comes from recordings of individual cells in behaving animals; however, it is notoriously difficult to record from dopamine neurons due to their sparsity and depth, as well as the presence of intermingled non-dopaminergic neurons. Here we show that much of the information that can be learned from recordings of individual dopamine and non-dopamine neurons is also revealed by changes in specific components of the local field potential. This technique provides an accessible measurement that could prove critical to our burgeoning understanding of the molecular, functional, and anatomical diversity of neuron populations in the midbrain.

Mindfulness meditation modulates reward prediction errors in a passive conditioning task.

Reinforcement learning models have demonstrated that phasic activity of dopamine neurons during reward expectation encodes information about the predictability of reward and cues that predict reward. Self-control strategies such as those practiced in mindfulness-based approaches is claimed to reduce negative and positive reactions to stimuli suggesting the hypothesis that such training may influence basic reward processing. Using a passive conditioning task and fMRI in a group of experienced mindfulness meditators and age-matched controls, we tested the hypothesis that mindfulness meditation influence reward and reward prediction error (PE) signals. We found diminished positive and negative PE-related blood-oxygen level-dependent (BOLD) responses in the putamen in meditators compared with controls. In the meditator group this decrease in striatal BOLD responses to reward PE was paralleled by increased activity in posterior insula, a primary interoceptive region. Critically, responses in the putamen during early trials of the conditioning procedure (run 1) were elevated in both meditators and controls. Overall, these results provide evidence that experienced mindfulness meditators are able to attenuate reward prediction signals to valenced stimuli, which may be related to interoceptive processes encoded in the posterior insula.

Disruption of Dopamine Neuron Activity Pattern Regulation through Selective Expression of a Human KCNN3 Mutation

The calcium-activated small conductance potassium channel SK3 plays an essential role in the regulation of dopamine neuron activity patterns. Here we demonstrate that expression of a human disease-related SK3 mutation (hSK3Δ) in dopamine neurons of mice disrupts the balance between tonic and phasic dopamine neuron activity. Expression of hSK3Δ suppressed endogenous SK currents, reducing coupling between SK channels and NMDA receptors (NMDARs) and increasing permissiveness for burst firing. Consistent with enhanced excitability of dopamine neurons, hSK3Δ increased evoked calcium signals in dopamine neurons in vivo and potentiated evoked dopamine release. Specific expression of hSK3Δ led to deficits in attention and sensory gating and heightened sensitivity to a psychomimetic drug. Sensory-motor alterations and psychomimetic sensitivity were recapitulated in a mouse model of transient, reversible dopamine neuron activation. These results demonstrate the cell-autonomous effects of a human ion channel mutation on dopamine neuron physiology and the impact of activity pattern disruption on behavior.

Mapping anhedonia onto reinforcement learning: a behavioural meta-analysis

BackgroundDepression is characterised partly by blunted reactions to reward. However, tasks probing this deficiency have not distinguished insensitivity to reward from insensitivity to the prediction errors for reward that determine learning and are putatively reported by the phasic activity of dopamine neurons. We attempted to disentangle these factors with respect to anhedonia in the context of stress, Major Depressive Disorder (MDD), Bipolar Disorder (BPD) and a dopaminergic challenge.MethodsSix behavioural datasets involving 392 experimental sessions were subjected to a model-based, Bayesian meta-analysis. Participants across all six studies performed a probabilistic reward task that used an asymmetric reinforcement schedule to assess reward learning. Healthy controls were tested under baseline conditions, stress or after receiving the dopamine D2 agonist pramipexole. In addition, participants with current or past MDD or BPD were evaluated. Reinforcement learning models isolated the contributions of variation in reward sensitivity and learning rate.ResultsMDD and anhedonia reduced reward sensitivity more than they affected the learning rate, while a low dose of the dopamine D2 agonist pramipexole showed the opposite pattern. Stress led to a pattern consistent with a mixed effect on reward sensitivity and learning rate.ConclusionReward-related learning reflected at least two partially separable contributions. The first related to phasic prediction error signalling, and was preferentially modulated by a low dose of the dopamine agonist pramipexole. The second related directly to reward sensitivity, and was preferentially reduced in MDD and anhedonia. Stress altered both components. Collectively, these findings highlight the contribution of model-based reinforcement learning meta-analysis for dissecting anhedonic behavior.

Dopamine-mediated action discovery promotes optimal behavior ‘for free’

Redgrave and Gurney [1] have recently proposed a theory of action discovery in which the occurrence of an unexpected salient sensory event causes short-latency phasic dopamine (DA) neuron activity. Due to convergent projections from DA neurons and cortical neurons onto the striatum, the DA signal reinforces (i.e., biases the animal to repeat) preceding behaviour. In the case that the animal has caused the sensory outcome, continued repetition, along with variations in behaviour, allows the animal to discover the novel action-outcome contingency. A key aspect of this theory is that the DA signal decreases as the action-outcome association becomes predictable. In contrast is the well-known theory that the DA signal specifically represents a reward prediction error [2], and that the tendency to select behaviour is based on its ‘value’ (often represented as the expected subsequent cumulative costs and rewards). The latter theory is attractive in part because it conforms with Reinforcement Learning (RL), a computational formulation of learning from the consequences of actions [3]. RL describes well behaviour observed during many types of goal-directed tasks and allows behaviour to be described as optimal given a particular reward/cost function (e.g., cost per movement). However, is a notion of optimality necessary to produce all behaviour thought to derive from DA-mediated mechanisms? If so, a representation of every movement’s cost and value is required to govern learning. The theory of action discovery [1] suggests that some behaviour is developed through much simpler mechanisms: the occurrence of the salient event rather than a prediction of some quantity (value) associated with each action selected. Here, we implement such a scheme in a learning agent that chooses discrete actions within a discrete state environment (analogous to pushing buttons). Only when the agent has reached a particular state (the salient event) is a reinforcement signal delivered. The tendency to select every action that was selected en route to that state is increased, but with a learning rate that decreases with temporal distance from the reinforcement signal (i.e., an eligibility trace). Importantly, no notion of optimality is represented. Figure ​Figure11 below illustrates the development of behavior due to our scheme (black line) and an RL scheme based on cost (grey). In both, the number of actions chosen en route to the salient event decreases toward a minimum. Such behavior is usually taken as evidence that it approaches optimal given some cost function on which learning depends. However, with our scheme, such behavior emerges ‘for free’ from the statistics of behavior over repeated trials. Actions that lie along the most direct path simply get reinforced at a greater rate than other actions. Figure 1 If, as suggested by [1], learning is truncated after the outcome becomes predictable (via gradual extinction of phasic DA neuron activity), behavior under both schemes will stabilize to the most direct path. However, our scheme also predicts that, without such a truncation, prolonged ‘overlearning’ will result in suboptimal behavior where there is an increasing tendency to select spurious actions (see figure ​figure1).1). Such behavior may be analogous to behavior arising from certain types of DA dysfunction. We discuss the implications of this learning scheme within the context of action discovery and describe additional predicted behaviors.

In vivo effects of pardoprunox (SLV308), a partial D2/D3 receptor and 5‐HT1A receptor agonist, on rat dopamine and serotonin neuronal activity

The nonergot ligand pardoprunox (SLV308) is a dopamine (DA) D₂/D₃ and serotonin (5-HT)(1A) receptor agonist and a new candidate for the treatment of Parkinson's disease. We used in vivo electrophysiological paradigm in the rat to assess the effects of pardoprunox on DA neuronal activity in ventral tegmental area (VTA) and substantia nigra pars compacta (SNc) as well as on 5-HT neuronal activity in dorsal raphe nucleus (DRN). In the VTA, pardoprunox (2-20 μg kg⁻¹, i.v.) decreased partially the firing activity of DA neurons. Interestingly, the bursting activity of VTA DA neurons was completely suppressed. This compound both reversed and prevented the inhibition of firing rate induced by the full D₂-like receptor agonist apomorphine, confirming its partial D₂-like receptor agonistic property. Surprisingly in the SNc, pardoprunox (10 μg kg⁻¹, i.v.) either partially or fully suppressed the firing activity in two separate populations of DA neurons. Finally, in the DRN, pardoprunox (5-40 μg kg⁻¹, i.v.) completely suppressed the firing activity of 5-HT neurons. Moreover, the selective 5-HT(1A) receptor antagonist WAY-100,635 prevented and reversed the effects of pardoprunox. The present study shows that pardoprunox acts in the VTA as a potent partial D₂-like receptor agonist reducing preferentially the burst activity linked to the phasic activity of DA neurons. Unexpectedly in the SNc, pardoprunox behaves either as apartial or a full D₂-like receptor agonist. Finally in the DRN, pardoprunox is a potent full 5-HT(1A) receptor agonist. Hence, this in vivo study suggests that pardoprunox represents a promising approach for the treatment of Parkinson's disease.

Selective impairment of prediction error signaling in human dorsolateral but not ventral striatum in Parkinson's disease patients: evidence from a model-based fMRI study

Animal studies have found that the phasic activity of dopamine neurons during reward-related learning resembles a “prediction error” (PE) signal derived from a class of computational models called reinforcement learning (RL). An apparently similar signal can be measured using fMRI in the human striatum, a primary dopaminergic target. However, the fMRI signal does not measure dopamine per se, and therefore further evidence is needed to determine if these signals are related to each other. Parkinson's disease (PD) involves the neurodegeneration of the dopamine system and is accompanied by deficits in reward-related decision-making tasks. In the current study we used a computational RL model to assess striatal error signals in PD patients performing an RL task during fMRI scanning. Results show that error signals were preserved in ventral striatum of PD patients, but impaired in dorsolateral striatum, relative to healthy controls, a pattern reflecting the known selective anatomical degeneration of dopamine nuclei in PD. These findings support the notion that PE signals measured in the human striatum by the BOLD signal may reflect phasic DA activity. These results also provide evidence for a deficiency in PE signaling in the dorsolateral striatum of PD patients that may offer an explanation for their deficits observed in other reward learning tasks.

Disruption of NMDAR-dependent burst firing by dopamine neurons provides selective assessment of phasic dopamine-dependent behavior

Midbrain dopamine (DA) neurons fire in 2 characteristic modes, tonic and phasic, which are thought to modulate distinct aspects of behavior. However, the inability to selectively disrupt these patterns of activity has hampered the precise definition of the function of these modes of signaling. Here, we addressed the role of phasic DA in learning and other DA-dependent behaviors by attenuating DA neuron burst firing and subsequent DA release, without altering tonic neural activity. Disruption of phasic DA was achieved by selective genetic inactivation of NMDA-type, ionotropic glutamate receptors in DA neurons. Disruption of phasic DA neuron activity impaired the acquisition of numerous conditioned behavioral responses, and dramatically attenuated learning about cues that predicted rewarding and aversive events while leaving many other DA-dependent behaviors unaffected.

Effects of bifeprunox and aripiprazole on rat serotonin and dopamine neuronal activity and anxiolytic behaviour

The atypical antipsychotic bifeprunox is a partial dopamine D(2) and 5-HT(1A) receptor agonist. Using in-vivo electrophysiological and behavioural paradigms in the rat, the effects of bifeprunox and aripiprazole were assessed on ventral tegmental area (VTA) dopamine and dorsal raphe serotonin (5-HT) cell activity and on foot shock-induced ultrasonic vocalisation (USV). In VTA, bifeprunox and aripiprazole decreased (by 20-50%) firing of dopamine neurons. Interestingly, bursting activity was markedly reduced (by 70-100%), bursting being associated with a larger synaptic dopamine release than single spike firing. Both ligands reduced inhibition of firing rate induced by the full dopamine receptor agonist apomorphine, whereas the D(2) receptor antagonist haloperidol prevented these inhibitory effects, confirming partial D(2)-like agonistic properties. On 5-HT neurons, bifeprunox was more potent than aripiprazole to suppress firing activity. The 5-HT(1A) receptor antagonist WAY-100,635 prevented their effects. In the USV test of anxiolytic-like activity, bifeprunox had higher potency than aripiprazole to reduce vocalisations. Both WAY-100,635 and haloperidol reversed the effects of both agonists. The present in-vivo study shows that bifeprunox is a potent partial D(2)-like and 5-HT(1A) receptor agonist reducing preferentially the phasic activity of dopamine neurons. Thus, bifeprunox would be expected to be an effective compound against positive and negative symptoms of schizophrenia.

Temporal Difference Modeling of the Blood-Oxygen Level Dependent Response During Aversive Conditioning in Humans: Effects of Dopaminergic Modulation

The prediction error (PE) hypothesized by the temporal difference model has been shown to correlate with the phasic activity of dopamine neurons during reward learning and the blood-oxygen level dependent (BOLD) response during reward and aversive conditioning tasks. We hypothesized that dopamine would modulate the PE related signal in aversive conditioning and that haloperidol would reduce PE related activity, while an acute dose of amphetamine would increase PE related activity in the ventral striatum. Healthy participants took an acute dose of amphetamine, haloperidol, or placebo. We used functional magnetic resonance imaging (fMRI) to measure the BOLD signal while they carried out an aversive conditioning task, using cutaneous electrical stimulation as the unconditioned stimulus (US) and yellow and blue circles as conditioned stimulus (CS+ and CS-, respectively). Prediction error related BOLD activity was seen only in the ventral striatum in the placebo subjects. The subjects given amphetamine showed a wider network of PE related BOLD activity, including the ventral striatum, globus pallidus, putamen, insula, anterior cingulate, and substantia nigra/ventral tegmental area. Haloperidol subjects did not show PE related activity in any of these regions. Our results provide the first demonstration that the modulation of dopamine transmission affects both the physiological correlates and PE related BOLD activity during aversive learning.

Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating TD errors

Evidence that the delay-period activity of dopamine neurons corresponds to reward uncertainty rather than backpropagating TD errors

Extinction of Cocaine Self-Administration Reveals Functionally and Temporally Distinct Dopaminergic Signals in the Nucleus Accumbens

While Pavlovian and operant conditioning influence drug-seeking behavior, the role of rapid dopamine signaling in modulating these processes is unknown. During self-administration of cocaine, two dopaminergic signals, measured with 100 ms resolution, occurred immediately before and after the lever press (termed pre- and post-response dopamine transients). Extinction of self-administration revealed that these two signals were functionally distinct. Pre-response transients, which could reflect the motivation to obtain the drug, did not decline during extinction. Remarkably, post-response dopamine transients attenuated as extinction progressed, suggesting that they encode the learned association between environmental cues and cocaine. A third type of dopamine transient, not time locked to overt stimuli, decreased in frequency during extinction and correlated with calculated cocaine concentrations. These results show that dopamine release transients involved in different aspects of cocaine self-administration are highly plastic--differentially governed by motivation, learned associations linked with environmental stimuli, and the pharmacological actions of cocaine.