Reward-dependent Learning Research Articles

Acute and long-term effects of adolescent methylphenidate on decision-making and dopamine receptor mRNA expression in the orbitofrontal cortex

Though commonly used as a treatment for ADHD, the psychostimulant methylphenidate (MPH) is also misused and abused in adolescence in both clinical and general populations. Although MPH acts via pathways activated by other drugs of abuse, the short- and long-term effects of MPH on reward processing in learning and decision-making are not clearly understood. We examined the effect of adolescent MPH treatment on a battery of reward-directed behaviors both in adolescence during its administration and in adulthood after its discontinuation. We further measured whether MPH had lasting effects on dopamine receptor mRNA expression in orbitofrontal cortex (OFC) that may correspond with behavior. Long-Evans rats were injected with MPH (0, 1, 2.5, or 5mg/kg IP) twice daily from middle to late adolescence (PD38-57). During adolescence, the high dose of MPH reduced preference for large rewards in a Reward Magnitude Discrimination task, but did not affect preference for smaller-sooner rewards in a Delay Discounting task. In adulthood, after discontinuation of MPH, animals previously treated with the moderate dose of MPH showed improved acquisition, but not reversal, in a Reversal Learning task. MPH exposure did not increase preference for large-risky rewards in a Risk task in adulthood. We then quantified mRNA expression of D1, D2, and D3 receptors in the OFC using qPCR. MPH increased mRNA expression of dopamine D3 receptor subtype, but not D1 or D2. Overall, these results indicate that MPH has both immediate and lasting effects on reward-dependent learning and decisions, as well as dopaminergic function in rodents.

Reward-based plasticity of spatial priority maps: Exploiting inter-subject variability to probe the underlying neurobiology

ABSTRACTRecent evidence indicates that the attentional priority of objects and locations is altered by the controlled delivery of reward, reflecting reward-based attentional learning. Here, we take an approach hinging on intersubject variability to probe the neurobiological bases of the reward-driven plasticity of spatial priority maps. Specifically, we ask whether an individual’s susceptibility to the reward-based treatment can be accounted for by specific predictors, notably personality traits that are linked to reward processing (along with more general personality traits), but also gender. Using a visual search protocol, we show that when different target locations are associated with unequal reward probability, different priorities are acquired by the more rewarded relative to the less rewarded locations. However, while males exhibit the expected pattern of results, with greater priority for locations associated with higher reward, females show an opposite trend. Critically, both the extent and the direction of reward-based adjustments are further predicted by personality traits indexing reward sensitivity, indicating that not only male and female brains are differentially sensitive to reward, but also that specific personality traits further contribute to shaping their learning-dependent attentional plasticity. These results contribute to a better understanding of the neurobiology underlying reward-dependent attentional learning and cross-subject variability in this domain.

Neural substrates of cognitive biases during probabilistic inference.

Decision making often requires simultaneously learning about and combining evidence from various sources of information. However, when making inferences from these sources, humans show systematic biases that are often attributed to heuristics or limitations in cognitive processes. Here we use a combination of experimental and modelling approaches to reveal neural substrates of probabilistic inference and corresponding biases. We find systematic deviations from normative accounts of inference when alternative options are not equally rewarding; subjects' choice behaviour is biased towards the more rewarding option, whereas their inferences about individual cues show the opposite bias. Moreover, inference bias about combinations of cues depends on the number of cues. Using a biophysically plausible model, we link these biases to synaptic plasticity mechanisms modulated by reward expectation and attention. We demonstrate that inference relies on direct estimation of posteriors, not on combination of likelihoods and prior. Our work reveals novel mechanisms underlying cognitive biases and contributions of interactions between reward-dependent learning, decision making and attention to high-level reasoning.

Editorial: Emergent Neural Computation from the Interaction of Different Forms of Plasticity.

Editorial: Emergent Neural Computation from the Interaction of Different Forms of Plasticity.

Emergence of task-dependent representations in working memory circuits.

A wealth of experimental evidence suggests that working memory circuits preferentially represent information that is behaviorally relevant. Still, we are missing a mechanistic account of how these representations come about. Here we provide a simple explanation for a range of experimental findings, in light of prefrontal circuits adapting to task constraints by reward-dependent learning. In particular, we model a neural network shaped by reward-modulated spike-timing dependent plasticity (r-STDP) and homeostatic plasticity (intrinsic excitability and synaptic scaling). We show that the experimentally-observed neural representations naturally emerge in an initially unstructured circuit as it learns to solve several working memory tasks. These results point to a critical, and previously unappreciated, role for reward-dependent learning in shaping prefrontal cortex activity.

Synchrony can destabilize reward-sensitive networks.

When exposed to rewarding stimuli, only some animals develop persistent craving. Others are resilient and do not. How the activity of neural populations relates to the development of persistent craving behavior is not fully understood. Previous computational studies suggest that synchrony helps a network embed certain patterns of activity, although the role of synchrony in reward-dependent learning has been less studied. Increased synchrony has been reported as a marker for both susceptibility and resilience to developing persistent craving. Here we use computational simulations to study the effect of reward salience on the ability of synchronous input to embed a new pattern of activity into a neural population. Our main finding is that weak stimulus-reward correlations can facilitate the short-term repetition of a pattern of neural activity, while blocking long-term embedding of that pattern. Interestingly, synchrony did not have this dual effect on all patterns, which suggests that synchrony is more effective at embedding some patterns of activity than others. Our results demonstrate that synchrony can have opposing effects in networks sensitive to the correlation structure of their inputs, in this case the correlation between stimulus and reward. This work contributes to an understanding of the interplay between synchrony and reward-dependent plasticity.

Striatal Cholinergic Interneurons Drive GABA Release from Dopamine Terminals

Striatal cholinergic interneurons are implicated in motor control, associative plasticity, and reward-dependent learning. Synchronous activation of cholinergic interneurons triggers large inhibitory synaptic currents in dorsal striatal projection neurons, providing one potential substrate for control of striatal output, but the mechanism for these GABAergic currents is not fully understood. Using optogenetics and whole-cell recordings in brain slices, we find that a large component of these inhibitory responses derive from action-potential-independent disynaptic neurotransmission mediated by nicotinic receptors. Cholinergically driven IPSCs were not affected by ablation of striatal fast-spiking interneurons but were greatly reduced after acute treatment with vesicular monoamine transport inhibitors or selective destruction of dopamine terminals with 6-hydroxydopamine, indicating that GABA release originated from dopamine terminals. These results delineate a mechanism in which striatal cholinergic interneurons can co-opt dopamine terminals to drive GABA release and rapidly inhibit striatal output neurons.

EPA-0992 – Alterations in cue-reactivity and learning mechanism in alcohol dependent patients

Alcohol addiction is assumed to reflect the endpoint of a series of transitions: from initial alcohol intake that causes hedonic feelings, through loss of control over this behaviour, such that it becomes compulsive. Alcohol dependent patients are dominated by their addiction despite the negative consequences and experience other stimuli as providing little reward. Still, some patients manage to abstain from alcohol, whereas others relapse quiet often. Two recent studies (Beck et al., 2012; Charlet et al., 2013) could show that relapsers not only display greater atrophy in limbic and prefrontal areas, but also seem to have attenuated functional connectivity of these areas during cue-reactivity and emotional tasks. It has been proposed that habitual stimulus-triggered responses result in the compulsive nature of alcohol consumption (Everitt and Robbins, 2005), the formation of these inflexible stimulus-response associations being described as habit-based learning which is commonly seen as the counterpart of outcome-directed actions, so called goal-directed learning. Our current research project therefore aims to comprehensively assess how learning alterations might contribute to the assignment of aberrantly high value in the development and recurrence of alcoholism. The project compares alcohol-dependent patients to healthy controls on a battery of tasks assessing Pavlovian, habitual and goal-directed reward-dependent learning behaviourally and with functional MRI. We are going to report preliminary results of our ongoing research.

GABAergic inputs from direct and indirect striatal projection neurons onto cholinergic interneurons in the primate putamen.

Striatal cholinergic interneurons (ChIs) are involved in reward-dependent learning and the regulation of attention. The activity of these neurons is modulated by intrinsic and extrinsic γ-aminobutyric acid (GABA)ergic and glutamatergic afferents, but the source and relative prevalence of these diverse regulatory inputs remain to be characterized. To address this issue, we performed a quantitative ultrastructural analysis of the GABAergic and glutamatergic innervation of ChIs in the postcommissural putamen of rhesus monkeys. Postembedding immunogold localization of GABA combined with peroxidase immunostaining for choline acetyltransferase showed that 60% of all synaptic inputs to ChIs originate from GABAergic terminals, whereas 21% are from putatively glutamatergic terminals that establish asymmetric synapses, and 19% from other (non-GABAergic) sources of symmetric synapses. Double pre-embedding immunoelectron microscopy using substance P and Met-/Leu-enkephalin antibodies to label GABAergic terminals from collaterals of "direct" and "indirect" striatal projection neurons, respectively, revealed that 47% of the indirect pathway terminals and 36% of the direct pathway terminals target ChIs. Together, substance P- and enkephalin-positive terminals represent 24% of all synapses onto ChIs in the monkey putamen. These findings show that ChIs receive prominent GABAergic inputs from multiple origins, including a significant contingent from axon collaterals of direct and indirect pathway projection neurons.

∆FosB differentially modulates nucleus accumbens direct and indirect pathway function

Synaptic modifications in nucleus accumbens (NAc) medium spiny neurons (MSNs) play a key role in adaptive and pathological reward-dependent learning, including maladaptive responses involved in drug addiction. NAc MSNs participate in two parallel circuits, direct and indirect pathways that subserve distinct behavioral functions. Modification of NAc MSN synapses may occur in part via changes in the transcriptional potential of certain genes in a cell type–specific manner. The transcription factor ∆FosB is one of the key proteins implicated in the gene expression changes in NAc caused by drugs of abuse, yet its effects on synaptic function in NAc MSNs are unknown. Here, we demonstrate that overexpression of ∆FosB decreased excitatory synaptic strength and likely increased silent synapses onto D1 dopamine receptor–expressing direct pathway MSNs in both the NAc shell and core. In contrast, ∆FosB likely decreased silent synapses onto NAc shell, but not core, D2 dopamine receptor–expressing indirect pathway MSNs. Analysis of NAc MSN dendritic spine morphology revealed that ∆FosB increased the density of immature spines in D1 direct but not D2 indirect pathway MSNs. To determine the behavioral consequences of cell type-specific actions of ∆FosB, we selectively overexpressed ∆FosB in D1 direct or D2 indirect MSNs in NAc in vivo and found that direct (but not indirect) pathway MSN expression enhances behavioral responses to cocaine. These results reveal that ∆FosB in NAc differentially modulates synaptic properties and reward-related behaviors in a cell type- and subregion-specific fashion.

Reward-dependent perceptual learning of objects (a pilot study)

Reward-dependent perceptual learning of objects (a pilot study)

Subcellular Location of PKA Controls Striatal Plasticity: Stochastic Simulations in Spiny Dendrites

Dopamine release in the striatum has been implicated in various forms of reward dependent learning. Dopamine leads to production of cAMP and activation of protein kinase A (PKA), which are involved in striatal synaptic plasticity and learning. PKA and its protein targets are not diffusely located throughout the neuron, but are confined to various subcellular compartments by anchoring molecules such as A-Kinase Anchoring Proteins (AKAPs). Experiments have shown that blocking the interaction of PKA with AKAPs disrupts its subcellular location and prevents LTP in the hippocampus and striatum; however, these experiments have not revealed whether the critical function of anchoring is to locate PKA near the cAMP that activates it or near its targets, such as AMPA receptors located in the post-synaptic density. We have developed a large scale stochastic reaction-diffusion model of signaling pathways in a medium spiny projection neuron dendrite with spines, based on published biochemical measurements, to investigate this question and to evaluate whether dopamine signaling exhibits spatial specificity post-synaptically. The model was stimulated with dopamine pulses mimicking those recorded in response to reward. Simulations show that PKA colocalization with adenylate cyclase, either in the spine head or in the dendrite, leads to greater phosphorylation of DARPP-32 Thr34 and AMPA receptor GluA1 Ser845 than when PKA is anchored away from adenylate cyclase. Simulations further demonstrate that though cAMP exhibits a strong spatial gradient, diffusible DARPP-32 facilitates the spread of PKA activity, suggesting that additional inactivation mechanisms are required to produce spatial specificity of PKA activity.

Neural correlates of individual differences in manual imitation fidelity

Imitation is crucial for social learning, and so it is important to identify what determines between-subject variability in imitation fidelity. This might help explain what makes some people, like those with social difficulties such as in autism spectrum disorder (ASD), significantly worse at performance on these tasks than others. A novel paradigm was developed to provide objective measures of imitation fidelity in which participants used a touchscreen to imitate videos of a model drawing different shapes. Comparisons between model and participants' kinematic data provided three measures of imitative fidelity. We hypothesized that imitative ability would predict variation in BOLD signal whilst performing a simple imitation task in the MRI-scanner. In particular, an overall measure of accuracy (correlation between model and imitator) would predict activity in the overarching imitation system, whereas bias would be subject to more general aspects of motor control. Participants lying in the MRI-scanner were instructed to imitate different grips on a handle, or to watch someone or a circle moving the handle. Our hypothesis was partly confirmed as correlation between model and imitator was mediated by somatosensory cortex but also ventromedial prefrontal cortex, and bias was mediated mainly by cerebellum but also by the medial frontal and parietal cortices and insula. We suggest that this variance differentially reflects cognitive functions such as feedback-sensitivity and reward-dependent learning, contributing significantly to variability in individuals' imitative abilities as characterized by objective kinematic measures.

S23 * NEUROBIOLOGICAL ENDOPHENOTYPES IN ALCOHOL DEPENDENCE * S23.1 * GENETIC DETERMINANTS OF ALCOHOL ADDICTION: SEARCHING FOR AN ENDOPHENOTYPE ASSOCIATED WITH SWEET LIKING IN POPULATION OF FAMILIES WITH HISTORY OF ALCOHOL ADDICTION

S23 * NEUROBIOLOGICAL ENDOPHENOTYPES IN ALCOHOL DEPENDENCE * S23.1 * GENETIC DETERMINANTS OF ALCOHOL ADDICTION: SEARCHING FOR AN ENDOPHENOTYPE ASSOCIATED WITH SWEET LIKING IN POPULATION OF FAMILIES WITH HISTORY OF ALCOHOL ADDICTION

Stochastic gradient ascent learning with spike timing dependent plasticity

Stochastic gradient ascent learning exploits correlations of parameter variations with overall success of a system. This algorithmic idea has been related to neuronal network learning by postulating eligibility traces at synapses, which make them selectable for synaptic changes depending on later reward signals ([1] and [2]). Formalizations of the synaptic and neuronal dynamics supporting gradient ascent learning in terms of differential equations exhibit strong similarities with a recent formulation of spike timing dependent plasticity (STDP) [3] when it is combined with a reward signal. Here we present conditions under which reward modulated STDP is in fact guaranteed to maximize expected reward. We present numerical simulations underlining the relevance of realistic STDP models for reward dependent learning. In particular, we find that the nonlinear adaptation to pre- and post-synaptic activities of STDP [3] contributes to stable learning. Figure 1 Learning the XOR function with a reward modulated STDP rule. Left: Output activity versus training episode in a feed forward network with Poisson-like neurons (2 input nodes, 10 hidden nodes and 1 output node). The output activity for the [true, false] ...

Bayesian cue integration as a developmental outcome of reward mediated learning.

Average human behavior in cue combination tasks is well predicted by Bayesian inference models. As this capability is acquired over developmental timescales, the question arises, how it is learned. Here we investigated whether reward dependent learning, that is well established at the computational, behavioral, and neuronal levels, could contribute to this development. It is shown that a model free reinforcement learning algorithm can indeed learn to do cue integration, i.e. weight uncertain cues according to their respective reliabilities and even do so if reliabilities are changing. We also consider the case of causal inference where multimodal signals can originate from one or multiple separate objects and should not always be integrated. In this case, the learner is shown to develop a behavior that is closest to Bayesian model averaging. We conclude that reward mediated learning could be a driving force for the development of cue integration and causal inference.

Postsynaptic TRPV1 triggers cell type–specific long-term depression in the nucleus accumbens

Synaptic modifications in the nucleus accumbens (NAc) are important for adaptive and pathological reward-dependent learning. Medium spiny neurons (MSNs), the major cell type in the NAc, participate in two parallel circuits that subserve distinct behavioral functions, yet little is known about differences in their electrophysiological and synaptic properties. Using bacterial artificial chromosome transgenic mice, we found that synaptic activation of group I metabotropic glutamate receptors in NAc MSNs in the indirect, but not direct, pathway led to the production of endocannabinoids, which activated presynaptic CB1 receptors to trigger endocannabinoid-mediated long-term depression (eCB-LTD) as well as postsynaptic transient receptor potential vanilloid 1 (TRPV1) channels to trigger a form of LTD resulting from endocytosis of AMPA receptors. These results reveal a previously unknown action of TRPV1 channels and indicate that the postsynaptic generation of endocannabinoids can modulate synaptic strength in a cell type-specific fashion by activating distinct pre- and postsynaptic targets.

Developing a working memory with reward-modulated STDP

Event Abstract Back to Event Developing a working memory with reward-modulated STDP It is commonly believed that the neural correlate of working memory is the selective persistent neuronal activity during delay period. Such dynamics are traditionally modelled by recurrent networks exhibiting attractor dynamics, where a non-selective background activity attractor coexists with multiple stimulus-specific attractors [1]. These models provide a good match to experimental data, but so far completely ignore how working memory develops in the context of goal directed behavior. Given that working memory evolves significantly after birth, in both capacity and maintenance time [2, 3], it is interesting to ask to what extent this development is driven by goal directed learning. To address this question, we investigate whether working memory properties can emerge in a generic recurrent neural network due to reward-dependent learning. Specifically, we consider the classical delayed response task requiring a stimulus to be stored in memory for a variable time such that a specific response can be delivered after an external cue. The model consists of a sparsely connected neural network receiving localized, stimulus specific inputs. This neuron population is connected to a layer of motor neurons, which decide the action to be taken through a winner-take-all mechanism. Reward-modulated STDP [4] is used to shape the recurrent connectivity and the synapses to motor neurons. We show that the recurrent neural network can reshape its connectivity to successfully perform the task. Synaptic changes lead to a segregation of the network in stimulus-specific subpopulations. Moreover, these subpopulations exhibit stimulus-specific activation during maintenance, as seen in neurophysiological studies. The final performance in the task is significantly higher compared to that of networks with fixed connectivity or networks trained with STDP only. Our model demonstrates that working memory dynamics may naturally emerge in a recurrent neural network with reward-modulated STDP, suggesting that reward dependent learning may be a central aspect of the development of working memory. The authors are supported by EC MEXT project PLICON and the German Federal Ministry of Education and Research (BMBF) within the "Bernstein Focus: Neurotechnology" through research grant 01GQ0840.

Exploring GABAergic and dopaminergic effects in a minimal model of a medium spiny projection neuron

Striatum is the input stage of the basal ganglia, a collection of nuclei in the midbrain. The basal ganglia are involved in cognitive and motor behaviour, including reward-dependent learning. The reward system in the brain is heavily linked to the dopaminergic system, and many striatal neurons react in a reward-dependent manner. This study explores a minimal model of a striatal medium spiny (MS) projection neuron displaying dopamine-induced bistability. MS neurons mostly fluctuate between two states, a hyperpolarised down-state and a depolarised up-state. MS neurons are only active in the up-state and therefore spiking requires the transition from the down-state. For high dopamine levels in the model, the appearance of a bifurcation results in more distinct state transitions. GABAergic input from local fast-spiking interneurons to MS neurons results in a small depolarisation, but far from causing a transition to the up-state by itself. We investigate if a GABAergic PSP could facilitate the transition to the up-state elicited by glutamatergic input. The model predicts that GABAergic input to MS neurons might facilitate and speed up the transition to the up-state. The prerequisite for this is that the GABAergic enhancement starts slightly before the glutamatergic increase which causes the up-state transition.

Opposing BOLD responses to reciprocated and unreciprocated altruism in putative reward pathways

Mesencephalic dopamine neurons are believed to facilitate reward-dependent learning by computing errors in reward predictions. We used fMRI to test whether this system was activated as expected in response to errors in predictions about whether a social partner would reciprocate an act of altruism. Nineteen subjects received fMRI scans as they played a series of single-shot Prisoner's Dilemma games with partners who were outside the scanner. In both ventromedial prefrontal cortex and ventral striatum, reciprocated and unreciprocated cooperation were associated with positive and negative BOLD responses, respectively. Our results are consistent with the hypothesis that mesencephalic dopamine projection sites carry information about errors in reward prediction that allow us to learn who can and cannot be trusted to reciprocate favors.