Reward-based Learning Research Articles

A neural correlate of reward-based behavioral learning in caudate nucleus: a functional magnetic resonance imaging study of a stochastic decision task.

Humans can acquire appropriate behaviors that maximize rewards on a trial-and-error basis. Recent electrophysiological and imaging studies have demonstrated that neural activity in the midbrain and ventral striatum encodes the error of reward prediction. However, it is yet to be examined whether the striatum is the main locus of reward-based behavioral learning. To address this, we conducted functional magnetic resonance imaging (fMRI) of a stochastic decision task involving monetary rewards, in which subjects had to learn behaviors involving different task difficulties that were controlled by probability. We performed a correlation analysis of fMRI data by using the explanatory variables derived from subject behaviors. We found that activity in the caudate nucleus was correlated with short-term reward and, furthermore, paralleled the magnitude of a subject's behavioral change during learning. In addition, we confirmed that this parallelism between learning and activity in the caudate nucleus is robustly maintained even when we vary task difficulty by controlling the probability. These findings suggest that the caudate nucleus is one of the main loci for reward-based behavioral learning.

Postnatal dexamethasone and long term learning and memory functions in developing rats: Effect of postnatal age and gender

In this study, we investigated the effect of dexamethasone on the long-term learning and memory functions in developing rats. In Sprague-Dawley rat pups, we administered a daily dose of dexamethasone (0.5 mg/kg/day) for three consecutive days in three groups of animals: the “ultra-early” group received steroids on postnatal days (PND) 1–3; the “early” group received the drug on PNDs 8–10, and the “late” group received the drug on PNDs 28–30. The control group was not given any medication. All animals underwent structured CNS examinations beginning on PND 15, and continued through PND 20. The pups were tested for spatial learning and memory functions using the Morris Water Maze (MWM) on PNDs 31 through 35, 45 through 49, and 59 through 63. They were also tested for reward-based learning and memory functions using Radial Arm Maze (RAM) on PNDs 70 through 72. We analyzed the effect of dexamethasone, postnatal age, and sex on neurological milestones, and learning and memory functions. We found that neurological examination findings were similar in all groups, as were the results of the reward-based learning using RAM. However, in the MWM, the total distance of swimming and the total time to find the hidden platform showed considerable difference among the groups. Although these functions improved with postnatal age, the female pups in all three steroid groups, and the male pups in the late-steroid group lagged significantly in learning and memory functions compared to the controls, and such lags were transient. However, the interaction terms between dexamethasone, age, and sex were also significant in MWM test results. Steroids administered postnatally may have transient, retarding effect on learning and memory functions, and that animal age and sex may modify such effects. Such lags are not global, but specific to the types of memory tests used, implicating different neural circuitries in the pathogenesis of such abnormalities. Although transient, if such adverse effects occur at critical phases during brain maturation, the implications for poor, long-term outcomes may be more significant. The mechanisms underlying such changes need to be explored.

Correlated Coding of Motivation and Outcome of Decision by Dopamine Neurons

We recorded the activity of midbrain dopamine neurons in an instrumental conditioning task in which monkeys made a series of behavioral decisions on the basis of distinct reward expectations. Dopamine neurons responded to the first visual cue that appeared in each trial [conditioned stimulus (CS)] through which monkeys initiated trial for decision while expecting trial-specific reward probability and volume. The magnitude of neuronal responses to the CS was approximately proportional to reward expectations but with considerable discrepancy. In contrast, CS responses appear to represent motivational properties, because their magnitude at trials with identical reward expectation had significant negative correlation with reaction times of the animal after the CS. Dopamine neurons also responded to reinforcers that occurred after behavioral decisions, and the responses precisely encoded positive and negative reward expectation errors (REEs). The gain of coding REEs by spike frequency increased during learning act-outcome contingencies through a few months of task training, whereas coding of motivational properties remained consistent during the learning. We found that the magnitude of CS responses was positively correlated with that of reinforcers, suggesting a modulation of the effectiveness of REEs as a teaching signal by motivation. For instance, rate of learning could be faster when animals are motivated, whereas it could be slower when less motivated, even at identical REEs. Therefore, the dual correlated coding of motivation and REEs suggested the involvement of the dopamine system, both in reinforcement in more elaborate ways than currently proposed and in motivational function in reward-based decision-making and learning.

The Suprachiasmatic Nucleus is not Required for Temporal Gating of Performance on a Reward-based Learning and Memory Task

In hamsters, the expression of a learned preference for context depends upon a temporal match between the time of training and testing. In the present experiments, we investigated the role of the biological clock in the suprachiasmatic nucleus (SCN) as a provider of temporal information underlying this time dependent modulation of cognitive performance. Hamsters were tested using the conditioned place preference task (CPP) before and after ablation of the SCN. Arrhythmic animals continued to show time-of-day modulation of the CPP when trained and tested in the absence of the SCN. This supports the notion that time of day information is a component of context representation for the hamster (Antoniadis et al., 1999), and indicates that an oscillator outside of the SCN is responsible for time discrimination in reward-based learning.

On the nature and evolution of the neural bases of human language

The traditional theory equating the brain bases of language with Broca's and Wernicke's neocortical areas is wrong. Neural circuits linking activity in anatomically segregated populations of neurons in subcortical structures and the neocortex throughout the human brain regulate complex behaviors such as walking, talking, and comprehending the meaning of sentences. When we hear or read a word, neural structures involved in the perception or real-world associations of the word are activated as well as posterior cortical regions adjacent to Wernicke's area. Many areas of the neocortex and subcortical structures support the cortical-striatal-cortical circuits that confer complex syntactic ability, speech production, and a large vocabulary. However, many of these structures also form part of the neural circuits regulating other aspects of behavior. For example, the basal ganglia, which regulate motor control, are also crucial elements in the circuits that confer human linguistic ability and abstract reasoning. The cerebellum, traditionally associated with motor control, is active in motor learning. The basal ganglia are also key elements in reward-based learning. Data from studies of Broca's aphasia, Parkinson's disease, hypoxia, focal brain damage, and a genetically transmitted brain anomaly (the putative "language gene," family KE), and from comparative studies of the brains and behavior of other species, demonstrate that the basal ganglia sequence the discrete elements that constitute a complete motor act, syntactic process, or thought process. Imaging studies of intact human subjects and electrophysiologic and tracer studies of the brains and behavior of other species confirm these findings. As Dobzansky put it, "Nothing in biology makes sense except in the light of evolution" (cited in Mayr, 1982). That applies with as much force to the human brain and the neural bases of language as it does to the human foot or jaw. The converse follows: the mark of evolution on the brains of human beings and other species provides insight into the evolution of the brain bases of human language. The neural substrate that regulated motor control in the common ancestor of apes and humans most likely was modified to enhance cognitive and linguistic ability. Speech communication played a central role in this process. However, the process that ultimately resulted in the human brain may have started when our earliest hominid ancestors began to walk.

Cellular mechanisms of striatum-dependent behavioral plasticity and drug addiction.

The striatum has long been known to be involved in the control of motor behavior, since disruption of dopamine-mediated function in this brain structure is directly linked to Parkinson's disease and other disorders of movement. However, it is now accepted that both dorsal and ventral striatal nuclei are also essential for a variety of cognitive processes, which depend on reward-based stimulus-response learning. Since the neuroanatomical and neurochemical organization of dorsal and ventral striatum is only partially overlapping, it is likely that both common and nucleus-specific cellular and molecular events contribute to synaptic plasticity, learning and memory processes mediated by these cerebral structures. Alterations in cell signaling in the striatum may be particularly important in the response to both acute and chronic administration of drugs of abuse, resulting in maladaptive changes in the reward-based associative learning involved in addiction, withdrawal and relapse.

Limbic Thalamic Lesions, Appetitively Motivated Discrimination Learning, and Training-Induced Neuronal Activity in Rabbits

A substantial literature implicates the anterior and mediodorsal (limbic) thalamic nuclei and the reciprocally interconnected areas of cingulate cortex in learning, memory, and attentional processes. Previous studies have shown that limbic thalamic lesions severely impair discriminative avoidance learning and that they block development of training-induced neuronal activity in the cingulate cortex. The present study investigated the possibility that the limbic thalamus and cingulate cortex are involved in reward-based discriminative approach learning, wherein head-extension responses yielding oral contact with a drinking spout that was inserted into the conditioning chamber after a positive conditional stimulus (CS+) were reinforced with a water reward but responses to the spout after a negative conditional stimulus (CS-) were not reinforced. In this task, the rabbits learned primarily to omit their prepotent responses to the spout on CS- trials. Acquisition was severely impaired in rabbits given limbic thalamic lesions before training. As during avoidance learning, posterior cingulate cortical neurons of control rabbits developed learning-related neuronal responses to task-relevant stimuli, but this activity was severely attenuated in rabbits with lesions. These results support a general involvement of the cingulothalamic circuitry in instrumental approach and avoidance learning. The fact that learning consisted of response omission indicated that the cingulothalamic role is not limited to acquisition or production of active behavioral responses, such as locomotion. It is proposed that cingulothalamic neurons mediate associative attention, wherein enhanced neuronal responses to stimuli associated with reinforcement facilitate the selection and production of task-relevant responses.

The significance of circadian phase for performance on a reward-based learning task in hamsters

In humans and animal models, circadian modulation of learning has been demonstrated on numerous tests. However, it is unclear which aspects of the cognitive process are rhythmically regulated. In these experiments, we used a conditioned place preference task in hamsters to ask whether memory acquisition (hypothesis 1) or memory recall and performance (hypothesis 2) were subject to circadian modulation. In golden hamsters, access to a running wheel has been used as a reward to condition a place preference, but when given unrestricted access to a wheel, animals perform most of their spontaneous running within a few hours each day or circadian cycle. This suggested that either the perceived reward value of the wheel changes through the day or that the response to this reward is temporally restricted. Contrary to the hypotheses, we found that learning was not tied to the time of training nor to the time of testing, but rather animals showed a preference for a reward-paired context only at the circadian time that training had taken place. Timing is not an explicit discriminative cue in these experiments. Hence, the learning mechanism must be predisposed to register circadian time as an attribute during context learning.

Dopamine D2 receptors are present in prefrontal cortical afferents and their targets in patches of the rat caudate‐putamen nucleus

Glutamatergic neurons within the deep layers of the prefrontal cortex and dopaminergic neurons of the substantia nigra pars compacta preferentially terminate in patch-like regions within the caudate putamen nucleus (CPN). Activation of dopamine D2 receptors is known to potently modulate striatal glutamatergic transmission and may play a role in reward-based motor learning. To determine the cellular substrate for D2-mediated regulation of prefrontal corticostriatal transmission in striatal patches, we combined anterograde transport of biotinylated dextran amine (BDA) with immunogold-silver labeling of a D2 receptor antipeptide antiserum in rat brain. Injections centered in deep layers of the dorsal part of the anterior cingulate cortex, one of the prefrontal cortical regions, produced varicose axonal BDA labeling in a patch-like distribution in the dorsomedial CPN. Electron microscopy showed that in these patch compartments, BDA labeling was present exclusively in axons and terminals (total number = 581), 9% of which contained detectable D2-like immunoreactivity. Thirty percent of the BDA-labeled terminals formed asymmetric excitatory synapses with dendritic spine heads, and the remainder were without recognizable junctions. The recipient spines were unlabeled or contained immunogold-silver particles for D2 receptors. A few of the D2-labeled spines also received convergent, often nonsynaptic contact from D2-labeled terminals resembling dopaminergic afferents. In addition, the corticostriatal terminals often apposed spiny and nonspiny neuronal profiles that contained D2 labeling. These results suggest that dopamine D2 receptors are strategically positioned for presynaptic and postsynaptic modulation of prefrontal corticostriatal excitation of spiny neurons in striatal patches. The findings have direct implications for D2-mediated control of reward-related motor learning.

Spatial cognition and neuro-mimetic navigation: a model of hippocampal place cell activity

A computational model of hippocampal activity during spatial cognition and navigation tasks is presented. The spatial representation in our model of the rat hippocampus is built on-line during exploration via two processing streams. An allothetic vision-based representation is built by unsupervised Hebbian learning extracting spatio-temporal properties of the environment from visual input. An idiothetic representation is learned based on internal movement-related information provided by path integration. On the level of the hippocampus, allothetic and idiothetic representations are integrated to yield a stable representation of the environment by a population of localized overlapping CA3-CA1 place fields. The hippocampal spatial representation is used as a basis for goal-oriented spatial behavior. We focus on the neural pathway connecting the hippocampus to the nucleus accumbens. Place cells drive a population of locomotor action neurons in the nucleus accumbens. Reward-based learning is applied to map place cell activity into action cell activity. The ensemble action cell activity provides navigational maps to support spatial behavior. We present experimental results obtained with a mobile Khepera robot.

Considerations upon the anatomical model of reward-based learning in the basal ganglia

The nigrostriatal pathway appears to be very important in the reward-based learning. The dopaminergic neurons in the substantia nigra pars compacta (SNC) fire in relation to primary rewards and reward-conditioned stimuli, but not to rewards that are expected. It has been hypothesized that the anatomical framework for the selective response of these neurons is organized in the projections from some paralimbic areas in the frontal lobe to the striosomes of the caudate nucleus, which are also directly connected with the dopaminergic neurons of the SNC. Here, we present two additional pathways that may be related with this neurophysiological finding. We hypothesize that the connections of the paralimbic cortices with the ventral system of the basal ganglia and then with the thalamus and the hypothalamus, and the circuit ventral striatum-substantia nigra pars reticulata-thalamus-striatum could be also involved in the reward-based learning.

Probabilistic learning and reversal deficits in patients with Parkinson’s disease or frontal or temporal lobe lesions: possible adverse effects of dopaminergic medication

Three groups of patients with Parkinson’s disease (PD) — mild, unmedicated (UPD), mild, medicated (MPD) and severe, medicated (SPD) — and patients with lesions of the frontal lobe (FLL) or temporal lobe (TLL) were compared with matched controls on the learning and reversal of probabilistic and two-pair concurrent colour discriminations. Both of the cortical lesion groups showed reversal deficits, with no increase in perseverative responding. The UPD group, although impaired on a spatial recognition task, showed intact discrimination learning and reversal; the MPD and SPD patients showed non-perseverative reversal impairments on both reversal tasks. Two hypotheses — based on disease severity and possible deleterious effects of medication — are offered to explain the reversal impairments of the PD patients and the results are discussed in terms of the role of dopamine in reward-based learning.

Building action repertoires: memory and learning functions of the basal ganglia

Research on the basal ganglia suggests that they are critically involved in building up sequences of behavior into meaningful, goal-directed repertoires. Work on rodents, monkeys and humans suggests that the basal ganglia act as part of a distributed forebrain system that helps to encode such repertoires through behavioral learning, and that is engaged in the expression of such repertoires once they have been internalized. The basal ganglia also may be critical to the expression of innate behavioral routines. Experimental findings on reward-based learning suggest that neural activity in the striatum and substantia nigra, pars compacta changes during behavioral learning. New evidence also suggests extreme specificity in the neural connections interrelating the basal ganglia, cerebral cortex and thalamus. Adaptive control of behavior may centrally depend on these circuits and the evaluator—reinforcement circuits that modulate them.