Reward-based Learning Research Articles

Orbitofrontal cortex grey matter volume is related to children’s depressive symptoms

Adults with a history of depression show distinct patterns of grey matter volume (GMV) in frontal cortical (e.g., prefrontal cortex, orbitofrontal cortex) and limbic (e.g., anterior cingulate, amygdala, hippocampus, dorsal striatum) structures, regions relevant to the processing and regulation of reward, which is impaired in the context of depression. However, it is unclear whether these GMV associations with depression precede depressive disorder onset or whether GMV is related to early emerging symptoms or familial depression. To address these questions, we used voxel-based morphometry (VBM) to examine GMV in 85 community-dwelling children (M = 11.12 years, SD = 0.63 years) screened for current and lifetime depression. Associations between children’s depressive symptoms (self- and mother-report of children’s symptoms), children’s maternal depression history, and GMV were examined. Although maternal depression history was unrelated to children’s GMV, child GMV in the orbitofrontal cortex (OFC) was negatively related to children’s self-reported depressive symptoms, using both a priori ROI and whole-brain analyses. Moderated regression analyses indicated that girls’ GMV was negatively related to girls’ depressive symptoms (as indexed by both self- and mother-report of girls’ symptoms), whereas boys’ symptoms were positively related to GMV. Our findings suggest that brain morphology in the OFC, a region with functional roles in processes relevant to depressive symptoms (i.e., reward-based learning and reward processing), is associated with early depressive symptoms prior to the development of clinically significant depression.

Endocannabinoid Signaling from 2-Arachidonoylglycerol to CB1 Cannabinoid Receptor Facilitates Reward-based Learning of Motor Sequence

The endocannabinoid system modulates synaptic transmission, controls neuronal excitability, and is involved in various brain functions including learning and memory. 2-arachidonoylglycerol, a major endocannabinoid produced by diacylglycerol lipase-α (DGLα), is released from postsynaptic neurons, retrogradely activates presynaptic CB1 cannabinoid receptors, and induces short-term or long-term synaptic plasticity. To examine whether and how the endocannabinoid system contributes to reward-based learning of a motor sequence, we subjected male CB1-knockout (KO) and DGLα-KO mice to three types of operant lever-press tasks. First, we trained mice to press one of three levers labeled A, B, and C for a food reward (one-lever task). Second, we trained mice to press the three levers in the order of A, B, and C (three-lever task). Third, the order of the levers was reversed to C, B, and A (reverse three-lever task). We found that CB1-KO mice and DGLα-KO mice exhibited essentially the same deficits in the operant lever-press tasks. In the one-lever task, both strains of knockout mice showed a slower rate of learning to press a lever for food. In the three-lever task, both strains of knockout mice showed a slower rate of learning of the motor sequence. In the reverse three-lever task, both strains of knockout mice needed more lever presses for reversal learning. These results suggest that the endocannabinoid system facilitates reward-based learning of a motor sequence by conferring the flexibility with which animals can switch between strategies.

Social Bayes: Using Bayesian Modeling to Study Autistic Trait–Related Differences in Social Cognition

BackgroundThe autistic spectrum is characterized by profound impairments of social interaction. The exact subpersonal processes, however, that underlie the observable lack of social reciprocity are still a matter of substantial controversy. Recently, it has been suggested that the autistic spectrum might be characterized by alterations of the brain’s inference about the causes of socially relevant sensory signals. MethodsWe used a novel reward-based learning task that required integration of nonsocial and social cues in conjunction with computational modeling. Thirty-six healthy subjects were selected based on their score on the Autism-Spectrum Quotient (AQ), and AQ scores were assessed for correlations with cue-related model parameters and task scores. ResultsIndividual differences in AQ scores were significantly correlated with participants’ total task scores, with high AQ scorers performing more poorly in the task (r = −.39, 95% confidence interval = −0.68 to −0.13). Computational modeling of the behavioral data unmasked a learning deficit in high AQ scorers, namely, the failure to integrate social context to adapt one’s belief precision—the precision afforded to prior beliefs about changing states in the world—particularly in relation to the nonsocial cue. ConclusionsMore pronounced autistic traits in a group of healthy control subjects were related to lower scores associated with misintegration of the social cue. Computational modeling further demonstrated that these trait-related performance differences are not explained by an inability to process the social stimuli and their causes, but rather by the extent to which participants consider social information to infer the nonsocial cue.

Domain-Specific Working Memory, But Not Dopamine-Related Genetic Variability, Shapes Reward-Based Motor Learning.

The addition of rewarding feedback to motor learning tasks has been shown to increase the retention of learning, spurring interest in its possible utility for rehabilitation. However, motor tasks using rewarding feedback have repeatedly been shown to lead to great interindividual variability in performance. Understanding the causes of such variability is vital for maximizing the potential benefits of reward-based motor learning. Thus, using a large human cohort of both sexes (n = 241), we examined whether spatial (SWM), verbal, and mental rotation (RWM) working memory capacity and dopamine-related genetic profiles were associated with performance in two reward-based motor tasks. The first task assessed the participant's ability to follow a slowly shifting reward region based on hit/miss (binary) feedback. The second task investigated the participant's capacity to preserve performance with binary feedback after adapting to the rotation with full visual feedback. Our results demonstrate that higher SWM is associated with greater success and an enhanced capacity to reproduce a successful motor action, measured as change in reach angle following reward. In contrast, higher RWM was predictive of an increased propensity to express an explicit strategy when required to make large reach angle adjustments. Therefore, SWM and RWM were reliable, but dissociable, predictors of success during reward-based motor learning. Change in reach direction following failure was also a strong predictor of success rate, although we observed no consistent relationship with working memory. Surprisingly, no dopamine-related genotypes predicted performance. Therefore, working memory capacity plays a pivotal role in determining individual ability in reward-based motor learning.SIGNIFICANCE STATEMENT Reward-based motor learning tasks have repeatedly been shown to lead to idiosyncratic behaviors that cause varying degrees of task success. Yet, the factors determining an individual's capacity to use reward-based feedback are unclear. Here, we assessed a wide range of possible candidate predictors, and demonstrate that domain-specific working memory plays an essential role in determining individual capacity to use reward-based feedback. Surprisingly, genetic variations in dopamine availability were not found to play a role. This is in stark contrast with seminal work in the reinforcement and decision-making literature, which show strong and replicated effects of the same dopaminergic genes in decision-making. Therefore, our results provide novel insights into reward-based motor learning, highlighting a key role for domain-specific working memory capacity.

Multi-Level Policy and Reward-Based Deep Reinforcement Learning Framework for Image Captioning

Image captioning is one of the most challenging tasks in AI because it requires an understanding of both complex visuals and natural language. Because image captioning is essentially a sequential prediction task, recent advances in image captioning have used reinforcement learning (RL) to better explore the dynamics of word-by-word generation. However, the existing RL-based image captioning methods rely primarily on a single policy network and reward function-an approach that is not well matched to the multi-level (word and sentence) and multi-modal (vision and language) nature of the task. To solve this problem, we propose a novel multi-level policy and reward RL framework for image captioning that can be easily integrated with RNN-based captioning models, language metrics, or visual-semantic functions for optimization. Specifically, the proposed framework includes two modules: 1) a multi-level policy network that jointly updates the word- and sentence-level policies for word generation; and 2) a multi-level reward function that collaboratively leverages both a vision-language reward and a language-language reward to guide the policy. Furthermore, we propose a guidance term to bridge the policy and the reward for RL optimization. The extensive experiments on the MSCOCO and Flickr30k datasets and the analyses show that the proposed framework achieves competitive performances on a variety of evaluation metrics. In addition, we conduct ablation studies on multiple variants of the proposed framework and explore several representative image captioning models and metrics for the word-level policy network and the language-language reward function to evaluate the generalization ability of the proposed framework.

Classical conditioning drives learned reward prediction signals in climbing fibers across the lateral cerebellum.

Classical models of cerebellar learning posit that climbing fibers operate according to a supervised learning rule to instruct changes in motor output by signaling the occurrence of movement errors. However, cerebellar output is also associated with non-motor behaviors, and recently with modulating reward association pathways in the VTA. To test how the cerebellum processes reward related signals in the same type of classical conditioning behavior typically studied to evaluate reward processing in the VTA and striatum, we have used calcium imaging to visualize instructional signals carried by climbing fibers across the lateral cerebellum in mice before and after learning. We find distinct climbing fiber responses in three lateral cerebellar regions that can each signal reward prediction. These instructional signals are well suited to guide cerebellar learning based on reward expectation and enable a cerebellar contribution to reward driven behaviors, suggesting a broad role for the lateral cerebellum in reward-based learning.

Mobile robot navigation with the combination of supervised learning in cerebellum and reward-based learning in basal ganglia

Autonomous navigation of mobile robot in unknown environment has attracted much attention of scholars over the past decades, and many bio-inspired heuristic navigation models have been presented, such as the cerebellum and basal ganglia models. Pervious cerebellum and basal ganglia model, however, treats them as parallel and independent system, without interaction between them, or combines their function together, but with only unidirectional communication between them. Based on the cognition and developmental mechanism of the biology, this paper proposes a novel navigation model, which uses the motivated developmental network (MDN) to mimic the supervised learning of the cerebellum and the reinforcement learning based on the radial basis function neural network (RBFNN) to simulate the reward-based learning of the basal ganglia, and integrates them together to construct a hybrid complex cognition model, to navigate a mobile robot in unknown environment. During the environment exploration, for the unexplored places, the artificial agent uses the cerebellum model to choose action, instead of the ∊-greedy method, to accelerate the learning convergence speed of the basal ganglia. For the explored places, it directly uses the basal ganglia based on the RBFNN to explore, then update and perfect the knowledge base of the cerebellum, which enable the cerebellum to achieve better decision in the following exploration. Hence, it realizes not only the two way communication between the cerebellum and basal ganglia, but also the co-development of them. Experimental results show that this model can enable the agent to autonomously development its intelligence through the hybrid learning. As far as we know, this is the first work to realize the direct communication between the basal ganglia and the cerebellum.

Neuromorphic Hardware Learns to Learn.

Hyperparameters and learning algorithms for neuromorphic hardware are usually chosen by hand to suit a particular task. In contrast, networks of neurons in the brain were optimized through extensive evolutionary and developmental processes to work well on a range of computing and learning tasks. Occasionally this process has been emulated through genetic algorithms, but these require themselves hand-design of their details and tend to provide a limited range of improvements. We employ instead other powerful gradient-free optimization tools, such as cross-entropy methods and evolutionary strategies, in order to port the function of biological optimization processes to neuromorphic hardware. As an example, we show these optimization algorithms enable neuromorphic agents to learn very efficiently from rewards. In particular, meta-plasticity, i.e., the optimization of the learning rule which they use, substantially enhances reward-based learning capability of the hardware. In addition, we demonstrate for the first time Learning-to-Learn benefits from such hardware, in particular, the capability to extract abstract knowledge from prior learning experiences that speeds up the learning of new but related tasks. Learning-to-Learn is especially suited for accelerated neuromorphic hardware, since it makes it feasible to carry out the required very large number of network computations.

Neural signatures of reward and sensory error feedback processing in motor learning.

At least two distinct processes have been identified by which motor commands are adapted according to movement-related feedback: reward-based learning and sensory error-based learning. In sensory error-based learning, mappings between sensory targets and motor commands are recalibrated according to sensory error feedback. In reward-based learning, motor commands are associated with subjective value, such that successful actions are reinforced. We designed two tasks to isolate reward- and sensory error-based motor adaptation, and we used electroencephalography in humans to identify and dissociate the neural correlates of reward and sensory error feedback processing. We designed a visuomotor rotation task to isolate sensory error-based learning that was induced by altered visual feedback of hand position. In a reward learning task, we isolated reward-based learning induced by binary reward feedback that was decoupled from the visual target. A fronto-central event-related potential called the feedback-related negativity (FRN) was elicited specifically by binary reward feedback but not sensory error feedback. A more posterior component called the P300 was evoked by feedback in both tasks. In the visuomotor rotation task, P300 amplitude was increased by sensory error induced by perturbed visual feedback and was correlated with learning rate. In the reward learning task, P300 amplitude was increased by reward relative to nonreward and by surprise regardless of feedback valence. We propose that during motor adaptation the FRN specifically reflects a reward-based learning signal whereas the P300 reflects feedback processing that is related to adaptation more generally. NEW & NOTEWORTHY We studied the event-related potentials evoked by feedback stimuli during motor adaptation tasks that isolate reward- and sensory error-based learning mechanisms. We found that the feedback-related negativity was specifically elicited by binary reward feedback, whereas the P300 was observed in both tasks. These results reveal neural processes associated with different learning mechanisms and elucidate which classes of errors, from a computational standpoint, elicit the feedback-related negativity and P300.

Abstract # 3104 Where is my motivation? – experimentally induced inflammation reduces maintenance of motivated behavior and goal-directed behavior

Fatigue and dysphoria are highly prevalent in patients with chronic medical diseases, characterized by impairments in motivated behavior. Inflammation has been proposed as a biological mechanism, but supporting evidence has mostly been accrued through animal studies and correlational human data, while experimental human studies have remained sparse. Hence, the current study tested the effects of induced inflammation on motivated and goal-directed behavior in humans. This double-blinded placebo-controlled within-subjects study (N = 40 healthy males, mean age = 25 SD = 5) used salmonella typhoid vaccination (0.025 mg; Typhim Vi, Sanofi Pasteur) to induce acute inflammation; saline was used as placebo-control. Participants completed a reward-learning task that assessed distinguishable aspects of motivational learning. These are contingency learning, i.e., association of Abstract symbols with monetary rewards/punishments; maintenance of reward-based responses, and goal-directed adaptation to an unannounced change in reward-contingencies. Vaccination increased IL-6 levels (vaccination + 419%; placebo −10%; p .001), confirming presence of inflammation. While in both vaccine and placebo conditions participants reached the same optimum performance level, vaccinated participants required more trials to reach this optimal level, indicating slower reward-based learning. Moreover, in the inflammation condition participants were less able to maintain optimal performance over successive trials and showed less goal-directed behavior, p’s .05. The current results provide the first human evidence that inflammation impaired motivated learning, specifically reducing stability and maintenance of motivated behaviors, and goal-directed behavior.

Post-encoding frontal theta activity predicts incidental memory in the reward context.

Memories for daily events require that individuals integrate initial fragile traces of events over time. Recent evidence suggests that reward anticipation enhances memory performance and amplifies frontal theta activity for remembered items vs. forgotten items. However, little is known about how incidental rewards after item presentation retrospectively modulate memory and the neural basis of this processing. Here, we used EEG combined with an incidental memory task to study how incidental reward association biased the post-encoding process. In the anticipatory stage, participants saw photos in win, loss and neutral contexts. Each photo was presented in a color frame that indicated the incentive condition (win vs. loss vs. neutral) and participants were asked to make a binary choice to predict whether the photo was associated with the left/right button. Feedback was presented to indicate arbitrary correctness and monetary outcomes. Recognition memory was tested after a short delay. During the encoding phase, left central-parietal theta power predicted subsequent memory performance in the win context. The post-encoding theta power at right central-frontal and central-parietal sites predicted later memory performance only in the win context. The size of frontal post-encoding related theta activity in the win context was correlated with the discriminate accuracy of the test stimulus. Our results suggest that post-encoding theta activity is closely linked to reward-based associative learning, providing evidence of a potential post-encoding mechanism of information binding.

The learning ability and memory retention of broiler breeders: 2 transgenerational effects of reduced balanced protein diet on reward-based learning

The effect of reduced balanced protein (RP) diet in the F0 and F1 generation of broiler breeders on the learning ability and memory retention of the F2 generation was investigated by means of a reward v. no reward discrimination T-maze test. There were two treatments for the F0 generation: control (C) group, reared on standard commercial diets, and reduced balanced protein (RP) group, fed with RP diets (25% reduction in CP and amino acids). The female F0-progeny of each treatment was again separated into the two dietary treatments, resulting in four treatments for the F1 generation: C/C, C/RP, RP/C and RP/RP (breeder feed in F0/F1 generation). The RP diets fed breeders received on average 10% more feed than C diets fed breeders to achieve a similar target BW. The F2 generation was composed of four treatments coming from the female F1-progeny of the four treatments and were all fed with C diet (namely C/C/C, C/RP/C, RP/C/C and RP/RP/C). All four F2 generation groups were able to complete the T-maze learning test with a slight difference in success rate but a significant difference within groups was observed regarding the time needed to complete the test. In general, the RP/RP/C group needed more time for completing the test compared with the other three groups and the shortest time was recorded for the RP/C/C group. At similar ages, breeders with early learning experience spent significantly less time in completing the test compared with unexperienced breeders. Long-term memory retention was observed in all four groups whereas the learning ability in solving the test decreased with age. It took longer for the breeders to complete the test at older ages. In conclusion, under our experimental conditions, the RP dietary treatment in previous generations had no influence on the T-maze learning ability and memory retention of broiler breeders of the third generation, although it might have effects on the working performance in the T-maze learning test of F2 generation breeders.

Closed-Loop Theta Stimulation in Orbitofrontal Cortex Prevents Reward-Based Learning

Neuronal oscillations in frontal cortex have been hypothesized to play a role in the organization of high-level cognition. Within orbitofrontal cortex (OFC), there is a prominent oscillation in the theta frequency (4-8 Hz) during reward-guided behavior, but it is unclear whether this oscillation has causal significance. One methodological challenge is that it is difficult to manipulate theta without affecting other neural signals, such as single neuron firing rates. A potential solution is to use closed-loop control to record theta in real-time and use this signal to control the application of electrical microstimulation to OFC. Using this method, we show that theta oscillations in OFC are critically important for reward-guided learning and that they are driven by theta oscillations in hippocampus (HPC). The ability to disrupt OFC computations via spatially localized and temporally precise stimulation, could lead to novel treatment strategies for neuropsychiatric disorders involving OFC dysfunction.

Orphan Receptor GPR88 as an Emerging Neurotherapeutic Target.

Although G protein-coupled receptors (GPCRs) are recognized as pivotal drug targets involved in multiple physiological and pathological processes, the majority of GPCRs including orphan GPCRs (oGPCRs) are unexploited. GPR88, a brain-specific oGPCR with particularly robust expression in the striatum, regulates diverse brain and behavioral functions, including cognition, mood, movement control, and reward-based learning, and is thus emerging as a novel drug target for central nervous system disorders including schizophrenia, Parkinson's disease, anxiety, and addiction. Nevertheless, no effective GPR88 synthetic ligands have yet entered into clinical trials, and GPR88 endogenous ligands remain unknown. Despite the recent discovery and early stage study of several GPR88 agonists, such as 2-PCCA, RTI-13951-33, and phenylglycinol derivatives, further research into GPR88 pharmacology, medicinal chemistry, and chemical biology is urgently needed to yield structurally diversified GPR88-specific ligands. Drug-like pharmacological tool function and relevant signaling elucidation will also accelerate the evaluation of this receptor as a viable neurotherapeutic target.

On Intelligent Traffic Control for Large-Scale Heterogeneous Networks: A Value Matrix-Based Deep Learning Approach

Recently, deep learning has emerged as an attractive technique to intelligently control network traffic. However, the contemporary researches only focused on small-/medium-scale networks, since the computational complexity of deep learning based traffic control algorithm significantly increases with the network size. In this paper, we address this issue and envision a reward-based deep learning structure, which jointly employs deep convolutional neural network (CNN) and a deep belief network (DBN) to predict the traffic load value matrix and construct the final action matrix, respectively. In our proposal, the deep CNN is used to construct the award prediction network, while the deep DBN constructs the action decision network. Thus, the final action space is simplified to a next destination action matrix, and the computational complexity is substantially reduced. Computer-based simulation results demonstrate that our proposal is able to achieve an improved performance in the large-scale network in terms of the packets loss rate and throughput in contrast with those in the conventional routing method.

Evidence for Cognitive Placebo and Nocebo Effects in Healthy Individuals

Inactive interventions can have significant effects on cognitive performance. Understanding the generation of these cognitive placebo/nocebo effects is crucial for evaluating the cognitive impacts of interventional methods, such as non-invasive brain stimulation (NIBS). We report both cognitive placebo and nocebo effects on reward-based learning performance induced using an active sham NIBS protocol, verbal suggestions and conditioning in 80 healthy participants. Whereas our placebo manipulation increased both expected and perceived cognitive performance, nocebo had a detrimental effect on both. Model-based analysis suggests manipulation-specific strategic adjustments in learning-rates: Participants in the placebo group showed stronger learning from losses and reduced behavioral noise, participants in the nocebo group showed stronger learning from gains and increased behavioral noise. We conclude that experimentally induced expectancy can impact cognitive functions of healthy adult participants. This has important implications for the use of double-blind study designs that can effectively maintain blinding in NIBS studies.

Motor primitives in space and time via targeted gain modulation in cortical networks.

Motor cortex (M1) exhibits a rich repertoire of neuronal activities to support the generation of complex movements. Although recent neuronal-network models capture many qualitative aspects of M1 dynamics, they can generate only a few distinct movements. Additionally, it is unclear how M1 efficiently controls movements over a wide range of shapes and speeds. We demonstrate that modulation of neuronal input–output gains in recurrent neuronal-network models with fixed architecture can dramatically reorganize neuronal activity and thus downstream muscle outputs. Consistent with the observation of diffuse neuromodulatory projections to M1, a relatively small number of modulatory control units provide sufficient flexibility to adjust high-dimensional network activity using a simple reward-based learning rule. Furthermore, it is possible to assemble novel movements from previously learned primitives, and one can separately change movement speed while preserving movement shape. Our results provide a new perspective on the role of modulatory systems in controlling recurrent cortical activity.

A New Look at Visual System Plasticity

Reward-based learning is known to induce cortical plasticity in primary sensory areas. A new study by Goltstein, Meijer, and Pennartz [1] (eLife2018;7:e37683), adopting a dual-scale approach (single-unit and population level), shows how associative learning in mice tunes cortical processing, but unlike other primary sensory cortices it does not modify the retinotopic map.

How the Level of Reward Awareness Changes the Computational and Electrophysiological Signatures of Reinforcement Learning.

The extent to which subjective awareness influences reward processing, and thereby affects future decisions, is currently largely unknown. In the present report, we investigated this question in a reinforcement learning framework, combining perceptual masking, computational modeling, and electroencephalographic recordings (human male and female participants). Our results indicate that degrading the visibility of the reward decreased, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increased their tendency to repeat previous choices. We dissociated electrophysiological signatures evoked by the reward-based learning processes from those elicited by the reward-independent repetition of previous choices and showed that these neural activities were significantly modulated by reward visibility. Overall, this report sheds new light on the neural computations underlying reward-based learning and decision-making and highlights that awareness is beneficial for the trial-by-trial adjustment of decision-making strategies.SIGNIFICANCE STATEMENT The notion of reward is strongly associated with subjective evaluation, related to conscious processes such as "pleasure," "liking," and "wanting." Here we show that degrading reward visibility in a reinforcement learning task decreases, without completely obliterating, the ability of participants to learn from outcomes, but concurrently increases subjects' tendency to repeat previous choices. Electrophysiological recordings, in combination with computational modeling, show that neural activities were significantly modulated by reward visibility. Overall, we dissociate different neural computations underlying reward-based learning and decision-making, which highlights a beneficial role of reward awareness in adjusting decision-making strategies.

Left-shifting prism adaptation boosts reward-based learning

Visuospatial cognition has an inherent lateralized bias. Individual differences in the direction and magnitude of this bias are associated with asymmetrical D2/3 dopamine binding and dopamine system genotypes. Dopamine level affects feedback-based learning and dopamine signaling asymmetry is related to differential learning from reward and punishment. High D2 binding in the left hemisphere is associated with preference for reward. Prism adaptation (PA) is a simple sensorimotor technique, which modulates visuospatial bias according to the direction of the deviation. Left-deviating prism adaptation (LPA) induces rightward bias in healthy subjects. It is therefore possible that the right side of space increases in saliency along with left hemisphere dopaminergic activity. Right-deviating prism adaptation (RPA) has been used mainly as a control condition because it does not modulate behavior in healthy individuals. Since LPA induces a rightward visuospatial bias as a result of left hemisphere modulation, and higher dopaminergic activity in the left hemisphere is associated with preference for rewarding events we hypothesized that LPA would increase the preference for learning with reward. Healthy volunteers performed a computer-based probabilistic classification task before and after LPA or RPA. Consistent with our predictions, PA altered the preference for rewarded versus punished learning, with the LPA group exhibiting increased learning from reward. These results suggest that PA modulates dopaminergic activity in a lateralized fashion.