Reward-based Task Research Articles

Involvement of A13 dopaminergic neurons in prehensile movements but not reward in the rat

Tyrosine hydroxylase (TH)-containing neurons of the dopamine (DA) cell group A13 are well positioned to impact known DA-related functions as their descending projections innervate target regions that regulate vigilance, sensory integration, and motor execution. Despite this connectivity, little is known regarding the functionality of A13-DA circuits. Using TH-specific loss-of-function methodology and techniques to monitor population activity in transgenic rats invivo, we investigated the contribution of A13-DA neurons in reward and movement-related actions. Our work demonstrates a role for A13-DA neurons in grasping and handling of objects but not reward. A13-DA neurons responded strongly when animals grab and manipulate food items, whereas their inactivation or degeneration prevented animals from successfully doing so-a deficit partially attributed to a reduction in grip strength. By contrast, there was no relation between A13-DA activity and food-seeking behavior when animals were tested on a reward-based task that did not include a reaching/grasping response. Motivation for food was unaffected, as goal-directed behavior for food items was in general intact following A13 neuronal inactivation/degeneration. An anatomical investigation confirmed that A13-DA neurons project to the superior colliculus (SC) and also demonstrated a novel A13-DA projection to the reticular formation (RF). These results establish a functional role for A13-DA neurons in prehensile actions that are uncoupled from the motivational factors that contribute to the initiation of forelimb movements and help position A13-DA circuits into the functional framework regarding centrally located DA populations and their ability to coordinate movement.

Temporally specific patterns of neural activity in interconnected corticolimbic structures during reward anticipation

Functional neuroimaging studies indicate that interconnected parts of the subcallosal anterior cingulate cortex (ACC), striatum, and amygdala play a fundamental role in affect in health and disease. Yet, although the patterns of neural activity engaged in the striatum and amygdala during affective processing are well established, especially during reward anticipation, less is known about subcallosal ACC. Here, we recorded neural activity in non-human primate subcallosal ACC and compared this with interconnected parts of the basolateral amygdala and rostromedial striatum while macaque monkeys performed reward-based tasks. Applying multiple analysis approaches, we found that neurons in subcallosal ACC and rostromedial striatum preferentially signal anticipated reward using short bursts of activity that form temporally specific patterns. By contrast, the basolateral amygdala uses a mixture of both temporally specific and more sustained patterns of activity to signal anticipated reward. Thus, dynamic patterns of neural activity across populations of neurons are engaged in affect, especially in subcallosal ACC.

Risky decision-taking task: A novel paradigm to assess the risk-taking behaviour in rats predisposed to early-life stress

One of the characteristic features of adolescence is risk-taking behavioural traits. Uncontrolled risk-taking without proper assessment may have harmful impact on mental health later in life. Therefore, it is essential to identify it early for the preventable health problems. In the present study, we have designed a novel paradigm, viz. Risky Decision-taking Task (RDTT), to evaluate the spontaneous risk-taking behavioural repertoire in adolescent rodents. The task was designed based on both risk and cognitive factors. To validate and compare the risk-taking tendency, we have used early maternal separation and isolation (MS) stress model, as it is known to increase anxiety and curiosity-like behaviour at adolescence. We have used Sprague-Dawley rats of both sexes. Rats were exposed to MS stress for 10 days daily for six hours during stress hyporesponsive period (SHRP) from postnatal day 4–13. These rats were subjected to RDTT during adolescence. This task is a reward-based task where the latency to collect reward in the presence or absence of a risk factor is assessed. It consists of habituation, training to find the location of small and large rewards, reward preference for small and large reward and testing period under risky situation. Rats were trained individually to retrieve the valuation-based rewards under the risky, but innate aversive environments. The results from RDTT showed that as compared to controls, MS rats from both sexes showed reduced latency to collect large reward in the presence of a risk element and a reduced risk-index which is indicative of a higher risk-taking tendency in these rats. In addition, MS rats showed a trend towards anxiety-like behaviour as compared to controls in the Light-Dark Test. These results together show decreased risk latency for the large reward and reduced risk assessment in MS rats which is suggestive of more risk-taking tendency in these rats. Thus, we propose that RDTT paradigm can be used to evaluate the spontaneous risk-taking behavioural repertoire based on innate, spontaneous aversion and cognitive factors in rats.

Slow or sudden: Re-interpreting the learning curve for modern systems neuroscience.

Learning is fundamental to animal survival. Animals must learn to link sensory cues in the environment to actions that lead to reward or avoid punishment. Rapid learning can then be highly adaptive and the difference between life or death. To explore the neural dynamics and circuits that underlie learning, however, has typically required the use of laboratory paradigms with tight control of stimuli, action sets, and outcomes. Learning curves in such reward-based tasks are reported as slow and gradual, with animals often taking hundreds to thousands of trials to reach expert performance. The slow, highly variable, and incremental learning curve remains the largely unchallenged belief in modern systems neuroscience. Here, we provide historical and contemporary evidence that instrumental forms of reward-learning can be dissociated into two parallel processes: knowledge acquisition which is rapid with step-like improvements, and behavioral expression which is slower and more variable. We further propose that this conceptual distinction may allow us to isolate the associative (knowledge-related) and non-associative (performance-related) components that influence learning. We then discuss the implications that this revised understanding of the learning curve has for systems neuroscience.

Motor Synergies Measurement Reveals the Relevant Role of Variability in Reward-Based Learning.

Currently, it is not fully understood how motor variability is regulated to ease of motor learning processes during reward-based tasks. This study aimed to assess the potential relationship between different dimensions of motor variability (i.e., the motor variability structure and the motor synergies variability) and the learning rate in a reward-based task developed using a two-axis force sensor in a computer environment. Forty-four participants performed a pretest, a training period, a posttest, and three retests. They had to release a virtual ball to hit a target using a vertical handle attached to a dynamometer in a computer-simulated reward-based task. The participants’ throwing performance, learning ratio, force applied, variability structure (detrended fluctuation analysis, DFA), and motor synergy variability (good and bad variability ratio, GV/BV) were calculated. Participants with higher initial GV/BV displayed greater performance improvements than those with lower GV/BV. DFA did not show any relationship with the learning ratio. These results suggest that exploring a broader range of successful motor synergy combinations to achieve the task goal can facilitate further learning during reward-based tasks. The evolution of the motor variability synergies as an index of the individuals’ learning stages seems to be supported by our study.

Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimate

In a reward-free environment, what is a suitable intrinsic objective for an agent to pursue so that it can learn an optimal task-agnostic exploration policy? In this paper, we argue that the entropy of the state distribution induced by finite-horizon trajectories is a sensible target. Especially, we present a novel and practical policy-search algorithm, Maximum Entropy POLicy optimization (MEPOL), to learn a policy that maximizes a non-parametric, $k$-nearest neighbors estimate of the state distribution entropy. In contrast to known methods, MEPOL is completely model-free as it requires neither to estimate the state distribution of any policy nor to model transition dynamics. Then, we empirically show that MEPOL allows learning a maximum-entropy exploration policy in high-dimensional, continuous-control domains, and how this policy facilitates learning meaningful reward-based tasks downstream.

Neuronal Representation of Locomotion During Motivated Behavior in the Mouse Anterior Cingulate Cortex.

The anterior cingulate cortex (ACC) is located within the dorsomedial prefrontal cortex (PFC), and processes and facilitates goal-directed behaviors relating to emotion, reward, and motor control. However, it is unclear how ACC neurons dynamically encode motivated behavior during locomotion. In this study, we examined how information for locomotion and behavioral outcomes is temporally represented by individual and ensembles of ACC neurons in mice during a self-paced locomotor reward-based task. By recording and analyzing the activity of ACC neurons with a microdrive tetrode array while the mouse performed the locomotor task, we found that more than two-fifths of the neurons showed phasic activity relating to locomotion or the reward behavior. Some of these neurons showed significant differences in their firing rate depending on the behavioral outcome. Furthermore, by applying a demixed principal component analysis, the ACC population activity was decomposed into components representing locomotion and the previous/future outcome. These results indicated that ACC neurons dynamically integrate motor and behavioral inputs during goal-directed behaviors.

Brain Activity Reveals Multiple Motor-Learning Mechanisms in a Real-World Task.

Many recent studies found signatures of motor learning in neural beta oscillations (13–30 Hz), and specifically in the post-movement beta rebound (PMBR). All these studies were in controlled laboratory-tasks in which the task designed to induce the studied learning mechanism. Interestingly, these studies reported opposing dynamics of the PMBR magnitude over learning for the error-based and reward-based tasks (increase vs. decrease, respectively). Here, we explored the PMBR dynamics during real-world motor-skill-learning in a billiards task using mobile-brain-imaging. Our EEG recordings highlight the opposing dynamics of PMBR magnitudes (increase vs. decrease) between different subjects performing the same task. The groups of subjects, defined by their neural dynamics, also showed behavioral differences expected for different learning mechanisms. Our results suggest that when faced with the complexity of the real-world different subjects might use different learning mechanisms for the same complex task. We speculate that all subjects combine multi-modal mechanisms of learning, but different subjects have different predominant learning mechanisms.

Persisting on the past: Cross-sectional and prospective associations between sunk cost propensity and cannabis use.

Prevalence of cannabis use in the United States continues to rise, and 30% of cannabis users eventually meet criteria for Cannabis Use Disorder (CUD). One response to this problem is to develop decision-making constructs that indicate vulnerability to CUD that might not be gleaned from diagnostic criteria. Unfortunately, there is limited evidence that decision-making constructs consistently relate to cannabis use. Interestingly, those who exhibit the sunk cost bias, an overgeneralized tendency to persist based on past investment, and those who use cannabis, both tend to focus on the past and perseverate more than their counterparts. Despite this overlap, no studies have assessed whether the sunk cost bias is positively associated with cannabis use. In 2 experiments with undergraduates, relations between cannabis use and the propensity to engage in the sunk cost bias were examined using negative binomial models. Experiment 1 (n = 46) evaluated the association between sunk cost bias propensity (using hypothetical costs and rewards) and frequency of cannabis use over the past 30 days. Greater sunk cost propensity was associated with more frequent cannabis use after controlling for demographics and alcohol use. In Experiment 2 (n = 103), more frequent cannabis use during a 6-week follow-up period was predicted by greater sunk cost propensity at baseline (using a real cost and reward-based task), independently and after controlling for mental health symptoms, alcohol use, and demographics. These findings provide preliminary evidence that a propensity to exhibit the sunk cost bias may be an important feature associated with cannabis use. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Human Brain Imaging Links Dopaminergic Systems to Impulsivity.

Molecular and functional imaging techniques have been used and combined with pharmacological probes to evaluate the role of dopamine in impulsivity. Overall, strong evidence links striatal dopaminergic function with impulsivity, measured by self-reports and laboratory tests of cognitive control and reward-based decision-making. The combination of molecular imaging using positron emission tomography (PET) with functional magnetic resonance imaging (fMRI) specifically implicates striatal D2-type dopamine receptors (i.e., D2 and D3) and corticostriatal connectivity in cognitive control. Low levels of striatal and midbrain D2-type receptor availability correlate with self-reported impulsivity, whereas striatal D2-type receptor availability shows positive correlation with motor response inhibition and cognitive flexibility. Impulsive choice on reward-based decision-making tasks also is related to deficits in striatal D2-type dopamine receptor availability, and there is evidence for an inverted U-shaped function in this relationship, reflecting an optimum of striatal dopaminergic activity. Findings from studies of clinical populations that present striatal dopamine D2-type receptor deficits as well as healthy control research participants identify D2-type receptors as therapeutic targets to improve cognitive control.

Functional inhibitory control dynamics in impulse control disorders in Parkinson's disease.

Impulse control disorders related to alterations in the mesocorticolimbic dopamine network occur in Parkinson's disease (PD). Our objective was to investigate the functional neural substrates of reward processing and inhibitory control in these patients. Eighteen PD patients with impulse control disorders, 17 without this complication, and 18 healthy controls performed a version of the Iowa Gambling Task during functional magnetic resonance scanning under 3 conditions: positive, negative, and mixed feedback. Whole-brain contrasts, regions of interest, time courses, functional connectivity analyses, and brain-behavior associations were examined. PD patients with impulse control disorders exhibited hyperactivation in subcortical and cortical regions typically associated with reward processing and inhibitory control compared with their PD and healthy control counterparts. Time-course analyses revealed that only PD patients with impulse control disorders exhibited stronger signal intensity during the initial versus final periods of the negative-feedback condition in bilateral insula, and right ventral striatum. Interestingly, hyperactivation of all the examined right-lateralized frontostriatal areas during negative feedback was positively associated with impulse control disorder severity. Importantly, positive associations between impulse control disorder severity and regional activations in the right insula and right inferior frontal gyrus, but not the right subthalamic nucleus, were mediated by functional connectivity with the right ventral striatum. During a reward-based task, PD patients with impulse control disorders showed hyperactivation in a right-lateralized network of regions including the subthalamic nucleus that was strongly associated with impulse control disorder severity. In these patients, the right ventral striatum in particular played a critical role in modulating the functional dynamics of right-lateralized inhibitory-control frontal regions when facing penalties. © 2019 International Parkinson and Movement Disorder Society.

Cognitive training improves the disturbed behavioral architecture of schizophrenia-like rats, “Wisket”

Translational schizophrenia research depends on the relevance of animal models supported by reliable tests. Human data suggest that the intensive cognitive training in schizophrenia improves the memory impairments and decreases the chance of acute psychiatric remission. Here we examined the effects of a 10-day long training session in the behavioral architecture of a new schizophrenia-like rat substrain (Wisket) in a narrow square corridor with food rewards (AMBITUS). The instrument was designed to model the natural environment of rats and enable the simultaneous recording of multiple behavioral parameters. For the compact visualization of differences between the Wisket and control animals in several parameters (behavioromics), color-coded grid plots were applied. The Wisket animals exhibited an altered pattern and/or amount of locomotion, exploratory and food collecting activity at the first few days, revealing impaired motivation, attention, anxiety and learning ability (face validity). Most of the parameters normalized with training, except for the decreased exploratory activity. This resembles the effects of cognitive behavioral therapy in human schizophrenics providing a significant support for the predictive validity of this substrain as an animal model of schizophrenia. This study also highlights the importance of behavior tests that investigate the egocentric learning ability during reward-based tasks.

Data from ‘Placebo Enhances Reward Learning in Healthy Individuals’

This dataset contains three repeated measures of a standard reward-based reinforcement-learning task from 29 healthy male individuals who participated in three experimental sessions exploring cognitive placebo effects on reward learning. The dataset includes behavioural data (accuracy, reaction times) during learning and transfer, estimates of model-free computational analysis, self-reported arousal values, and expectations about the interventions’ efficacy. The data were collected in 2014 at the Department of Clinical Neurophysiology, University Medical Center Goettingen, Germany. The data collection and formal analysis used a triple-blind study design as participants, operator and analyst were unaware of conditions. A github repository contains data and analyses for the paper “Placebo Intervention Enhances Reward Learning in Healthy Individuals”. The dataset can be used for further analysis, reference, validation studies, teaching purposes, and collaborative research. Funding statement: This study was supported by the DFG (PA 419/15-1) awarded to WP. The preparation of this manuscript was supported by the “Research program, University Medical Center, University of Goettingen” awarded to Z.T.

Effectiveness of Reward-Based Task on Affective Levels of Depressed Individuals

Effectiveness of Reward-Based Task on Affective Levels of Depressed Individuals

Model-based and model-free pain avoidance learning.

Background: While there is good evidence that reward learning is underpinned by two distinct decision control systems – a cognitive ‘model-based’ and a habitbased ‘model-free’ system, a comparable distinction for punishment avoidance has been much less clear. Methods: We implemented a pain avoidance task that placed differential emphasis on putative model-based and model-free processing, mirroring a paradigm and modelling approach recently developed for reward-based decision-making. Subjects performed a two-step decision-making task with probabilistic pain outcomes of different quantities. The delivery of outcomes was sometimes contingent on a rule signalled at the beginning of each trial, emulating a form of outcome devaluation. Results: The behavioural data showed that subjects tended to use a mixed strategy – favouring the simpler model-free learning strategy when outcomes did not depend on the rule, and favouring a model-based when they did. Furthermore, the data were well described by a dynamic transition model between the two controllers. When compared with data from a reward-based task (albeit tested in the context of the scanner), we observed that avoidance involved a significantly greater tendency for subjects to switch between model-free and model-based systems in the face of changes in uncertainty. Conclusion: Our study suggests a dual-system model of pain avoidance, similar to but possibly more dynamically flexible than reward-based decision-making.

144 Prefrontal Neurons Modulate Motor Behavior by Targeting Distinct Mediolateral Cortical Sites

Abstract INTRODUCTION Injury to the prefrontal cortex (PFC) can result in maladaptive and disinhibited behavior. However, the neural basis for behavioral control of the prefrontal cortex remains largely unknown. Here, we explored the role of the dorsolateral PFC (dlPFC) in orchestrating motor behavior by conducting simultaneous, invasive recordings of the DLPFC, supplementary motor area (SMA), and dorsal premotor (PMd) in primates. METHODS Cortical surface microarrays were implanted into the dlPFC, SMA, and PMd of two monkeys who then participated in a reward-based motor task. For each trial, a monkey received visual instructional cues correlating to a two-step joystick movement plan. They then received cues to either initiate or withhold each step of the plan. Coherence and Granger causality (GC) analysis of the local field potential (LFP) data was used to characterize the interactions between the cortical sites during various behavioral scenarios (initiation, withholding, continuing, or aborting an initiated motor task). RESULTS >Theta band (3-7 Hz) coherence activity was found to most greatly distinguish the four behavioral scenarios. Initiation and continuation cues were associated with increased dlPFC-SMA and dlPFC-PMd coherence (t-test, P < 10e-12), but a more significant increase was seen in dlPFC-SMA compared to dlPFC-PMd (ANOVA, P < 0.001). Inhibition of movement initiation was characterized by dlPFC-SMA coherence increase but lack of dlPFC-PMd coherence change (t-test, P = 0.9). Aborting an already initiated movement sequence was associated with a global decrement in coherence (t-test, P < 10e-35). GC analysis demonstrated that these coherence changes were generally associated with an increase of information flow from the PFC to the more distal mediolateral frontal sites. CONCLUSION We discovered two functional circuitries between the pre-frontal and pre-motor cortices that distinctly control initiation and inhibition of motor behavior. These findings provide an important circuit-based model on which to understand and prospectively treat neuro-cognitive disorders characterized by disinhibition and maladaptation.

Parsing Heterogeneous Striatal Activity.

The striatum is an input channel of the basal ganglia and is well known to be involved in reward-based decision making and learning. At the macroscopic level, the striatum has been postulated to contain parallel functional modules, each of which includes neurons that perform similar computations to support selection of appropriate actions for different task contexts. At the single-neuron level, however, recent studies in monkeys and rodents have revealed heterogeneity in neuronal activity even within restricted modules of the striatum. Looking for generality in the complex striatal activity patterns, here we briefly survey several types of striatal activity, focusing on their usefulness for mediating behaviors. In particular, we focus on two types of behavioral tasks: reward-based tasks that use salient sensory cues and manipulate outcomes associated with the cues; and perceptual decision tasks that manipulate the quality of noisy sensory cues and associate all correct decisions with the same outcome. Guided by previous insights on the modular organization and general selection-related functions of the basal ganglia, we relate striatal activity patterns on these tasks to two types of computations: implementation of selection and evaluation. We suggest that a parsing with the selection/evaluation categories encourages a focus on the functional commonalities revealed by studies with different animal models and behavioral tasks, instead of a focus on aspects of striatal activity that may be specific to a particular task setting. We then highlight several questions in the selection-evaluation framework for future explorations.

Brain activity and infant attachment history in young men during loss and reward processing.

There is now ample evidence that the quality of early attachment experiences shapes expectations for supportive and responsive care and ultimately serves to scaffold adaptation to the salient tasks of development. Nonetheless, few studies have identified neural mechanisms that might give rise to these associations. Using a moderately large sample of low-income male participants recruited during infancy (N = 171), we studied the predictive significance of attachment insecurity and disorganization at age 18 months (as measured in the Strange Situation Procedure) for patterns of neural activation to reward and loss at age 20 years (assessed during a reward-based task as part of a functional magnetic resonance imaging scan). Results indicated that individuals with a history of insecure attachment showed hyperactivity in (a) reward- and emotion-related (e.g., basal ganglia and amygdala) structures and (b) emotion regulation and self-referential processing (cortical midline structures) in response to positive and negative outcomes (and anticipation of those outcomes). Further, the neural activation of individuals with a history of disorganized attachment suggested that they had greater emotional reactivity in anticipation of reward and employed greater cognitive control when negative outcomes were encountered. Overall, results suggest that the quality of early attachments has lasting impacts on brain function and reward processing.

The Role of Dopamine D1 and D3 Receptors in N-Methyl-D-Aspartate (NMDA)/GlycineB Site-Regulated Complex Cognitive Behaviors following Repeated Morphine Administration.

Background:Opiate addiction is associated with complex cognitive impairment, which contributes to the development of compulsive drug use and relapses. Dopamine and N-methyl-D-aspartate receptors play critical roles in opiate-induced cognitive deficits. However, the roles of D1 and D3 receptors in the N-methyl-D-aspartate/glycineB receptor-regulated cognitive behaviors induced by morphine remain unknown.Methods:The 5-choice serial reaction time task was used to investigate the cognitive profiles associated with repeated morphine administration in D1 (D1-/-)- and D3 (D3-/-)-receptor knockout mice. The expression of phosphorylated NR1, Ca2+/calmodulin-dependent protein kinase II (CaMKII), and cAMP response element-binding protein (CREB) in the brain was examined by western blotting. D1-/- and D3-/- mice were treated with the N-methyl-D-aspartate/glycineB site agonist l-aminocyclopropanecarboxylic acid and the antagonist L-701,324 to chronically disrupt N-methyl-D-aspartate receptor function and investigate their effects on morphine-induced cognitive changes.Results:Repeated morphine administration impaired attentional function and caused impulsive and compulsive behaviors. D1-/- mice exhibited hardly any premature nosepokes. D3-/- mice showed robustly increased morphine-induced impulsive behavior. The numbers of premature responses were decreased by L-701,324 administration and increased by ACPC administration; these effects were completely abolished in D1-/- mice due to their inability to perform reward-based tasks. In contrast, the inhibitory effects of L-701,324 on impulsive behavior were significantly augmented in D3-/- mice.Conclusions:N-methyl-D-aspartate/glycineB site functions may contribute to morphine-induced cognitive deficits, especially those related to impulsive behavior. D1 and D3 receptors may have contrasting effects with respect to modulating impulsive behavior. D3 receptors have inhibitory effects on impulsive behaviors, and these effects are clearly mediated by N-methyl-D-aspartate/glycineB receptor and μ-opioid receptor interactions.

Can the structure of motor variability predict learning rate?

Recent studies show that motor variability is actively regulated as an exploration tool to promote learning in reward-based tasks. However, its role in learning processes during error-based tasks, when a reduction of the motor variability is required to achieve good performance, is still unclear. In this study, we hypothesized that error-based learning not only depends on exploration but also on the individuals' ability to measure and predict the motor error. Previous studies identified a less auto-correlated motor variability as a higher ability to perform motion adjustments. Two experiments investigated the relationship between motor learning and variability, analyzing the long-range autocorrelation of the center of pressure fluctuations through the α score of a Detrended Fluctuation Analysis in balance tasks. In Experiment 1, we assessed the relationship between variability and learning rate using a standing balance task. Based on the results of this experiment, and to maximize learning, we performed a second experiment with a more difficult sitting balance task and increased practice. The learning rate of the 2 groups with similar balance performances but different α scores was compared. Individuals with a lower α score showed a higher learning rate. Because the α scores reveal how the motor output changes over time, instead of the magnitude of those changes, the higher learning rate is mainly linked to the higher error sensitivity rather than the exploration strategies. The results of this study highlight the relevance of the structure of output motor variability as a predictor of learning rate in error-based tasks. (PsycINFO Database Record