Delay Of Reward Research Articles

Reward timing matters in motor learning

SummaryReward timing, that is, the delay after which reward is delivered following an action is known to strongly influence reinforcement learning. Here, we asked if reward timing could also modulate how people learn and consolidate new motor skills. In 60 healthy participants, we found that delaying reward delivery by a few seconds influenced motor learning. Indeed, training with a short reward delay (1 s) induced continuous improvements in performance, whereas a long reward delay (6 s) led to initially high learning rates that were followed by an early plateau in the learning curve and a lower performance at the end of training. Participants who learned the skill with a long reward delay also exhibited reduced overnight memory consolidation. Overall, our data show that reward timing affects the dynamics and consolidation of motor learning, a finding that could be exploited in future rehabilitation programs.

Investigating the associations between irritability and hot and cool executive functioning in those with ADHD

BackgroundIrritability is especially pertinent to those with Attention Deficit Hyperactivity Disorder (ADHD) as it is highly prevalent and associated with a more severe clinical presentation and poorer longitudinal outcomes. Preliminary evidence suggests that top-down cognitive processes taking place in emotional contexts (i.e., hot executive functions) as opposed to those evoked in abstract scenarios (i.e., cool executive functions) may be relevant to the presentation of irritability in ADHD. This study explored the cognitive mechanisms underlying irritability in young people with ADHD, hypothesising that irritability would be associated with hot, but not cool, executive function impairments.MethodsOur sample included 219 individuals with ADHD. A composite irritability score was derived extracting items from a parent interview, with scores ranging from 0 to 5. Associations were investigated using linear regression analyses, between irritability and four hot tasks measuring sensitivity to risk, risk-taking behaviour following reward or punishment, acceptance of reward delay and reaction to unfair behaviour from others, and two cool tasks measuring set-shifting and motor inhibition.ResultsAs hypothesised, there were no significant associations between irritability and cool executive functions in those with ADHD; however, contrary to expectations, there was also no significant evidence that hot executive functions were associated with irritability.ConclusionsThese results, in a large well characterised sample and using a comprehensive task battery, suggest that the variation in irritability in those with ADHD may not be associated with differences in hot or cool executive function performance.

Effects of emotion on intertemporal decision-making: Explanation from the single dimension priority model

<p id="p00005">Studies show that significant differences are found in intertemporal choice between people in positive and negative emotions. However, the mechanism underlying this effect is underexplored. This study aimed to reveal the process mechanism of emotions on intertemporal decision making on the basis of the single dimension priority model by using a process-oriented method-a visual analog scale. <break/>We conducted three studies, where emotions were induced by video clip-induced technique to trigger target emotions (positive, neutral and negative). In study 1, the effects of emotions on intertemporal choice was investigated. This study used a 3 (emotion: positive, neutral, negative) × 2 (delay amount: 100, 1000 yuan) × 5 (delay time: 10, 30, 90, 180, 360 days) mixed design with emotion as a between-subjects variable. Participants were randomly assigned to positive, neutral, and negative conditions and were asked to complete the delay discounting task. In study 2, the process mechanism of emotions on intertemporal choice with a between-subjects design (emotion: positive, negative) was explored. Participants in each condition were asked to complete the delay discounting task and to compare the difference in the delay dimension with that on the outcome dimension. In studies 3a and 3b, 2 (emotion: positive, negative) × 2 (priming: time priming/money priming, no priming) between-subjects designs were used to manipulate the difference-comparing process and explore whether the effect of emotions on intertemporal choice disappeared. <break/>The results were observed as follows: (1) Emotions affected intertemporal choice significantly. Participants in the positive condition preferred the delay reward more than participants in the negative and neutral condition. (2) The difference-comparing process played a mediating role in the relationship between emotion and intertemporal decision making. Participants in positive emotion perceived that the difference in outcome dimension was greater than that of delay dimension and preferred the delay option. Participants in negative emotions perceived opposite and tended to choose the immediate choice. (3) The effect of emotions on intertemporal decision making disappeared when the difference-comparing process was manipulated. <break/>In summary, this study revealed the psychological mechanism of emotions on intertemporal decision making from the perspective of decision-making process and added important procedural evidence to the single dimension priority model.

Are You Sure: Preference and Ambivalence in Delay Discounting.

Delay discounting (DD) research has become ubiquitous due to its robust associations with clinical outcomes. Typical DD tasks involve multiple trials in which participants indicate preference between smaller, sooner and larger, later rewards. Scoring of these binary choice tasks has not considered trial-level ambivalence as a possible decision-making construct. The present study explored the extent to which trial-level ambivalence varied within-individual using an established assessment of DD (the Monetary Choice Questionnaire). Results indicate that degree of ambivalence peaks around the trials associated with the DD rate. Moreover, ambivalence is associated with a diminished impact of reward delay differences on choice, where greater delay differences decrease the odds of choosing the larger, later rewards. Taken together, we believe ambivalence to be a relevant construct for research on intertemporal decision making, and it may be particularly useful in the study of manipulations on individual rates of DD.

Addressing anhedonia to increase depression treatment engagement.

Anhedonia, or reward system dysfunction, is associated with poorer treatment outcomes among depressed individuals. The role of anhedonia in treatment engagement, however, has not yet been explored. We review research on components of reward functioning impaired in depression, including effort valuation, reward anticipation, initial responsiveness, reward learning, reward probability, and reward delay, highlighting potential barriers to treatment engagement associated with these components. We then propose interventions to improve treatment initiation and continuation by addressing deficits in each component of reward functioning, focusing on modifications of existing evidence-based interventions to meet the needs of individuals with heightened anhedonia. We describe potential settings for these interventions and times at which they can be delivered during the process of referring individuals to mental health treatment, conducting intakes or assessments, and providing treatment. Additionally, we note the advantages of using screening processes already in place in primary care, workplace, school, and online settings to identify individuals with heightened anhedonia who may benefit from these interventions. We conclude with suggestions for future research on the impact of anhedonia on treatment engagement and the efficacy of interventions to address it. PRACTITIONER POINTS: Many depressed individuals who might benefit from treatment do not initiate it or discontinue early. One barrier to treatment engagement may be anhedonia, a core symptom of depression characterized by loss of interest or pleasure in usual activities. We describe brief interventions to improve treatment engagement in individuals with anhedonia that can be implemented during the referral process or early in treatment. We argue that interventions aiming to improve treatment engagement in depressed individuals that target anhedonia may be particularly effective.

Efficient Learning of Goal-Oriented Push-Grasping Synergy in Clutter

We focus on the task of goal-oriented grasping, in which a robot is supposed to grasp a pre-assigned goal object in clutter and needs some pre-grasp actions such as pushes to enable stable grasps. However, in this task, the robot gets positive rewards from environment only when successfully grasping the goal object. Besides, joint pushing and grasping elongates the action sequence, compounding the problem of reward delay. Thus, sample inefficiency remains a main challenge in this task. In this letter, a goal-conditioned hierarchical reinforcement learning formulation with high sample efficiency is proposed to learn a push-grasping policy for grasping a specific object in clutter. In our work, sample efficiency is improved by two means. First, we use a goal-conditioned mechanism by goal relabeling to enrich the replay buffer. Second, the pushing and grasping policies are respectively regarded as a generator and a discriminator and the pushing policy is trained with supervision of the grasping discriminator, thus densifying pushing rewards. To deal with the problem of distribution mismatch caused by different training settings of two policies, an alternating training stage is added to learn pushing and grasping in turn. A series of experiments carried out in simulation and real world indicate that our method can quickly learn effective pushing and grasping policies and outperforms existing methods in task completion rate and goal grasp success rate by less times of motion. Furthermore, we validate that our system can also adapt to goal-agnostic conditions with better performance. Note that our system can be transferred to the real world without any fine-tuning. Our code is available at https://github.com/xukechun/Efficient_goal-oriented_push-grasping_synergy

The protective effect of operant social reward on cocaine self-administration, choice, and relapse is dependent on delay and effort for the social reward.

Social reinforcement-based treatments are effective for many, but not all, people with addictions to drugs. We recently developed an operant rat model that mimics features of one such treatment, the community-reinforcement approach. In this model, rats uniformly choose social interaction over methamphetamine or heroin. Abstinence induced by social preference protects against the incubation of drug-seeking that would emerge during forced abstinence. Here, we determined whether these findings generalize to cocaine and whether delaying or increasing effort for social interaction could reveal possibly human-relevant individual differences in responsiveness. We trained male and female rats for social self-administration (6 days) and then for cocaine self-administration, initially for 2-h/day for 4 days, and then for 12-h/day continuously or intermittently for 8 days. We assessed relapse to cocaine seeking after 1 and 15 days. Between tests, the rats underwent either forced abstinence or social-choice-induced abstinence. After establishing stable social preference, we manipulated the delay for both rewards or for social reward alone, or the response requirements (effort) for social reward. Independent of cocaine-access conditions and sex, operant social interaction inhibited cocaine self-administration and prevented incubation of cocaine seeking. Preference for social access was decreased by the delay of both rewards or social reward alone, or by increased response requirements for social reward, with notable individual variability. This choice procedure can identify mechanisms of individual differences in an animal model of cocaine use and could thereby help screen medications for people who are relatively unresponsive to treatments based on rewarding social interaction.

Cognitive characterization of adult attention deficit hyperactivity disorder by domains: a systematic review.

Attention deficit hyperactivity disorder (ADHD) is reportedly the most frequent neurodevelopmental disorder diagnosed during childhood, and it is recognized as a common condition in adulthood. We review the evidence to help identify cognitive domains associated to deficits in adult ADHD. A systematic review with narrative synthesis was performed, assessing studies on adult ADHD, neuropsychology and research on involved cognitive domains in adults 18+ years old with an established diagnosis of ADHD, in seven electronic databases (PubMed, PsychInfo, WebOfScience, Embase, Scopus, OvidSPMedline, and Teseo), and Worldcat and OpenGrey grey literature databases. 93 studies were included for this review, encompassing findings from a total 5574 adults diagnosed only with ADHD, medication-naïve or non-medicated at the moment of the assessment and 4880 healthy controls. Adults diagnosed with ADHD may show, when compared to healthy controls, a cognitive profile characterized by deficits across all attention modalities, processing speed, executive function (mainly working memory and inhibition with emphasis on reward delay and interference control), verbal memory, reading skills, social cognition and arithmetic abilities. A cognitive characterization of adult ADHD by domains is established beyond the sole consideration of attention and executive function problems. Along with these, verbal memory, language (mainly reading), social cognition and arithmetic abilities may also contribute to a more comprehensive characterization of the cognitive profile in adult ADHD.

An examination of the acquired preparedness model among college student marijuana users

Objective To examine the Acquired Preparedness Model using a behavioral impulsivity facet and positive marijuana expectancies to examine direct and indirect effects on marijuana use and related problems. Participants: 250 college students (61.7% female, 54% white) recruited from a southeastern university. Methods: Participants completed an online survey of delay reward discounting, marijuana expectancies, consideration of future consequences, and marijuana-related outcomes. Results: Delay reward discounting and consideration of future consequences related to marijuana-related problems, but not marijuana use. However, positive marijuana expectancies did not mediate the relation between impulsivity and marijuana outcomes. Conclusions: These results emphasize delay reward discounting and consideration of future consequences as important factors associated with marijuana-related problems. Interventions aimed at decreasing delay reward discounting and augmenting future orientation may be effective in college students who report light to moderate marijuana use. Future studies would benefit from longitudinal study designs using multiple impulsivity measures among light and heavy users.

Hypothalamic-Extended Amygdala Circuit Regulates Temporal Discounting.

Choice behavior is characterized by temporal discounting, i.e., preference for immediate rewards given a choice between immediate and delayed rewards. Agouti-related peptide (AgRP)-expressing neurons located in the arcuate nucleus of the hypothalamus (ARC) regulate food intake and energy homeostasis, yet whether AgRP neurons influence choice behavior and temporal discounting is unknown. Here, we demonstrate that motivational state potently modulates temporal discounting. Hungry mice (both male and female) strongly preferred immediate food rewards, yet sated mice were largely indifferent to reward delay. More importantly, selective optogenetic activation of AgRP-expressing neurons or their axon terminals within the posterior bed nucleus of stria terminalis (BNST) produced temporal discounting in sated mice. Furthermore, activation of neuropeptide Y (NPY) type 1 receptors (Y1Rs) within the BNST is sufficient to produce temporal discounting. These results demonstrate a profound influence of hypothalamic signaling on temporal discounting for food rewards and reveal a novel circuit that determine choice behavior.SIGNIFICANCE STATEMENT Temporal discounting is a universal phenomenon found in many species, yet the underlying neurocircuit mechanisms are still poorly understood. Our results revealed a novel neural pathway from agouti-related peptide (AgRP) neurons in the hypothalamus to the bed nucleus of stria terminalis (BNST) that regulates temporal discounting in decision-making.

Reward Timing Matters in Motor Learning

Reward can improve motor learning and the consolidation of motor memories. Identifying the features of reward feedback that are critical for motor learning is a necessary step for successful integration into rehabilitation programs. One central feature of reward feedback that may affect motor learning is its timing – that is, the delay after which reward is delivered following movement execution. In fact, research on associative learning has shown that short and long reward delays (e.g., 1 and 6 s following action execution) activate preferentially the striatum and the hippocampus, respectively, which both contribute with varying degrees to motor learning. Given the distinct functional role of these two areas, we hypothesized that reward timing could modulate how people learn and consolidate a new motor skill. In sixty healthy participants, we found that delaying reward delivery by a few seconds influenced motor learning dynamics. Indeed, training with a short reward delay (i.e., 1 s) induced slow, yet continuous gains in performance, while a long reward delay (,i>i.e., 6 s) led to initially high learning rates that were followed by an early plateau in the learning curve and a lower endpoint performance. Moreover, participants who successfully learned the skill with a short reward delay displayed overnight consolidation, while those who trained with a long reward delay exhibited an impairment in the consolidation of the motor memory. Overall, our data show that reward timing affects motor learning, potentially by modulating the engagement of different learning processes, a finding that could be exploited in future rehabilitation programs.

Reinforcer pathology II: Reward magnitude, reward delay, and demand for alcohol collectively relate to college students' alcohol related problems.

The reinforcer pathologies model of addiction posits that two characteristic patterns of operant behavior characterize addiction. Specifically, individuals suffering from addiction have elevated levels of behavioral economic demand for their substances of abuse and have an elevated tendency to devalue delayed rewards (reflected in high delay discounting rates). Prior research has demonstrated that these behavioral economic markers are significant predictors of many of college students' alcohol-related problems. Delay discounting, however, is a complex behavioral performance likely undergirded by multiple behavioral processes. Emerging analytical approaches have isolated the role of participants' sensitivity to changes in reinforcer magnitude and changes in reinforcer delay. The current study uses these analytic approaches to compare participants' discounting of money versus alcohol, and to build regression models that leverage these new insights to predict a wider range of college students' alcohol related problems. Using these techniques, we were able to 1) demonstrate that individuals differed in their sensitivity to magnitudes of alcohol versus money, but not sensitivity to delays to those commodities and 2) that we could use our behavioral economic measures to predict a range of students' alcohol related problems.

Increasing pool funds in public goods: The effects of deposit-based delayed rewards

• The deposit system is introduced into the public goods game. • The delay is linked to the rank of contributions, which better promotes the increase of individuals’ contributions. • This model not only promotes long-term cooperation, but also increases the pool reward fund. • One proper combination of deposit, reward and delay can realize the best cooperation effect. The inspiration for this paper comes from the deposit-based delayed rewards that Mobike company used in its initial promotion. This paper goes to investigate the viability of this method on increasing pool funds in the supply of public goods. We take into account different deposit, reward value and delay time of reward, and take the fraction of cooperators, the sum of contributions and the net profit of organizers as indicators of cooperation success. Our results show that the lower deposit, higher reward or shorter delay time encourages fast cooperation. Furthermore, proper matching of these three factors helps the organizers earn higher profits. Through this approach, organizers have more funds for rewarding from deposits and the delayed-rewards. And the delay of rewards based on the players’ contribution rankings can improve the donations of players. For players, the cost of getting the reward is small because the deposit can be refundable. To sum up, the deposit-based delayed rewards mechanism complements various ways to promote cooperation.

A single-player Monte Carlo tree search method combined with node importance for virtual network embedding

As a critical technology in network virtualization, virtual network embedding (VNE) focuses on how to allocate physical resources to virtual network requests efficiently. Because the VNE problem is NP-hard, most of the existing approaches are heuristic-based algorithms that tend to converge to a local optimal solution and have a low performance. In this paper, we propose an algorithm that combines the basic Monte Carlo tree search (MCTS) method with node importance to apply domain-specific knowledge. For a virtual network request, we first model the embedding process as a finite Markov decision process (MDP), where each virtual node is embedded in one state in the order of node importance that we design. The shortest-path algorithm is then applied to embed links in the terminal state and return the cost as a part of the reward. Due to the reward delay mechanism of the MDP, the result of link mapping can affect the action selected in the previous node mapping stage, coordinating the two embedding stages. With node importance, domain-specific knowledge can be used in the Expansion and Simulation stages of MCTS to speed up the search and estimate the simulation value more accurately. The experimental results show that, compared with the existing classic algorithms, our proposed algorithm can improve the performance of VNE in terms of the average physical node utilization ratio, acceptance ratio, and long-term revenue to cost ratio.

The disengaging brain: Dynamic transitions from cognitive engagement and alcoholism risk

Human functional brain connectivity is usually measured either at “rest” or during cognitive tasks, ignoring life’s moments of mental transition. We propose a different approach to understanding brain network transitions. We applied a novel independent component analysis of functional connectivity during motor inhibition (stop signal task) and during the continuous transition to an immediately ensuing rest. A functional network reconfiguration process emerged that: (i) was most prominent in those without familial alcoholism risk, (ii) encompassed brain areas engaged by the task, yet (iii) appeared only transiently after task cessation. The pattern was not present in a pre-task rest scan or in the remaining minutes of post-task rest. Finally, this transient network reconfiguration related to a key behavioral trait of addiction risk: reward delay discounting. These novel findings illustrate how dynamic brain functional reconfiguration during normally unstudied periods of cognitive transition might reflect addiction vulnerability, and potentially other forms of brain dysfunction.

Naloxone-precipitated withdrawal ameliorates impairment of cost-benefit decision making in morphine-treated rats: Involvement of BDNF, p-GSK3-β, and p-CREB in the amygdala

Several studies indicated that morphine administration impairs cognitive brain functions. Therefore, in the current study, we investigated the effect of subchronic exposure to morphine and its withdrawal on effort- and/or delay-based forms of cost-benefit decision making and alterations in p-CREB/CREB ratio, p-GSK3β/GSK3β ratio, and BDNF level during decision making in the amygdala. Our data displayed an impairment of both forms of cost-benefit decision making following subchronic exposure to morphine. However, preference of high reward/high effort and/or high delay reward increased after naloxone injection. In molecular section, levels of BDNF and p-CREB/CREB ratio increased during cost-benefit decision making while p-GSK3β/GSK3β ratio decreased in both forms of decision making. In morphine-treated rats, level of BDNF and p-CREB/CREB ratio reduced during both forms of decision making while p-GSK3β/GSK3β ratio increased during delay-based and did not have a significant difference with the control group during effort-based decision making. On the withdrawal day, BDNF level raised while p-GSK3β/GSK3β ratio attenuated compared to morphine-treated group in both form of decision making. In addition, p-CREB/CREB ratio increased only during delay-based decision making on the withdrawal day. In conclusion, our data revealed that subchronic exposure to morphine interferes with the cost-benefit decision making may be via changes in level of BDNF, p-CREB/CREB and p-GSK3β/GSK3β ratio in the amygdala.

Adaptive early classification of temporal sequences using deep reinforcement learning

In this article, we address the problem of early classification (EC) of temporal sequences with adaptive prediction times. We frame EC as a sequential decision making problem and we define a partially observable Markov decision process (POMDP) fitting the competitive objectives of classification earliness and accuracy. We solve the POMDP by training an agent for EC with deep reinforcement learning (DRL). The agent learns to make adaptive decisions between classifying incomplete sequences now or delaying its prediction to gather more measurements. We adapt an existing DRL algorithm for batch and online learning of the agent’s action value function with a deep neural network. We propose strategies of prioritized sampling, prioritized storing and random episode initialization to address the fact that the agent’s memory is unbalanced due to (1): all but one of its actions terminate the process and thus (2): actions of classification are less frequent than the action of delay. In experiments, we show improvements in accuracy induced by our specific adaptation of the algorithm used for online learning of the agent’s action value function. Moreover, we compare two definitions of the POMDP based on delay reward shaping against reward discounting. Finally, we demonstrate that a static naive deep neural network, i.e. trained to classify at static times, is less efficient in terms of accuracy against speed than the equivalent network trained with adaptive decision making capabilities.

Contributions of anterior cingulate cortex and basolateral amygdala to decision confidence and learning under uncertainty

The subjective sense of certainty, or confidence, in ambiguous sensory cues can alter the interpretation of reward feedback and facilitate learning. We trained rats to report the orientation of ambiguous visual stimuli according to a spatial stimulus-response rule that must be learned. Following choice, rats could wait a self-timed delay for reward or initiate a new trial. Waiting times increase with discrimination accuracy, demonstrating that this measure can be used as a proxy for confidence. Chemogenetic silencing of BLA shortens waiting times overall whereas ACC inhibition renders waiting times insensitive to confidence-modulating attributes of visual stimuli, suggesting contribution of ACC but not BLA to confidence computations. Subsequent reversal learning is enhanced by confidence. Both ACC and BLA inhibition block this enhancement but via differential adjustments in learning strategies and consistent use of learned rules. Altogether, we demonstrate dissociable roles for ACC and BLA in transmitting confidence and learning under uncertainty.

To eat or not to eat: Reward delay impulsivity in children with loss of control eating, attention deficit / hyperactivity disorder, a double diagnosis, and healthy children

Reward delay impulsivity is a feature of attention deficit/hyperactivity disorder (ADHD) and a likely feature of loss of control eating (LOC-E), which might explain the higher risk of children with ADHD or LOC-E to become obese. The goal of this study was to investigate reward delay impulsivity in children with LOC-E, ADHD, or a double diagnosis, in contrast to healthy children. Children (8 to 13 years) with LOC-E (n = 24), ADHD (n = 33), a double diagnosis (n = 9), and healthy children (n = 34) performed a computer game (door opening task [DOT]) and the delay of gratification task (DoGT) to assess food related facets of reward delay impulsivity. In addition, children reported whether they worried to lose control over eating during the DoGT. There were no group differences in the DOT. However, children with ADHD or a double diagnosis had a significantly higher risk to eat prematurely during the DoGT than children with LOC-E, who were not significantly different from healthy children. Children with a double diagnosis were most likely to worry about losing control over eating during the DoGT, followed by children with LOC-E, and both had a significantly higher probability to worry than healthy children. For children with a double diagnosis the probability to worry was significantly higher than for children with ADHD. If replicated, these findings point to a special relevance of reward delay impulsivity in children with ADHD or a double diagnosis, compared to children with LOC-E. ADHD should be regularly assessed in children with LOC-E.

Normative Data for Executive Function and Emotion Interaction Assessment in Adolescents from Rural Areas

Abstract Objective The present work is part of the ongoing project of examining the validation and standardization of the Executive Functions and Emotion Assessment for Adolescents with Risk Behaviors (EFEAA) test in the rural areas of Northwest Mexico. The objectives of the present study were (1) to examine the effects of age and risk behavior on emotionally interfered working memory ability, measured as performance on the Emotionally Interfered Working Memory Task (EIWMT) section of the EFEAA in a group of healthy adolescents from rural areas; and (2) to use these results to construct normative data that can be used in clinical practice. Participants and Method A total of 275 adolescents, 14.1 years old, geographically distributed among northwest Mexico were included in the sample. They were individually assessed using the Montreal Cognitive Assessment, Risk Behavior Questionnaire, including: sexual risk behavior, impulsiveness, drug, alcohol, and tobacco consumption, unhealthy food intake, and physical inactivity; for measuring emotion-cognition interaction we used the EIWMT. Results Results of this study present a database supported by factors of risk behavior and working memory, which includes: predictors for conduct disorders, drug use, and delinquency, in addition to correlations between the performance of emotionally interfered working memory and risk behaviors associated to health. Percentile and normal distribution tables were generated, demonstrating that there is a high prevalence of risk behavior in rural areas. Conclusions This study demonstrates data related to rural areas in northwest Mexico, a region where adolescents are frequently involved in risk behaviors. Additionally, protective factors are understudied in Mexico-US border cities. Further analyses are needed to find reward delay skills on the studied sample as well as urban area comparisons.