Delay Of Reward Research Articles

The Dynamic Response of Dual Cellular-Connected UAVs for Random Real-Time Communication Requests from Multiple Hotspots: A Deep Reinforcement Learning Approach

It is gradually becoming popular to use multiple cellular-connected UAVs as inspectors to fulfill automatic surveillance and monitoring. However, in actual situations, UAVs should respond to several service requests from different hotspots, whilst the requests usually present randomness in the arrival time, data amount, and the concurrency. This paper proposes a dynamic dual-UAV response policy for multi-hotspot services based on single-agent deep Q-learning, where the UAVs controlled by a ground base station can be dispatched automatically to hotspots and then send videos back. First, this issue is formulated as an optimization problem, whose goal is to maximize the number of successfully served requests with the constraints of both the UAV’s energy limit and request waiting time. Second, a reward function based on service completion is designed to overcome the potential challenges posed by the delay reward. Finally, a simulation was conducted, comparing the conventional time priority algorithm and distance priority algorithm, respectively, to the proposed algorithm. The results illustrate that the proposed algorithm can achieve one more response than the others under different service densities, with the lowest failure number and appropriate average waiting time. This method can give a technical solution for the joint communication-and-control problem of multiple UAVs within complex situations.

The differential effect of optogenetic serotonergic manipulation on sustained motor actions and waiting for future rewards in mice.

Serotonin is an essential neuromodulator that affects behavioral and cognitive functions. Previous studies have shown that activation of serotonergic neurons in the dorsal raphe nucleus (DRN) promotes patience to wait for future rewards. However, it is still unclear whether serotonergic neurons also regulate persistence to act for future rewards. Here we used optogenetic activation and inhibition of DRN serotonergic neurons to examine their effects on sustained motor actions for future rewards. We trained mice to perform waiting and repeated lever-pressing tasks with variable reward delays and tested effects of optogenetic activation and inhibition of DRN serotonergic neurons on task performance. Interestingly, in the lever-pressing task, mice tolerated longer delays as they repeatedly pressed a lever than in the waiting task, suggesting that lever-pressing actions may not simply be costly, but may also be subjectively rewarding. Optogenetic activation of DRN serotonergic neurons prolonged waiting duration in the waiting task, consistent with previous studies. However, its effect on lever presses was nuanced, and was detected only by focusing on the period before premature reward check and by subtracting the trends within and across sessions using generalized linear model. While optogenetic inhibition decreased waiting, it did not affect lever pressing time or numbers. These results revealed that the necessity of motor actions may increase motivation for delayed rewards and that DRN serotonergic neurons more significantly promote waiting rather than persistent motor actions for future rewards.

Influence of amount and delay of reward on choice and response rate: A free-operant, multiple-schedule analogue of a discrete-trial procedure.

The current study explored a free-operant analogue of discrete-trial procedures to study the effects of amount and delay of reinforcement on choice and response rate. Rats responded on a multiple variable-interval (VI) 45-s, 45-s schedule, with interspersed choice probe trials. Comparison of relative response rates and percentage of choice revealed some discrepancies between the free-operant analogue and discrete-trial procedures. Amount of reward controlled choice behavior when the ratios of delays were similar. When reward delays were more discrepant, delay length controlled choice behavior. Whereas the percentage of choice was larger for the larger magnitude reward, the relative rate of response for the larger magnitude was less than .50. In contrast, when the percentage of choice generally fell to below 50% (with large amount and large delay differences between alternatives), relative response rate indicated a preference for the larger amount alternative. This study shows the feasibility and utility of a free-operant analogue of discrete-choice studies that could be used to develop an analysis of preference.

Cortico-striatal beta oscillations as a reward-related signal

The value associated with reward is sensitive to external factors, such as the time between the choice and reward delivery as classically manipulated in temporal discounting tasks. Subjective preference for two reward options is dependent on objective variables of reward magnitude and reward delay. Single neuron correlates of reward value have been observed in regions, including ventral striatum, orbital, and medial prefrontal cortex. Brain imaging studies show cortico-striatal-limbic network activity related to subjective preferences. To explore how oscillatory dynamics represent reward processing across brain regions, we measured local field potentials of rats performing a temporal discounting task. Our goal was to use a data-driven approach to identify an electrophysiological marker that correlates with reward preference. We found that reward-locked oscillations at beta frequencies signaled the magnitude of reward and decayed with longer temporal delays. Electrodes in orbitofrontal/medial prefrontal cortex, anterior insula, ventral striatum, and amygdala individually increased power and were functionally connected at beta frequencies during reward outcome. Beta power during reward outcome correlated with subjective value as defined by a computational model fit to the discounting behavior. These data suggest that cortico-striatal beta oscillations are a reward signal correlated, which may represent subjective value and hold potential to serve as a biomarker and potential therapeutic target.

Subthalamic control of impulsive actions: insights from deep brain stimulation in Parkinson's disease.

Control of actions allows adaptive, goal-directed behaviour. The basal ganglia, including the subthalamic nucleus, are thought to play a central role in dynamically controlling actions through recurrent negative feedback loops with the cerebral cortex. Here, we summarize recent translational studies that used deep brain stimulation to record neural activity from and apply electrical stimulation to the subthalamic nucleus in people with Parkinson's disease. These studies have elucidated spatial, spectral and temporal features of the neural mechanisms underlying the controlled delay of actions in cortico-subthalamic networks and demonstrated their causal effects on behaviour in distinct processing windows. While these mechanisms have been conceptualized as control signals for suppressing impulsive response tendencies in conflict tasks and as decision threshold adjustments in value-based and perceptual decisions, we propose a common framework linking decision-making, cognition and movement. Within this framework subthalamic deep brain stimulation can lead to suboptimal choices by reducing the time that patients take for deliberation before committing to an action. However, clinical studies have consistently shown that the occurrence of impulse control disorders is reduced, not increased, after subthalamic deep brain stimulation surgery. This apparent contradiction can be reconciled when recognizing the multifaceted nature of impulsivity, its underlying mechanisms and modulation by treatment. While subthalamic deep brain stimulation renders patients susceptible to making decisions without proper forethought, this can be disentangled from effects related to dopamine comprising sensitivity to benefits vs. costs, reward delay aversion and learning from outcomes. Alterations in these dopamine-mediated mechanisms are thought to underlie the development of impulse control disorders, and can be relatively spared with reduced dopaminergic medication after subthalamic deep brain stimulation. Together, results from studies using deep brain stimulation as an experimental tool have improved our understanding of action control in the human brain and have important implications for treatment of patients with Neurological disorders.

Maze runners: monkeys show restricted Arabic numeral summation during computerized two-arm maze performance

Mazes have been used in many forms to provide compelling results showcasing nonhuman animals’ capacities for spatial navigation, planning, and numerical competence. The current study presented computerized two-arm mazes to four rhesus macaques. Using these mazes, we assessed whether the monkeys could maximize rewards by overcoming mild delays in gratification and sum the values of Arabic numerals. Across four test phases, monkeys used a joystick controller to choose one of two maze arms on the screen. Each maze arm contained zero, one or two Arabic numerals, and any numerals in the chosen maze arm provided the monkeys with rewards equivalent to the value of those numerals. When deciding which arm to enter, monkeys had to consider distance to numerals and numeral value. In some tests, gaining the maximum reward required summing the value of two numerals within a given arm. All four monkeys successfully maximized reward when comparing single numerals and when comparing arms that each contained two numerals. However, some biases occurred that were suboptimal: the largest single numeral and the delay of reward (by placing numerals farther into an arm from the start location) sometimes interfered with the monkeys’ abilities to optimize. These results indicate that monkeys experience difficulties with inhibition toward single, high valence stimuli in tasks where those stimuli must be considered in relation to overall value when represented by symbolic stimuli such as numerals.

Memristive Neural Network Circuit of Operant Conditioning With Reward Delay and Variable Punishment Intensity

Memristive Neural Network Circuit of Operant Conditioning With Reward Delay and Variable Punishment Intensity

Pavlovian-to-instrumental transfer in intertemporal choice

Abstract We often forego a larger future reward in order to obtain a smaller reward immediately, known as impatient intertemporal choice. The current study investigated the role of Pavlovian-to-instrumental transfer (PIT) as a mechanism contributing to impatient intertemporal choice, following a theoretical framework proposing that cues associated with immediate gratification trigger a Pavlovian approach response, interfering with goal-directed (instrumental) inhibitory behavior. We developed a paradigm in which participants first learned to make instrumental go/no-go responses in order to win rewards and avoid punishments. Next, they learned the associations between Pavlovian cues and rewards varying in amount and delay. Finally, we tested whether these (task-irrelevant) cues exerted transfer effects by influencing instrumental actions while participants again completed the go/no-go task. Across two experiments, Pavlovian cues associated with larger (versus smaller) and immediate (versus delayed) rewards were evaluated more positively, reflecting the successful acquisition of Pavlovian cue–outcome associations. These findings replicated the previously reported classical transfer effect of reward amount on instrumental behavior, as large (versus smaller) cues increased instrumental approach. In contrast, we found no evidence for the hypothesized transfer effects for reward delay, contrary to the proposed role of PIT in impatient intertemporal choice. These results suggest that although both reward amount and delay were important in the evaluation of cues, only the amount associated with cues influenced instrumental choice. We provide concrete suggestions for future studies, addressing instrumental outcome identity, competition between cue–amount and cue–delay associations, and individual differences in response to Pavlovian cues.

Pain catastrophizing is associated with reduced neural response to monetary reward.

Pain catastrophizing, a measure of an individual's negative emotional and cognitive appraisals of pain, has been included as a key treatment target in many psychological interventions for pain. However, the neural correlates of pain catastrophizing have been understudied. Prior neuroimaging evidence suggests that adults with pain show altered reward processing throughout the mesocorticolimbic reward circuitry. In this study, we tested the association between Pain Catastrophizing Scale (PCS) scores and neural activation to the Monetary Incentive Delay (MID) reward neuroimaging task in 94 adults reporting a range of pain, insomnia, and mood symptoms. Results indicated that PCS score but not pain intensity was significantly associated with blunted activation in the caudate and putamen in response to feedback of successful vs. unsuccessful trials on the MID task. Mediation analyses indicated that PCS score fully mediated the relationship between depression symptoms and reward activation. These findings provide evidence that pain catastrophizing is independently associated with altered striatal function apart from depression symptoms and pain intensity. Thus, in individuals experiencing pain and/or co- morbid conditions, reward dysfunction is directly related to pain catastrophizing.

Reward delays quitting in visual search.

Reward motivates goal-directed behaviors, leading to faster reaction time (RT) and lower error rate in searching for a target in the reward condition than in the no-reward condition in target-discrimination tasks. However, it is unclear how reward influences target detection in which participants are required to judge whether a predesignated target is present or absent. Here, we asked participants to complete a target-detection search task in which the color of the search array indicated the reward availability of the current trial. Correct and faster (than a baseline) responses would be rewarded if the search array had the reward-related color. In Experiments 1A and 1B, the target was presented in 50% of the trials. Experiment 1B had the same design as Experiment 1A, except that different baselines were set for the target-present and target-absent conditions. In Experiment 2, the proportion of target presence was manipulated to be high (80%), moderate (50%), or low (20%) in different blocks of stimuli. Results showed that, across all the experiments, participants responded faster and made fewer errors in the reward than in the no-reward condition when the target was present. However, this facilitatory effect was reversed when the target was absent, showcasing a reward-induced interference. The signal detection analysis suggested that reward biased the report criterion to the "yes" response. These findings demonstrate that the impact of reward on goal-directed behavior can be detrimental and reward prolongs the search process by rendering participants reluctant to say "no" in visual search termination.

Architecture and relationships among cognition, mental health and other human domains revealed by network analysis perspective

Despite the connectome or network “era”, the variety of individual facets (e.g., cognition, personality, mental health) are typically studied as separate watertight compartments. This study adopted advanced methodological approaches to (i) highlight a network depicting distinct domains of human mind and behavior, and to (ii) delineate their possible direct influences. Through an Exploratory Graph Analysis on the Human Connectome Project’s database, we found a network composed by seven separate but related domains - Mental Health, Externalizing problems, High-level Cognitive Functions, Basic Cognitive Functions, Substances use/abuse, Reward Delay Discounting and Pain - unveiling a low dimensionality of human mind and behavior. Finally, by applying Bayesian Networks on this model, we observed that some facets have also specific and oriented relationships, while other facets are instead independent from each other. In such a way, we drew a potential architecture of individual’s facets that can pave the way for future neuroimaging studies and clinical practice.

Learning the Policy for Mixed Electric Platoon Control of Automated and Human-Driven Vehicles at Signalized Intersection: A Random Search Approach

The upgrading and updating of vehicles have accelerated in the past decades. Out of the need for environmental friendliness and intelligence, electric vehicles (EVs) and connected and automated vehicles (CAVs) have become new components of transportation systems. This paper develops a reinforcement learning framework to implement adaptive control for an electric platoon composed of CAVs and human-driven vehicles (HDVs) at a signalized intersection. Firstly, a Markov Decision Process (MDP) model is proposed to describe the decision process of the mixed platoon. Novel state representation and reward function are designed for the model to consider the behavior of the whole platoon. Secondly, in order to deal with the delayed reward, an Augmented Random Search (ARS) algorithm is proposed. The control policy learned by the agent can guide the longitudinal motion of the CAV, which serves as the leader of the platoon. Finally, a series of simulations are carried out in simulation suite SUMO. Compared with several state-of-the-art (SOTA) reinforcement learning approaches, the proposed method can obtain a higher reward. Meanwhile, the simulation results demonstrate the effectiveness of the delay reward, which is designed to outperform distributed reward mechanism} Compared with normal car-following behavior, the sensitivity analysis reveals that the energy can be saved to different extends (39.27%-82.51%) by adjusting the relative importance of the optimization goal. On the premise that travel delay is not sacrificed, the proposed control method can save up to 53.64% electric energy.

Conditioned approach behavior of SHR and SD rats during Pavlovian conditioning

Individual differences in reward-related learning are relevant to many behavioral disorders. Sensory cues that predict reward can become incentive stimuli that adaptively support behavior, or alternatively, cause maladaptive behaviors. The spontaneously hypertensive rat (SHR) expresses a genetically determined elevated sensitivity to delay of reward, and has been extensively studied as a behavioral model for attention deficit hyperactivity disorder (ADHD). We investigated reward-related learning in the SHR, comparing them to Sprague-Dawley (SD) rats as a reference strain. A standard Pavlovian conditioned approach task was used, in which a lever cue was followed by reward. Lever presses could occur while the lever was extended, but had no effect on reward delivery. The behavior of both the SHRs and the SD rats showed that they learnt that the lever cue predicted reward. However, the pattern of behavior differed between the strains. During lever cue presentation, SD rats pressed the lever more often and made fewer magazine entries than SHRs. When lever contacts that did not result in lever presses were analyzed, there was no significant difference between SHRs and SDs. These results suggest that the SHRs attributed less incentive value to the conditioned stimulus than the SD rats. During the presentation of the conditioned cue, cue directed responses are called sign tracking responses, whereas responses directed towards the food magazine are called goal tracking responses. Analysis of behavior using a standard Pavlovian conditioned approach index to quantify sign and goal tracking tendencies showed that both strains had a tendency towards goal tracking in this task. However, the SHRs showed a significantly greater goal tracking tendency than the SD rats. Taken together, these findings suggest that attribution of incentive value to reward predicting cues is attenuated in SHRs, which might explain their elevated sensitivity to delay of reward.

Large-scale brain correlates of sweet versus cocaine reward in rats.

Cocaine induces many supranormal changes in neuronal activity in the brain, notably in learning- and reward-related regions, in comparison with nondrug rewards-a difference that is thought to contribute to its relatively high addictive potential. However, when facing a choice between cocaine and a nondrug reward (e.g., water sweetened with saccharin), most rats do not choose cocaine, as one would expect from the extent and magnitude of its global activation of the brain, but instead choose the nondrug option. We recently showed that cocaine, though larger in magnitude, is also an inherently more delayed reward than sweet water, thereby explaining why it has less value during choice and why rats opt for the more immediate nondrug option. Here, we used a large-scale Fos brain mapping approach to measure brain responses to each option in saccharin-preferring rats, with the hope to identify brain regions whose activity may explain the preference for the nondrug option. In total, Fos expression was measured in 142 brain levels corresponding to 52 brain subregions and composing 5 brain macrosystems. Overall, our findings confirm in rats with a preference for saccharin that cocaine induces more global brain activation than the preferred nondrug option does. Only very few brain regions were uniquely activated by saccharin. They included regions involved in taste processing (i.e., anterior gustatory cortex) and also regions involved in processing reward delay and intertemporal choice (i.e., some components of the septohippocampal system and its connections with the lateral habenula).

A Rapid Assessment of Sensitivity to Reward Delays and Classwide Token Economy Savings for School-Aged Children.

Delay discounting tasks measure the relation between reinforcer delay and efficacy. The present study established the association between delay discounting and classroom behavior and introduced a brief measure quantifying sensitivity to reward delays for school-aged children. Study 1 reanalyzed data collected by Reed and Martens (J Appl Behav Anal 44(1):1-18, https://doi.org/10.1901/jaba.2011.44-1, 2011) and found that 1-month delay choices predicted student classroom behavior. Study 2 examined the utility of the 1-month delay indifference point in predicting saving and spending behavior of second-grade students using token economies with two different token production schedules. Collectively, results showed (a) the 1-month delay indifference point predicted classroom behavior, (b) children who discounted less and had greater self-regulation, accrued and saved more tokens, and (c) a variable token production schedule better correlated with discounting than a fixed schedule. Implications are discussed regarding utility of a rapid discounting assessment for applied use.

Discounting of Future Rewards and Punishments in Rats.

Temporal reward discounting describes the decrease of value of a reward as a function of delay. Decision-making between future aversive outcomes is much less studied, and there is no clear decision pattern across studies; while some authors suggest that human and nonhuman animals prefer sooner over later painful shocks, others found the exact opposite. In a series of three experiments, Long-Evans rats chose between differently timed electric shocks and rewards in a T-maze. In experiment 1, rats chose between early and late painful shocks with identical, long reward delays; in experiment 2, they chose between early reward and early shocks, or late rewards and late shocks; in experiment 3, they chose between early and late rewards, with identical, short delays to the shock. We tested the predictions of two competing hypotheses: the aversive discounting theory assumes that future shocks are discounted, and, hence, less unpleasant than early shocks. The utility from anticipation theory implies that rats derive negative utility from waiting for the shock; late shocks should, hence, be more unpleasant than early shocks. We did not find unanimous evidence for either theory. Instead, our results are more consistent with the post hoc idea that shocks may have negative spill-over effects on reward values, the closer in time a shock is to a subsequent reward, the stronger the reward is devalued. Interestingly and consistent with our theory, we find that, depending on the temporal shock-reward contiguity, rats can be brought to prefer later over sooner rewards of identical magnitudes.

Internally Triggered Experiences of Hedonic Valence in Nonhuman Animals: Cognitive and Welfare Considerations

Do any nonhuman animals have hedonically valenced experiences not directly caused by stimuli in their current environment? Do they, like us humans, experience anticipated or previously experienced pains and pleasures as respectively painful and pleasurable? We review evidence from comparative neuroscience about hippocampus-dependent simulation in relation to this question. Hippocampal sharp-wave ripples and theta oscillations have been found to instantiate previous and anticipated experiences. These hippocampal activations coordinate with neural reward and fear centers as well as sensory and cortical areas in ways that are associated with conscious episodic mental imagery in humans. Moreover, such hippocampal “re- and preplay” has been found to contribute to instrumental decision making, the learning of value representations, and the delay of rewards in rats. The functional and structural features of hippocampal simulation are highly conserved across mammals. This evidence makes it reasonable to assume that internally triggered experiences of hedonic valence (IHVs) are pervasive across (at least) all mammals. This conclusion has important welfare implications. Most prominently, IHVs act as a kind of “welfare multiplier” through which the welfare impacts of any given experience of pain or pleasure are increased through each future retrieval. However, IHVs also have practical implications for welfare assessment and cause prioritization.

Increment in the consummatory response induced by reward delay: An animal model of binge-like eating episodes

The omission of an expected palatable reinforcer produces an aversive emotional state, called frustration, that could function as an important factor in the etiology and maintenance of the binge-eating disorder. A series of experiments carried out in adult male Wistar rats are presented, in which the consummatory response of sugary drinks was evaluated after events of frustration due to reward delay. The animals were trained in a consummatory behavior acquisition phase, in which they had access to different concentrations of sucrose solutions. In the test phase, a control group received the reinforcer normally, while an experimental group received it after a delay. A significant increase in the consummatory behavior of a 32 % sucrose was found in the group that was exposed to a 2-min delay before re-introducing the reinforcer (Experiment 1). The increment found in the consummatory response was dependent on the aversive state of frustration. This could be concluded because, in a less frustrating condition, the animals did not exhibit the phenomenon (Experiment 2), and a more frustrating one produced an increment of the effect (Experiment 3). This result was also observed using a direct measure of consumption -kilocalorie intake- during several events of reinforcement delay (Experiment 4).

Optimal control of renewable energy in buildings using the machine learning method

In this paper, the optimal control of renewable energy in the green building using a powerful reinforcement learning control method is presented to minimize energy consumption and power losses in the distribution sector. Solar energy for heating, solar photovoltaic cells, and biomass boilers is considered the main renewable energy source. The control system is determined based on the energy sources usage of the building and what is stored for later use or injected into the environment to satisfy the requirement of renewable energy along with minimum use of other energies. The optimization is represented in the central management system in the building with a novel system which is called here Building Energy Enhancement Learning (BEEL). Optimized BEEL is maximized the numerical reward of the signal by the represented reinforcement learning algorithm and compare with a hybrid supporting vector-wavelet learning algorithm. The new method reveals the corresponding actions are based on the reward and have higher compliance than the other represented machine learning methods. Two unique features of reinforcement learning compared to artificial intelligence methods are trial and error and delay reward, which is done here with 99.98% accuracy.

Impulsivity, Depressive Mood, and Cannabis Use in a Representative Sample of French-Speaking Swiss Young Men.

Cannabis is the most popular psychoactive substance under international regulations, with more than 192 million users worldwide. It has been associated with an addictive pattern of use and negative social and health-related outcomes in a subgroup of users. Consequently, understanding the individual differences that contribute to cannabis use and problematic use is of much importance. The current study examined the impact of impulsivity traits (negative urgency, positive urgency, lack of premeditation, lack of perseverance, sensation seeking), delay reward discounting, and depressive mood on cannabis use status during the past 6 months as well as problematic use of cannabis in a representative sample of 635 French-speaking Swiss young men recruited during their conscription in a Swiss national military recruitment center. Binary logistic and multiple linear regressions indicated that cannabis use status was significantly associated with greater depressive mood, elevated sensation seeking, and lack of perseverance, whereas problematic cannabis use was significantly related to higher depressive mood and steeper delay reward discounting. The present study highlights the importance of emotional symptoms in cannabis use and misuse. Our results also shed light on the potential psychological processes related to problematic consumption of cannabis and open avenues for preventive actions and psychological interventions that target problematic use of cannabis.