Delay Of Reward Research Articles

Neural Hyporesponsiveness and Hyperresponsiveness During Immediate and Delayed Reward Processing in Adult Attention-Deficit/Hyperactivity Disorder

Dysfunctional reward processing, accompanied by a limited ability to tolerate reward delays, has been proposed as an important feature in attention-deficit/hyperactivity disorder (ADHD). Using functional magnetic resonance imaging (fMRI), brain activation in adult patients with ADHD (n=14) and healthy control subjects (n=12) was examined during a series of choices between two monetary reward options that varied by delay to delivery. Compared with healthy control subjects, hyporesponsiveness of the ventral-striatal reward system was replicated in patients with ADHD and was evident for both immediate and delayed rewards. In contrast, delayed rewards evoked hyperactivation in dorsal caudate nucleus and amygdala of ADHD patients. In both structures, neural activity toward delayed rewards was significantly correlated with self-rated ADHD symptom severity. The finding of ventral-striatal hyporesponsiveness during immediate and delayed reward processing in patients with ADHD further strengthens the concept of a diminished neural processing of rewards in ADHD. Hyperactivation during delayed reward processing, gradually increasing along the ventral-to-dorsal extension of the caudate nucleus, and especially the concomitant hyperactivation of the amygdala are in accordance with predictions of the delay aversion hypothesis.

Influence of Reward Delays on Responses of Dopamine Neurons

Psychological and microeconomic studies have shown that outcome values are discounted by imposed delays. The effect, called temporal discounting, is demonstrated typically by choice preferences for sooner smaller rewards over later larger rewards. However, it is unclear whether temporal discounting occurs during the decision process when differently delayed reward outcomes are compared or during predictions of reward delays by pavlovian conditioned stimuli without choice. To address this issue, we investigated the temporal discounting behavior in a choice situation and studied the effects of reward delay on the value signals of dopamine neurons. The choice behavior confirmed hyperbolic discounting of reward value by delays on the order of seconds. Reward delay reduced the responses of dopamine neurons to pavlovian conditioned stimuli according to a hyperbolic decay function similar to that observed in choice behavior. Moreover, the stimulus responses increased with larger reward magnitudes, suggesting that both delay and magnitude constituted viable components of dopamine value signals. In contrast, dopamine responses to the reward itself increased with longer delays, possibly reflecting temporal uncertainty and partial learning. These dopamine reward value signals might serve as useful inputs for brain mechanisms involved in economic choices between delayed rewards.

Learning and planning in environments with delayed feedback

This work considers the problems of learning and planning in Markovian environments with constant observation and reward delays. We provide a hardness result for the general planning problem and positive results for several special cases with deterministic or otherwise constrained dynamics. We present an algorithm, Model Based Simulation, for planning in such environments and use model-based reinforcement learning to extend this approach to the learning setting in both finite and continuous environments. Empirical comparisons show this algorithm holds significant advantages over others for decision making in delayed-observation environments.

Simultaneous and sequential choice as a function of reward delay and magnitude: Normative, descriptive and process-based models tested in the European starling (Sturnus vulgaris).

The authors analyze prevalent theoretical and empirical quantitative models of choice using data from European starlings (Sturnus vulgaris). The different models originate from diverse lines of thinking, including optimal foraging theory, descriptive behavioral analysis, and temporal discounting. The authors also investigate the Sequential Choice Model (SCM) that predicts choice between simultaneous alternatives as a function of behavior in sequential encounters. SCM assumes that simultaneous encounters are rare in nature, where animals often exploit or dismiss single opportunities. Mechanisms of choice adapted for sequential encounters may be evolutionarily stable even if they impose a cost in the rarer simultaneous cases. The best predictive power is achieved by the SCM and by hyperbolic discounting with value (or attractiveness) of each option proportional to reward amount and inversely proportional to the sum of delay to reward plus delay to respond. Choice probability matched the ratio of each option's value to the sum of values of all available options. The good fit of SCM supports the view that choice is driven by mechanisms adapted to sequential, rather than simultaneous encounters with feeding opportunities.

Effects of mGlu1 receptor blockade on working memory, time estimation, and impulsivity in rats

Metabotropic glutamate 1 (mGlu1) receptor antagonists were reported to induce cognitive deficits in several animal models using aversive learning procedures. The present study aimed to further characterize behavioral effects of mGlu1 receptor antagonists using appetitively motivated tasks that evaluate working memory, timing, and impulsivity functions. Separate groups of adult male Wistar rats were trained to perform four food-reinforced operant tasks: delayed non-matching to position (DNMTP), differential reinforcement of low rates of responding 18 s (DRL 18-s), signal duration discrimination (2-s vs 8-s bisection), and tolerance to delay of reward. Before the tests, rats were pretreated with (3-ethyl-2-methyl-quinolin-6-yl)-(4-methoxy-cyclohexyl)-methanone methanesulfonate (EMQMCM; 2.5-10 mg/kg, i.p.; JNJ16567083). In DNMTP task, EMQMCM produced delay-dependent increases in performance accuracy so that, at 10 mg/kg dose level, percentage of correct lever choices was enhanced at 8- and 16-s delays. In DRL task, at all three tested doses, response rates were higher, and reinforcement rates were lower than under control conditions. In signal duration discrimination tasks, EMQMCM did not have any specific effects on temporal control. In tolerance to delay of reward, EMQMCM (5 and 10 mg/kg) facilitated choice of the lever associated with large reward at longer delay levels. Blockade of mGlu1 receptors improves working memory and reduces impulsive choice at the doses that have no effects on time perception but appear to facilitate impulsive action.

Normative and descriptive models of decision making: time discounting and risk sensitivity.

The task of evolutionary psychologists is to produce precise predictions about psychological mechanisms using adaptationist thinking. This can be done combining normative models derived from evolutionary hypotheses with descriptive regularities across species found by experimental psychologists and behavioural ecologists. I discuss two examples. In temporal discounting, a normative model (exponential) fails while a descriptive one (hyperbolic) fits both human and non-human data. In non-humans hyperbolic discounting coincides with rate of gain maximization in repetitive choices. Humans may discount hyperbolically in non-repetitive choices because they treat them as a repetitive rate-maximizing problem. In risk sensitivity, a theory derived from fitness considerations produces inconclusive results in non-humans, but succeeds in predicting human risk proneness and risk aversion for both the amount and delay of reward in a computer game. Strikingly, and in contrast with the existing literature, risk aversion for delay occurs as predicted. The predictions of risk aversion for delay may fail in many animal experiments because the manipulations of the utility function are not appropriate. In temporal discounting animal experiments help the interpretation of human results, while in risk sensitivity studies human results help the analysis of non-human data.

Methylphenidate reduces impulsive behaviour in juvenile Wistar rats, but not in adult Wistar, SHR and WKY rats

Impulsivity is a core symptom of attention deficit/hyperactivity disorder (ADHD). The spontaneously hypertensive rats (SHR) is a strain commonly used as an animal model of ADHD. However, there is no clear evidence that psychostimulants, which are used for treatment of ADHD, reduce impulsivity in SHR. Because ADHD mainly affects children, it may be relevant to study psychostimulants on juvenile animals. Using tolerance to delay of reward as index of impulsivity, the effects of methylphenidate were assessed in adult SHR, Wistar Kyoto (WKY) and Wistar rats and in juvenile Wistar rats. Animals were trained in a T-maze to choose between a small-but-immediate and a large-but-delayed reward. Adult SHR, WKY and Wistar rats were compared for their ability to tolerate a 15-s delay. The effect of methylphenidate on the tolerance to a 30-s delay was studied in adult rats of the three strains and in juvenile (4.5 to 6.5-week-old) Wistar rats. In adult rats, the waiting ability was lower in SHR than in control strains. Waiting ability was improved by methylphenidate (3 and 5 mg/kg) in juveniles, but not by methylphenidate (3 mg/kg) in adults. These data support the idea that SHR are more impulsive than control strains. However, at the dose studied, methylphenidate fails to improve tolerance to delay in adult rats whatever the strain used. The reduction of impulsivity induced by methylphenidate in juvenile Wistar rats indicates that juvenile animals may be suitable for testing the therapeutic potential of drugs intended to the treatment of ADHD in children.

Activity of serotonergic neurons in the dorsal raphe nucleus of freely moving rats during reward and non-reward delay period

Activity of serotonergic neurons in the dorsal raphe nucleus of freely moving rats during reward and non-reward delay period

Impulsive Responding in Alcoholics

Impaired decision-making is one diagnostic characteristic of alcoholism. Quantifying decision-making with rapid and robust laboratory-based measures is thus desirable for the testing of novel treatments for alcoholism. Previous research has demonstrated the utility of delay discounting (DD) tasks for quantifying differences in decision-making in substance abusers and normal controls. In DD paradigms subjects choose between a small, immediate reward and a larger, delayed reward. We used a novel computerized DD task to demonstrate that abstinent alcoholics (AA, n=14) choose the larger, delayed option significantly less often than control subjects (n=14; p<0.02). This difference in choice tendency was independent of subject age, gender, years of education, or socio-economic status. All subjects discounted as a function of reward delay and amount, with alcoholics demonstrating steeper discounting curves for both variables. This tendency to discount delayed rewards was positively correlated with subjective reports of both alcohol addiction severity (Drug Use Screening Inventory-Revised, Domain 1, p<0.01), and impulsivity (Barratt Impulsivity Scale-11, p<0.004). Novel aspects of this new paradigm include an element of time pressure, an additional experimental condition that evaluated motor impulsivity by assessing the ability to inhibit a pre-potent response, and another control condition to requiring non-subjective choice. Non-alcoholic controls and alcoholics did not differ on motor impulsivity or non-subjective choice, suggesting that the differing choice behavior of the two groups was due mainly to differences in cognitive impulsivity.

Development of a behavioral task measuring reward “wanting” and “liking” in rats

It has been suggested that reward “wanting” and “liking” are mediated by separable brain systems. To facilitate neuropharmacological and neurophysiological research on this issue we developed a behavioral task with putative measures of reward “wanting” and “liking” available on a trial-by-trial basis. We were able to test whether our measures were sensitive to changes in thirsty rats' “wanting” and “liking” of liquid reward by manipulating its delay, taste and volume. We found that three of our putative “wanting” measures (anticipatory errors, reaction time and reward collection latency) were affected by upcoming reward delay and/or taste and our putative “liking” measure (post-reward licking) was sensitive to variations in reward taste and volume. To cross-validate our measures with previous pharmacological work we tested rats following acute, systemic administration of drug compounds that globally enhance serotonin and noradrenaline (imipramine), dopamine (GBR 12909) and opioid (morphine) function. Imipramine augmented the effects of delay and taste on reward “wanting”, GBR 12909 attenuated the effects of delay on reward “wanting” and the effects of taste on reward “liking”, and morphine reduced the effect of delay on a measure of reward “wanting”. Since morphine failed to affect reward “liking” but has been previously found to enhance reward “liking” in taste reactivity tests, our measure requires further pharmacological validation. However, this task shows potential to assess the specific neural mechanisms that contribute to the impact of reward parameters on “wanting” and “liking”.

Antidepressant-like effects of agomelatine, melatonin and the NK1 receptor antagonist GR205171 in impulsive-related behaviour in rats

Substance P receptor [neurokinin1 (NK1-R)] antagonists and melatonin(1/2) receptor (MT(1/2)-R) agonists have been claimed to be potential antidepressants (ADs). In animals, these compounds are active in validated models responsive to ADs, such as forced swimming test and chronic mild stress paradigms. Classical AD drugs are also known to be effective in pathologies characterized by an impulse control deficiency. In line with this clinical observation, previous studies demonstrated that classical ADs increased the capacity to wait for food reward in rats subjected to a paradigm aimed at assessing impulsive-related behaviour. This study was conducted to investigate the effects of two MT(1/2)-R agonists, melatonin and agomelatine, and a NK1-R antagonist, GR205171, on tolerance to delay of food reward in rats. Fasting rats were trained in a T-maze and allowed to choose between two magnitudes of reward: immediate but small reward (two pellets) vs 25-s delayed but large reward (ten pellets). Under this alternative, vehicle-injected rats selected the large-but-delayed reinforcer in less than 40% of the trials. Like the established ADs clomipramine (8 mg kg(-1), i.p.) and fluvoxamine (4 mg kg(-1), i.p.), melatonin (3 and 10 mg kg(-1), i.p.), agomelatine (10 and 30 mg kg(-1), i.p.) and GR205171 (30 mg kg(-1) but not 10 mg kg(-1), s.c.) significantly increased the number of choices of the large-but-delayed reward. The effect of melatonin (3 mg kg(-1), i.p.) was not counteracted by the MT(1/2)-R antagonist S22153 (40 mg kg(-1), i.p.) that exerted no effect on its own. These results suggest that MT(1/2)-R agonists and NK1-R antagonists enhance rats' tolerance to delay of gratification, an effect which may reflect their ability to improve impulse control. Further investigations are necessary to clarify the neurobiological mechanisms responsible for this effect.

Altered sensitivity to reward in children with ADHD: Dopamine timing is off

despite general agreement that altered reward sensitivity is involved in attention-deficit/hyperactivity disorder (adhd), a wide range of different alterations has been proposed. we cite work showing abnormal sensitivity to delay of reward, together with abnormal sensitivity to individual instances of reward. we argue that at the cellular level these behavioural characteristics might indicate that dopamine timing is off in children with adhd.

The psychopharmacology of impulsive behaviour in rats VIII: effects of amphetamine, methylphenidate, and other drugs on responding maintained by a fixed consecutive number avoidance schedule

Impulsive behaviour is a component of psychiatric disorders such as bipolar disorder, attention deficit hyperactivity disorder (ADHD), or personality disorders. Most experimental studies on impulsive behaviour punish impulsive choices by loss or delay of reward. In the present study, impulsive behaviour was punished by an explicitly aversive stimulus, using a novel fixed consecutive number (FCN) schedule of electric shock avoidance. This study was conducted to demonstrate stable performance using the FCN avoidance procedure, and examine the effects of drugs previously shown to affect impulsive behaviour using a conventional FCN schedule. First, rats were trained in the appetitive FCN procedure. Pressing the right lever in an operant conditioning chamber after having pressed the left lever at least six times delivered a food pellet (FCN6). Responses on the right lever before completing this ratio resulted in a time out and restarted the ratio. The rats were then switched to FCN avoidance. Responses on the right lever made before completion of the ratio also resulted in food delivery, but were accompanied by an electric shock. Chlordiazepoxide (10.0 mg/kg), ethanol (1.0 g/kg), and haloperidol (0.1 mg/kg) increased impulsive behaviour by reducing the number of left lever responses made before the right lever was pressed. Imipramine (1.0-10.0 mg/kg) and desipramine (1.0-10.0 mg/kg) had no effect on impulsive choice. Amphetamine (0.4 and 0.8 mg/kg) and methylphenidate (6.0 mg/kg) significantly increased the mean chain length, and the proportion of very long chains, indicative of reduced impulsivity, although this did not improve efficiency. The increases in impulsivity produced by chlordiazepoxide, ethanol, and haloperidol were similar to those under appetitive FCN schedules. In contrast, amphetamine and methylphenidate, by reducing impulsivity in the FCN avoidance, induced effects opposite to those observed in an appetitive FCN procedure. These results suggest that the therapeutic actions of stimulants, to reduce impulsive behaviour in ADHD, may arise in part by increasing the control of behaviour by aversive stimuli.

Laboratory behavioral measures of impulsivity

Previous research and theory have conceptualized impulsivity as a multifaceted construct that requires multiple modes of measurement for accurate assessment. This article describes a software package that includes four paradigms for measuring multiple and unique aspects of impulsivity. Specifically, four tasks are described: (1) the two choice impulsivity paradigm, (2) the single key impulsivity paradigm, (3) the GoStop impulsivity paradigm, and (4) the time paradigm. These tasks measure processes related to the capacity to tolerate delay for reward, to inhibit an already initiated response, and to estimate the passage of time. These processes have been found to be important to the understanding of impulsive behaviors. The programs are flexible and allow the experimenter to manipulate a number of parameters related to delay-reward contingencies, timing, performance feedback/payment, and data output variables. Manipulation of these parameters makes the paradigms scalable to a wide range of ability levels and appropriate for samples ranging from children to adults. The four paradigms in this software package are available at no cost and can be obtained by contacting the corresponding author.

Differential involvement of serotonin and dopamine systems in cost-benefit decisions about delay or effort

Although tasks assessing the role of dopamine in effort-reward decisions are similar to those concerned with the role of serotonin in impulsive choice in that both require analysis of the costs and benefits of possible actions, they have never been directly compared. This study investigated the involvement of serotonin and dopamine in two cost-benefit paradigms, one in which the cost was delay and the other in which it was physical effort. Sixteen rats were trained on a T-maze task in which they chose between high and low reward arms. In one version, the high reward arm was obstructed by a barrier, in the other, delivery of the high reward was delayed by 15 s. Serotonin and dopamine function were manipulated using systemic pCPA and haloperidol injections, respectively. Haloperidol-treated rats were less inclined either to exert more effort or to countenance a delay for a higher reward. pCPA had no effect on the performance of the rats on the effortful task, but significantly increased the rats' preference for an immediate but smaller reward. All animals (drug treated and controls) chose the high reward arm on the majority of trials when the delay or effort costs were matched in both high and low reward arms. A dissociation was found between the neurotransmitter systems involved in different types of cost-benefit decision making. While dopaminergic systems were required for decisions about both effort and delay, serotonergic systems were only needed for the latter.

Framing effects and risky decisions in starlings.

Animals are predominantly risk prone toward reward delays and risk averse toward reward amounts. Humans in turn tend to be risk-seeking for losses and risk averse for gains. To explain the human results, Prospect Theory postulates a convex utility for losses and concave utility for gains. In contrast, Scalar Utility Theory (SUT) explains the animal data by postulating that the cognitive representation of outcomes follows Weber's Law, namely that the spread of the distribution of expected outcomes is proportional to its mean. SUT also would explain human results if utility (even if it is linear on expected outcome) followed Weber's Law. We present an experiment that simulates losses and gains in a bird, the European Starling, to test the implication of SUT that risk proneness/aversion should extend to any aversive/desirable dimension other than time and amount of reward. Losses and gains were simulated by offering choices of fixed vs. variable outcomes with lower or higher outcomes than what the birds expected. The subjects were significantly more risk prone for losses than for gains but, against expectations, they were not significantly risk averse toward gains. The results are thus, in part, consistent with Prospect Theory and SUT and show that risk attitude in humans and birds may obey a common fundamental principle.

Sensitivity to Reward Delay in Children with Attention Deficit Hyperactivity Disorder (ADHD)

Sensitivity to Reward Delay in Children with Attention Deficit Hyperactivity Disorder (ADHD)

Sensitivity to Reward Delay in Children with Attention Deficit Hyperactivity Disorder (ADHD)

The present study compared the sensitivity to pre- and post-reward delays of children with and without DSM-IV combined type ADHD. Thirty-six children with DSM-IV combined type ADHD and 36 controls completed a signal-detection task. Correct identification of one stimulus produced an immediate reward and then a 3.5 s delay before the next trial (immediate reward). Correct identification of the other stimulus was associated with a 3.5 s delay before reward was delivered (delayed reward). Group differences emerged in response bias toward the immediately rewarded alternative. Children in the ADHD group showed a greater bias toward immediate reward than the controls. Children in the control group showed different patterns of response bias and discriminability following immediate and delayed rewards. For these children discriminability was lower and response bias was greater on trials following delayed reward. Both groups responded more slowly on trials following delayed reward. These findings support the hypothesis that children with ADHD are unusually sensitive to pre-reward delays.

RISK-SENSITIVE FORAGING IN COAL TITS

Current theory of risk-sensitive foraging predicts that foragers should choose feeding sites on the basis of variation in as well as mean reward rate when there is a shortfall in their food supply or a decrease in their energy budget. For a given mean reward delay, they should choose high variance feeding sites if they are running below energy requirement, but low variance sites if they are running above. It has been suggested that the smaller the animal size, the stronger the preference reversion between high and low variable feeding sites. Previous tests of the energy budget rule when there was time variability have used bird species heavier than 80 g. Hence we tested energy budget rule predictions with coal tits Parus ater, a bird of 9 g of body mass, foraging at two feeding sites with high or low variability in food delivering delay. We manipulated energy budgets by controlling air temperature in the laboratory. In one treatment (positive budget), individuals were allowed to eat at the level of their own ad-libitum daily consumption and the air temperature was set to 24°C, while for the other treatment (negative budget), temperature was set to 14°C, and food availability was limited to the maximum daily intake observed in the positive budget treatment. When air temperature was low, daily intake increased but body mass decreased. Birds were also less active in the low temperature treatment, hopping less times every day. Latency to peck decreased as well, pecking for food when it was available sooner than in the high temperature treatment. These results show that coal tits were living in a negative energy budget when air temperature was set to 14°C. Preference for the variable feeding site was greatest in the negative energy budget, as it was predicted by the energy budget rule. Contrary to the energy budget rule, coal tits consistently preferred the variable option also in the positive energy budget. Possible explanations for these results are explored, including alternative foraging models to the energy budget rule.

Effects of dopaminergic drugs on delayed reward as a measure of impulsive behavior in rats.

Impulsive individuals exhibit an exaggerated preference for immediate rewards over delayed but larger rewards, perhaps because they value the delayed rewards less. Dopamine (DA) has been postulated to mediate the incentive value of rewards, thus it may also mediate the exaggerated preference for immediate rewards in impulsive individuals. In this paper, we investigate the effects of DA agonists and antagonists on the value of delayed versus immediate rewards. The study had three main objectives: (1) to determine the effects of D1 and D2 type DA antagonists on the value of delayed rewards, (2) to determine the effect of the indirect DA agonist d-amphetamine on the value of delayed rewards, (3) to determine the sensitivity of the adjusting amount (AdjAmt) procedure to acute one-day changes in delay to reward, amount of reward, deprivation level and starting amount. In the AdjAmt procedure, rats choose between an adjusting amount of water given immediately (adjusting alternative) and a constant 150 microliters water delayed by 4 s (standard alternative). The immediate amount of water is adjusted across trials until the rat chooses both alternatives with equal frequency (indifference point). The final adjusted amount is an indicator of the subjective value of the standard alternative. The D1/D2 antagonist flupenthixol (25, 50, and 100 micrograms/kg) and the D2 antagonist raclopride (40, 80, and 120 micrograms/kg), decreased the indifference points, whereas the D1 antagonist SCH 23390 (5, 10, and 20 micrograms/kg) did not affect the indifference point. All three DA antagonists slowed responding. The indirect DA agonist amphetamine (0.5, and 1.0 mg/kg) had effects opposite to those of the DA antagonists, it decreased choice latency, increased the number of trials completed and increased the indifference point. Decreasing the water deprivation level (6, 24, and 48 h) had no effect on the indifference points but slowed responding. Increasing the delay to reward (2, 4, and 8 s) and decreasing the amount of water available for choosing the standard alternative (300, 150, 75 microliters) decreased the indifference point. The results indicate that amphetamine increased the value of delayed rewards (decreased impulsivity) and that D2 but not D1 receptor antagonists decreased the value of delayed rewards (increased impulsivity). The procedure was sensitive to acute 1-day changes in delay and magnitude of reward.