Subjective Value Of Reward Research Articles

The architecture of reward value coding in the human orbitofrontal cortex.

To ensure their survival, animals exhibit a number of reward-directed behaviors, such as foraging for food or searching for mates. This suggests that a core set of brain regions may be shared by many species to process different types of rewards. Conversely, many new brain areas have emerged over the course of evolution, suggesting potential specialization of specific brain regions in the processing of more recent rewards such as money. Here, using functional magnetic resonance imaging in humans, we identified the common and distinct brain systems processing the value of erotic stimuli and monetary gains. First, we provide evidence that a set of neural structures, including the ventral striatum, anterior insula, anterior cingulate cortex, and midbrain, encodes the subjective value of rewards regardless of their type, consistent with a general hedonic representation. More importantly, our results reveal reward-specific representations in the orbitofrontal cortex (OFC): whereas the anterior lateral OFC, a phylogenetically recent structure, processes monetary gains, the posterior lateral OFC, phylogenetically and ontogenetically older, processes more basic erotic stimuli. This dissociation between OFC representations of primary and secondary rewards parallels current views on lateral prefrontal cortex organization in cognitive control, suggesting an increasing trend in complexity along a postero-anterior axis according to more abstract representations. Together, our results support a modular view of reward value coding in the brain and propose that a unifying principle of postero-anterior organization can be applied to the OFC.

Subjective neuronal coding of reward: temporal value discounting and risk

A key question in the neurobiology of reward relates to the nature of coding. Rewards are objects that are advantageous or necessary for the survival of individuals in a variety of environmental situations. Thus reward appears to depend on the individual and its environment. The question arises whether neuronal systems in humans and monkeys code reward in subjective terms, objective terms or both. The present review addresses this issue by dealing with two important reward processes, namely the individual discounting of reward value across temporal delays, and the processing of information about risky rewards that depends on individual risk attitudes. The subjective value of rewards decreases with the temporal distance to the reward. In experiments using neurophysiology and brain imaging, dopamine neurons and striatal systems discount reward value across temporal delays of a few seconds, despite unchanged objective reward value, suggesting subjective value coding. The subjective values of risky outcomes depend on the risk attitude of individual decision makers; these values decrease for risk-avoiders and increase for risk-seekers. The signal for risk and the signal for the value of risky reward covary with individual risk attitudes in regions of the human prefrontal cortex, suggesting subjective rather than objective coding of risk and risky value. These data demonstrate that important parameters of reward are coded in a subjective manner in key reward structures of the brain. However, these data do not rule out that other neurons or brain structures may code reward according to its objective value and risk.

Performance in Temporal Discounting Tasks by People With Intellectual Disabilities Reveals Difficulties in Decision-Making and Impulse Control

The subjective value of rewards declines as a function of the delay to receive them (temporal discounting). Three temporal discounting tasks that assessed preferences between small amounts of money (10 pence) over short delays (60 s), moderate amounts of money (10 pound) over moderate delays (2 weeks), and large amounts of money (1000 pound) over long delays (12 months) were presented to people with intellectual disabilities (Full-Scale IQ < 70) and to a comparison group (ns = 20 for each group). Measures of IQ, financial knowledge, memory, and executive functioning were also obtained. Only a third of the service users were able to perform the temporal discounting tasks consistently, and they tended to respond impulsively. The proportion of participants responding consistently increased following training. Both the initial performance and the effect of training were related to executive functioning but not IQ.

Economic decision making and prefrontal cortex

Event Abstract Back to Event Economic decision making and prefrontal cortex Daeyeol Lee1* 1 Yale University School of Medicine, United States During decision making, the outcomes expected from alternative actions are often evaluated along multiple dimensions, such as the magnitude and delay of reward. We found that during inter-temporal choice in which the animals choose between a small immediate reward and a larger but more delayed reward, the neurons in the lateral prefrontal cortex and basal ganglia often encode the subjective value of a particular reward discounted by its delay. The lateral prefrontal cortex also plays an important role in updating the animal’s preference for alternative actions based on its previous experience. Using a computer-simulated rock-paper-scissors task, we found that the activity of individual prefrontal cortex tends to reflect not only the actual outcomes of the animal’s chosen actions, but also the hypothetical outcomes that could have been obtained from one of unchosen actions. These results suggest that the prefrontal cortex provides the learning signals necessary to maximize the efficiency of reinforcement learning in addition to the value signals during decision making.

Temporal Reward Discounting in Attention-Deficit/Hyperactivity Disorder: The Contribution of Symptom Domains, Reward Magnitude, and Session Length

Theoretical models have hypothesized that one core problem in attention-deficit/hyperactivity disorder (ADHD) is abnormal reward processing. Temporal reward discounting (decreases in subjective reward value due to prereward delay) is of interest because of its relation with a key symptom of ADHD-impulsivity. This study investigated 1) whether steep temporal reward discounting (TD) is associated with ADHD-combined type (ADHD-C)/symptoms of hyperactivity-impulsivity specifically; 2) the role of reward magnitude in TD in ADHD-C/participants with symptoms of hyperactivity-impulsivity; and 3) whether steep TD in ADHD-C/participants with symptoms of hyperactivity-impulsivity is affected by session length. Three TD tasks were administered to children and adolescents (aged 6-17) with ADHD-C (n = 25), ADHD-inattentive type (ADHD-I; n = 20), and matched typically developing participants (n = 37). Reward magnitude and session length were varied. Steep TD was observed in participants with ADHD-C but not in those with ADHD-I, independent of reward magnitude and session length. Dimensional analyses revealed that steep TD was associated with hyperactivity-impulsivity (transcending the arbitrary cutoff for ADHD subtypes), especially when reward magnitude at the trial level was small. These findings suggest that steep TD in ADHD is best thought of as a correlate of the symptom dimension of hyperactivity/impulsivity. Additionally, steep TD in ADHD is the result of a trade-off between delay and reward magnitude, with all factors contributing to choice preferences. These findings may help refine the delay aversion theory of ADHD, and provide evidence for the notion that unique reward processing is one mechanism associated with symptoms of hyperactivity-impulsivity.

Dissociation of neural networks for anticipation and consumption of monetary and social rewards

Human behaviour is generally guided by the anticipation of potential outcomes that are considered to be rewarding. Reward processing can thus be dissected into a phase of reward anticipation and a phase of reward consumption. A number of brain structures have been suggested to be involved in reward processing. However, it is unclear whether anticipation and consumption are mediated by the same or different neural networks. We examined the neural basis of these processes using functional magnetic resonance imaging (fMRI) in an incentive delay task offering either money or social approval. In both conditions participants (N=28) were given a cue indicating potential reward. In order to receive reward a target button had to be pushed within a certain time window (adapted for individual reaction time). Cues triggering either monetary or social reward anticipation were presented sessionwise. Imaging was performed on a 1.5-Tesla Philips scanner in an event-related design. Anticipation of both reward types activated brain structures constituting the brain reward system including the ventral striatum. In contrast to the task independent activity in the anticipation phase, reward consumption evoked different patterns of activation for money and social approval, respectively. While social stimuli were mainly associated with amygdala activation, the thalamus was more strongly activated by the presentation of monetary rewards. Our results identify dissociable neural networks for the anticipation and consumption of reward. The findings implicate that the neural mechanisms underlying reward consumption are more modality-specific than those for reward anticipation, and that they are mediated by subjective reward value.

Reward contrast in delay and probability discounting

In this study, we examined whether reward contrast influences choice between delayed and probabilistic outcomes. Specifically, we predicted that the subjective value of an intermediate reward would seem relatively larger or smaller, respectively, if it followed choices involving a smaller or larger reward and would produce corresponding changes in rates of delay and probability discounting. In Experiment 1, subjects made choices about hypothetical $5,000 or $50 outcomes and then made choices about $500 outcomes. Delay-discounting rates for the $500 outcome were larger for Group $5,000 than for Group $50, whereas the opposite result was obtained for probability-discounting rates. In Experiment 2, we used a design that allowed for contrast effects to be assessed within subjects. Two groups made choices about delayed or probabilistic rewards. After completing question blocks in which the amount was $5,000 or $50, subjects responded to questions with an intermediate amount ($475/$525). For Group Delay, the present value of the intermediate reward was greater after the $50 block than after the $5,000 block, whereas the opposite was obtained for Group Probability. The results from both experiments confirmed the predictions of reward contrast and suggested that the subjective value of a monetary reward varies inversely with the prior reward amount.

Strain differences in delay discounting using inbred rats

A heightened aversion to delayed rewards is associated with substance abuse and numerous other neuropsychiatric disorders. Many of these disorders are heritable, raising the possibility that delay aversion may also have a significant genetic or heritable component. To examine this possibility, we compared delay discounting in six inbred strains of rats (Brown Norway, Copenhagen, Lewis, Fischer, Noble and Wistar Furth) using the adjusting amount procedure, which provides a measure of the subjective value of delayed rewards. The subjective value of rewards decreased as the delay to receipt increased for all strains. However, a main effect of strain and a strain x delay interaction indicated that some strains were more sensitive to the imposition of delays than others. Fitting a hyperbolic discount equation showed significant strain differences in sensitivity to delay (k). These data indicate that there are significant strain differences in delay discounting. All strains strongly preferred the 10% sucrose solution (the reinforcer in the delay discounting task) over water and the amount of sucrose consumed was correlated with sensitivity to delay. Locomotor activity was not correlated with delay discounting behavior. Additional research will be required to disentangle genetic influences from maternal effects and to determine how these factors influence the underlying association between heightened delay discounting and neuropsychiatric disorders.

Temporal Discounting of Hypothetical Monetary Rewards by Adolescents, Adults, and Older Adults

The present experiment examined temporal discounting across 3 different age bands: adolescents, adults, and older adults (mean ages 14, 46, and 73 years, respectively). A computerized task was employed in which participants were asked to choose between larger rewards available at a specified time in the future—either £100 or £1,000 (approximately $200 and $2,000, respectively)— or a smaller reward available immediately. The subjective value of the reward decreased with increasing delay for each of the 3 age groups. A hyperbola-like function adequately described the group discounting data. The adolescent group discounted significantly more than the adult group when the larger later reward (LLR) was £100 but not when the LLR was £1,000. The adolescents discounted significantly more than the older adult group when the LLR was either £100 or £1,000. There were no significant differences in discounting between the adult and older adult groups. The results of the present study suggest that the rate of temporal discounting is higher in adolescents than in adults but is stable from middle adulthood to older adulthood. Furthermore, the process of temporal discounting appears to be quantitatively similar across the life span. The term temporal discounting refers to the tendency of individuals to prefer smaller sooner rewards (SSRs) over larger later rewards (LLRs). For example, if presented with the choice between $85 now and $100 in 1 year, some people would choose $85 now. Those who choose the SSR can be considered impulsive; those who prefer the LLR can be said to have exercised self-control (e.g., Rachlin & Green, 1972). One widely used procedure to measure the rate of temporal discounting involves a choice option between an amount of money, hypothetical or real, that is available immediately and a larger amount that is available at a later date. The SSR is manipulated over successive trials, while the LLR is kept constant. The aim of this procedure is to identify the current subjective value of the temporal rewards, which is defined as the magnitude of SSR that

Tsallis’ non-extensive free energy as a subjective value of an uncertain reward

Recent studies in neuroeconomics and econophysics revealed the importance of reward expectation in decision under uncertainty. Behavioral neuroeconomic studies have proposed that the unpredictability and the probability of an uncertain reward are distinctly encoded as entropy and a distorted probability weight, respectively, in the separate neural systems. However, previous behavioral economic and decision-theoretic models could not quantify reward-seeking and uncertainty aversion in a theoretically consistent manner. In this paper, we have: (i) proposed that generalized Helmholtz free energy in Tsallis’ non-extensive thermostatistics can be utilized to quantify a perceived value of an uncertain reward, and (ii) empirically examined the explanatory powers of the models. Future study directions in neuroeconomics and econophysics by utilizing the Tsallis’ free energy model are discussed.

The neural correlates of subjective value during intertemporal choice

Neuroimaging studies of decision-making have generally related neural activity to objective measures (such as reward magnitude, probability or delay), despite choice preferences being subjective. However, economic theories posit that decision-makers behave as though different options have different subjective values. Here we use functional magnetic resonance imaging to show that neural activity in several brain regions--particularly the ventral striatum, medial prefrontal cortex and posterior cingulate cortex--tracks the revealed subjective value of delayed monetary rewards. This similarity provides unambiguous evidence that the subjective value of potential rewards is explicitly represented in the human brain.

Learning reward expectations in honeybees

The aim of this study was to test whether honeybees develop reward expectations. In our experiment, bees first learned to associate colors with a sugar reward in a setting closely resembling a natural foraging situation. We then evaluated whether and how the sequence of the animals' experiences with different reward magnitudes changed their later behavior in the absence of reinforcement and within an otherwise similar context. We found that the bees that had experienced increasing reward magnitudes during training assigned more time to flower inspection 24 and 48 h after training. Our design and behavioral measurements allowed us to uncouple the signal learning and the nutritional aspects of foraging from the effects of subjective reward values. We thus found that the animals behaved differently neither because they had more strongly associated the related predicting signals nor because they were fed more or faster. Our results document for the first time that honeybees develop long-term expectations of reward; these expectations can guide their foraging behavior after a relatively long pause and in the absence of reinforcement, and further experiments will aim toward an elucidation of the neural mechanisms involved in this form of learning.

Temporal and probabilistic discounting of rewards in children and adolescents: Effects of age and ADHD symptoms

This study investigated whether age and ADHD symptoms affected choice preferences in children and adolescents when they chose between (1) small immediate rewards and larger delayed rewards and (2) small certain rewards and larger probabilistic uncertain rewards. A temporal discounting (TD) task and a probabilistic discounting (PD) task were used to measure the degree to which the subjective value of a large reward decreased as one had to wait longer for it (TD), and as the probability of obtaining it decreased (PD). Rewards used were small amounts of money. In the TD task, the large reward (10 cents) was delayed by between 0 and 30 s, and the immediate reward varied in magnitude (0–10 cents). In the PD task, receipt of the large reward (10 cents) varied in likelihood, with probabilities of 0, 0.25, 0.5, 0.75, and 1.0 used, and the certain reward varied in magnitude (0–10 cents). Age and diagnostic group did not affect the degree of PD of rewards: All participants made choices so that total gains were maximized. As predicted, young children, aged 6–11 years ( n = 25) demonstrated steeper TD of rewards than adolescents, aged 12–17 years ( n = 21). This effect remained significant even when choosing the immediate reward did not shorten overall task duration. This, together with the lack of interaction between TD task version and age, suggests that steeper discounting in young children is driven by reward immediacy and not by delay aversion. Contrary to our predictions, participants with ADHD ( n = 22) did not demonstrate steeper TD of rewards than controls ( n = 24). These results raise the possibility that strong preferences for small immediate rewards in ADHD, as found in previous research, depend on factors such as total maximum gain and the use of fixed versus varied delay durations. The decrease in TD as observed in adolescents compared to children may be related to developmental changes in the (dorsolateral) prefrontal cortex. Future research needs to investigate these possibilities.

Single Units in the Pigeon Brain Integrate Reward Amount and Time-to-Reward in an Impulsive Choice Task

Animals prefer small over large rewards when the delays preceding large rewards exceed an individual tolerance limit. Such impulsive choice behavior occurs even in situations in which alternative strategies would yield more optimal outcomes. Behavioral research has shown that an animal's choice is guided by the alternative rewards' subjective values, which are a function of reward amount and time-to-reward. Despite increasing knowledge about the pharmacology and anatomy underlying impulsivity, it is still unknown how the brain combines reward amount and time-to-reward information to represent subjective reward value. We trained pigeons to choose between small, immediate rewards and large rewards delivered after gradually increasing delays. Single-cell recordings in the avian Nidopallium caudolaterale, the presumed functional analog of the mammalian prefrontal cortex, revealed that neural delay activation decreased with increasing delay length but also covaried with the expected reward amount. This integrated neural response was modulated by reward amount and delay, as predicted by a hyperbolical equation, of subjective reward value derived from behavioral studies. Furthermore, the neural activation pattern reflected the current reward preference and the time point of the shift from large to small rewards. The reported activity was modulated by the temporal devaluation of the anticipated reward in addition to reward amount. Our findings contribute to the understanding of neuropathologies such as drug addiction, pathological gambling, frontal lobe syndrome, and attention-deficit disorders, which are characterized by inappropriate temporal discounting and increased impulsiveness.

Effects of inflation on the subjective value of delayed and probabilistic rewards

In the years prior to 1994, there were very high rates of inflation in Poland, and the zloty depreciated relative to the U.S. dollar. However, the new zloty, introduced in 1995, was associated with greatly decreased rates of inflation and provided a more stable currency. We report a series of three experiments that take advantage of these changes to examine the effects of inflation on the subjective value of delayed and probabilistic rewards. Subjects were Polish citizens familiar with both zlotys and dollars. The first two experiments, conducted in 1994, used dollars and old zlotys, and the third experiment, conducted in 1996, used dollars and new zlotys. In all three experiments, the dollar and zloty rewards were of equivalent worth, according to the then current exchange rates. In Experiment 1, subjects chose between immediate and delayed rewards and, in Experiment 2, chose between certain and probabilistic rewards. The subjective value of a delayed reward was greater when its amount was specified in dollars than when it was specified in old zlotys. In contrast, the currency in which a reward was specified had no effect on the subjective value of probabilistic rewards. The results of these two experiments suggest a selective effect of inflation on decisions involving delayed rewards. This was verified in the third experiment, in which, using new zlotys, no differences in discounting were observed between the two currencies with either probabilistic or delayed rewards. Importantly, in all three experiments, the discounting of both delayed and probabilistic rewards was well described by the same simple mathematical model, suggesting that similar decision-making processes underlie both phenomena. However, the present results argue against a single-process theory in which the discounting of probabilistic rewards is derived from the discounting of delayed rewards.

Effects of expected delay time on the subjective value of rewards and punishments.

Effects of expected delay time on the subjective value of rewards and punishments.