Future Rewards Research Articles

Neuroanatomical and functional substrates of the greed personality trait.

Greedy individuals often exhibit more impulsive decision-making and short-sighted behaviors. It has been assumed that altered reward circuitry and prospection network is associated with greed personality trait (GPT). In this study, we first explored the morphological characteristics (i.e., gray matter volume; GMV) of GPT combined with univariate and multivariate pattern analysis (MVPA) approaches. Second, we adopted a revised version of inter-temporal choice task and independently manipulated the amount and delay time of future rewards. Using brain-imaging design, reward- and prospection-related brain activations were assessed and their associations with GPT were further examined. The MVPA results showed that GPT was associated with the GMVs in the right lateral frontal pole cortex, left ventromedial prefrontal cortex, right lateral occipital cortex, and right occipital pole. Additionally, we observed that the amount-relevant brain activations (responding to reward circuitry) in the lateral orbitofrontal cortex were negatively associated with individual's variability in GPT scores, whereas the delay time-relevant brain activations (responding to prospection network system) in the dorsolateral prefrontal cortex, dorsomedial prefrontal cortex, superior parietal lobule, and anterior cingulate cortex were positively associated with individual's variability in GPT scores. These findings not only provide novel insights into the neuroanatomical substrates underlying the human dispositional greed, but also suggest the critical roles of reward and prospection processing on the greed.

Inference and preference in intertemporal choice

AbstractWhen choosing between immediate and future rewards, how do people deal with uncertainty about the value of the future outcome or the delay until its occurrence? Skylark et al. (2020) suggested that people employ a delay-reward heuristic: the inferred value of an ambiguous future reward is a function of the stated delay, and vice-versa. The present paper investigates the role of this heuristic in choice behaviour. In Studies 1a–2b, participants inferred the value of an ambiguous future reward or delay before the true value was revealed and a choice made. Preference for the future option was predicted by the discrepancy between the estimated and true values: the more pleasantly surprising the delayed option, the greater the willingness to choose it. Studies 3a–3c examined the association between inference and preference when the ambiguous values remained unknown. As predicted by the use of a delay-reward heuristic, inferred delays and rewards were positively related to stated rewards and delays, respectively. More importantly, choices were associated with inferred rewards and, in some circumstances, delays. Critically, estimates and choices were both order-dependent: when estimates preceded choices, estimates were more optimistic (people inferred smaller delays and larger rewards) and were subsequently more likely to choose the delayed option than when choices were made before estimates. These order effects argue against a simple model in which people deal with ambiguity by first estimating the unknown value and then using their estimate as the basis for decision. Rather, it seems that inferences are partly constructed from choices, and the role of inference in choice depends on whether an explicit estimate is made prior to choosing. Finally, we also find that inferences about ambiguous delays depend on whether the estimate has to be made in “days” or in a self-selected temporal unit, and replicate previous findings that older participants make more pessimistic inferences than younger ones. We discuss the implications and possible mechanisms for these findings.

Smoking, hazardous drinking and individual differences in cost and gain discounting.

Aims:Past research has linked substance use to individual differences in discounting of future rewards. Since behaviours such as smoking and excessive drinking appear to involve a devaluation of future negative consequences, discounting of costs may also be relevant in the understanding of such behaviour. The primary aims were to investigate the association between cost discounting and the behaviours smoking and hazardous drinking.Methods:In four studies, individuals recruited from the crowdsourcing marketplace Amazon Mechanical Turk responded to measures of discounting. Results were combined by meta-analysis of the standardised mean differences (d) between self-reported smokers and non-smokers, and between participants with AUDIT scores of 10+ (hazardous drinking) and those with scores below 10.Results:In comparison with non-smokers, smokers’ relative valuations of future gains were lower, d = –0.32, 95% CI [–0.47, –0.18]. There was no association between smoking and cost discounting, d = –0.02, 95% CI [–0.17, 0.13]. Participants with AUDIT scores of 10+ valued future gains somewhat lower than participants with scores below 10, d = –0.17, 95% CI [–0.35, 0.01]. There was no association between hazardous drinking and cost discounting, d = –0.02, 95% CI [–0.21, 0.16]. According to Bayes Factors (BF), the data supported an association between gain discounting and smoking (BF > 100). It was insensitive in the analysis of gain discounting and hazardous drinking (BF = 0.6), but it strongly supported null-effects in both analyses of cost discounting (BFs = 0.1).Conclusion:The results suggest that the robust link between gain discounting and smoking status does not reflect a general devaluation of future outcomes among smokers.

Neural Representations of Death in the Cortical Midline Structures Promote Temporal Discounting.

Death is an important reminder that our lives are finite. Although some studies have shown that thinking about one’s own death increases temporal discounting (i.e., the devaluing of future rewards), the underlying neural mechanisms are still unknown. In a functional magnetic resonance imaging experiment, we compared the neural and behavioral processes of temporal discounting across four conditions involving distinct types of future thinking (death related, negative, neutral, and positive). Replicating prior research, the behavioral evidence showed that temporal discounting increased when thinking about one’s own future death. Multivoxel pattern analysis showed that death-related future thinking was decoded in default mode regions, including the inferior parietal lobule, precuneus, and medial prefrontal cortex (MPFC). When future thinking was death related (vs. negative), increased temporal discounting was associated with a higher decoding accuracy in the precuneus and MPFC. The present findings suggest that death-related neural representations are distributed across default mode regions, and neural populations in the cortical midline structures play a crucial role in the integration of one's own death into economic decision-making.

Throughput Maximization of Hybrid Access in Multi-Class Cognitive Radio Networks With Energy Harvesting

In this paper, the hybrid interweave/underlay channel access mode is studied for an energy harvesting (EH) cognitive radio network with multi-class secondary users (SUs). The hybrid channel access mode combines the benefits of interweave transmission (opportunistic access with high throughput) and that of the underlay transmission (any time transmission with controlled power). EH upgrades the SUs' devices to be self sustainable. Additionally, classifying the SUs helps to meet their different quality of service (QoS) requirements. The system is modelled as a mixed observable Markov decision process (MOMDP) to handle the uncertainty in the primary user (PU) activity and consider future rewards. The MOMDP model is solved to maximize the SUs' throughput using two algorithms, namely, the point-based value iteration and the heuristic search value iteration (HSVI). Moreover, skipping the schedule of some SUs is proved to increase the channel utilization. The HSVI is proved to be efficient and reduces the time complexity significantly. Compared to related work in literature, the proposed model is proved to be superior in terms of throughput, tunable to meet the different QoS requirements of SU classes, and can accommodate any number of SU classes. Finally, the effect of some system parameters on the proposed system performance is studied and some insights are derived about the structure of the optimal policy and the system parameters values.

Evaluating effects of episodic future thinking on valuation of delayed reward in cocaine use disorder: a pilot study

ABSTRACT Background: Episodic future thinking (EFT; i.e., envisioning oneself in future contexts) has been demonstrated to reduce discounting of future reward in healthy adults. While this approach has the potential to support future-oriented decision-making in substance use recovery, the impact of EFT on discounting behavior in illicit stimulant users has not yet been evaluated. Objectives: This pilot study aimed to (1) assess the feasibility of utilizing EFT methods in individuals with cocaine use disorder (CUD) and (2) conduct preliminary measurement of the EFT effect on discounting behavior in this population. Methods: Eighteen treatment-seeking individuals with CUD (17 males) were interviewed about positive and neutral events expected to occur at a range of future latencies. Future event information identified by participants was subsequently included on a subset of trials in an intertemporal choice task to promote EFT; within-subject differences in discounting between standard and EFT conditions were evaluated. Results: Participants identified relevant events and demonstrated decreased discounting of future reward when event descriptors were included (relative to discounting without event descriptors; p = .039). It was further noted that most events identified by participants were goals, rather than plans or significant dates. Conclusion: While methods previously used to study the effect of EFT on discounting behavior in healthy individuals are also effective in individuals with CUD, methodological factors – including types of events identified – should be carefully considered in future work. These preliminary findings suggest that EFT can reduce impulsive decision-making in cocaine use disorder and may therefore have therapeutic value.

Delay discounting rate by a surrogate decision maker depends on the smoking status of the recipient.

The tendency to devalue future rewards is known as delay discounting. Discounting is measured using a series of intertemporal choices between smaller, sooner outcomes and larger, later outcomes. We used a surrogate delay discounting task to explore whether such choices would differ if a hypothetical recipient was a smoker or was an individual with good health habits. Across three studies, the descriptions of the recipient included only information about smoking status (n = 66), smoking status and equal annual income (n = 47), and smoking status and equal weekly expenditures (n = 42). Higher rates of delay discounting for the smoker recipient compared to the nonsmoker recipient were observed across all three studies. These results parallel previous findings showing group differences in discounting between actual smokers and nonsmokers. We discuss the similarities between the present results and previous studies in light of an extension of Bem's (1967) self-perception theory, which posits that choices in laboratory-based delay discounting tasks are informed by observation of real-world intertemporal choice. The theory asserts that there is no fundamental difference between a first-person account of such knowledge and a third-person account. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Patients with anxiety and depression likely to have cravings

Researchers have found that delay discounting (future versus immediate rewards) and cue‐induced craving were increased in patients in methadone maintenance treatment (MMT) compared to controls. The delay discounting and craving scores were unrelated to methadone dosage, duration on methadone, withdrawal symptoms or nicotine dependence. Elevated delay discounting is of course present with opioid use disorder (OUD) as it is with all substance use disorders, but whether it continues with treatment was unclear.

Intertemporal Choices Are Causally Influenced by Fluctuations in Visual Attention

Intertemporal discount rates vary widely across contexts and individuals. We propose that a sizable fraction of this variation results from differences in how visual attention is allocated to different features of the decision, such as earlier versus future rewards, and that fluctuations in attentional patterns alter choices. We first tested this hypothesis in an experiment in which participants chose between receiving smaller-sooner versus larger-later monetary rewards while their attention was recorded with eye tracking. We found that cross-participant variation in the allocation of attention explained between 40% and 53% of the individual differences in discounting and that cross-trial variation explained 16% of the participants’ propensity to choose the delayed option. To test causality, multiple additional experiments exogenously manipulated the allocation of visual attention and found that shifting attention to attributes that are relatively more attractive in a larger-later option increased patient decision making and altered purchasing behavior. Together, these results are consistent with the existence of a causal impact of visual attention on intertemporal choice and suggest that manipulating attention can have a sizeable impact for important managerial and public policy choice domains. This paper was accepted by Yuval Rottenstreich, decision analysis.

Quasi-Hyperbolic Present Bias: A Meta-Analysis

Quasi-hyperbolic discounting is one of the most well-known and widely-used models to capture self-control problems in the economics literature. The underlying assumption of this model is that agents have a toward current consumption such that all future rewards are downweighed relative to rewards in the present (in addition to standard exponential discounting for the length of delay). We report a meta-analytic dataset of estimates of the present bias parameter β based on searches of all major research databases (62 papers with 81 estimates in total). We find that the literature shows that people are on average present biased for both monetary rewards (β = 0.82, 95% confidence interval of [0.74, 0.90]) and nonmonetary rewards (β = 0.66, 95% confidence interval of [0.51, 0.85]) but that substantial heterogeneity exists across studies. The source of this heterogeneity comes from the subject pool, elicitation methodology, geographical location, payment method, mode of data collection (e.g. laboratory or field), and reward type. There is evidence of selective reporting and publication bias in the direction of overestimating the strength of present-bias (making β estimates smaller), but present bias still exists after correcting for these issues (for money β = 0.87 with 95% confidence interval of [0.82, 0.92] after correcting for selective reporting).

Dynamic Bandits with Temporal Structure

Multi-armed bandit (MAB) problems are mainly studied under two extreme settings known as stochastic and adversarial. The two aforementioned settings, however, does not capture realistic environments such as search engines and marketing and advertising, in which rewards stochastically change in time. Motivated by that, we introduce and study dynamic MAB problems with stochastic temporal structures. In this problem, the expected reward of each arm follows an auto-regressive (AR) model that governs the temporal structure of the rewards. Due to the dynamic nature of the rewards, simple explore and commit policies fail, as all arms have to get explored continuously over time. We formalize this by characterizing a per-round regret lower bound, where the regret is measured against a strong (dynamic) benchmark. We then present an algorithm whose per-round regret almost matches the regret lower bound. The algorithm relies on two mechanisms: (i) alternating between recently pulled arms and unpulled arms with potential, and (ii) restarting. These mechanisms enables the algorithm to not only dynamically adapt to changes, but also discard irrelevant past information at a suitable rate. In numerical studies, we further demonstrate the strength of our algorithm, especially when the future rewards are highly correlated with past observations.

Traffic Engineering in Software-defined Networks using Reinforcement Learning: A Review

With the exponential increase in connected devices and its accompanying complexities in network management, dynamic Traffic Engineering (TE) solutions in Software-Defined Networking (SDN) using Reinforcement Learning (RL) techniques has emerged in recent times. The SDN architecture empowers network operators to monitor network traffic with agility, flexibility, robustness and centralized control. The separation of the control and the forwarding plane in SDN has enabled the integration of RL agents in the networking architecture to enforce changes in traffic patterns during network congestions. This paper surveys major RL techniques adopted for efficient TE in SDN. We reviewed the use of RL agents in modelling TE policies for SDNs, with agents’ actions on the environment guided by future rewards and a new state. We further looked at the SARL and MARL algorithms the RL agents deploy in forming policies for the environment. The paper finally looked at agents design architecture in SDN and possible research gaps.

Quasi-hyperbolic Present Bias: A Meta-analysis

Quasi-hyperbolic discounting is one of the most well-known and widely-used models to capture selfcontrol problems in the economics literature. The underlying assumption of this model is that agents have a “present bias” toward current consumption such that all future rewards are downweighed relative to rewards in the present (in addition to standard exponential discounting for the length of delay). We report a meta-analytic dataset of estimates of the present bias parameter 𝛽 based on searches of all major research databases (62 papers with 81 estimates in total). We find that the literature shows that people are on average present biased for both monetary rewards (𝛽 = 0.82%-|-| confidence interval of [0.74, 0.90]) and non- monetary rewards (𝛽 = 0.66, 95% confidence interval of [0.51, 0.85]) but that substantial heterogeneity exists across studies. The source of this heterogeneity comes from the subject pool, elicitation methodology, geographical location, payment method, mode of data collection (e.g. laboratory or field), and reward type. There is evidence of selective reporting and publication bias in the direction of overestimating the strength of present-bias (making 𝛽 estimates smaller), but present bias still exists after correcting for these issues (for money 𝛽 = 0.87 with 95% confidence interval of [0.82, 0.92] after correcting for selective reporting).

A Deep Reinforcement Learning Based Scheduling Policy for Reconfigurable Manufacturing Systems

Reconfigurable manufacturing systems (RMS) is one of the trending paradigms toward a digitalised factory. With its rapid reconfiguring capability, finding a far-sighted scheduling policy is challenging. Reinforcement learning is well-equipped for finding highly efficient production plans that would bring near-optimal future rewards. For minimising reconfiguring actions, this paper uses a deep reinforcement learning agent to make autonomous decision with a built-in discrete event simulation model of a generic RMS. Aiming at the completion of the assigned order lists while minimising the reconfiguration actions, the agent outperforms the conventional first-in-first-out dispatching rule after self-learning.

Outlook for democracy and democratic institutions.

If countries are not ready for democracy until they have a well-developed civil society, which can take many years to develop, then we need to identify its essential elements, so that their development can be directly encouraged in new democracies. It is argued here that autonomously elected local governments are the essential elements of civil society. A theory of the foundations of the state should begin by recognizing that states are established by leaders. Both to win power and to wield it, a leader needs an ability to make credible long-term promises of future rewards. For a successful democracy, we need that our democratic candidates should also have reputations for respecting democratic norms and for using public resources effectively in public service, and not just for reliably rewarding loyal supporters. Such reputations for democratic leadership can be generated most abundantly in local politics. In a federal democracy, successful local leaders can demonstrate their qualifications to compete for higher office. In this way, autonomously elected local governments effectively reduce entry barriers into national politics. It is suggested that Americans should never again elect anybody president who has not served responsibly in some lower public office under our constitutional system of government.

Brain responses during delay discounting in youth at high-risk for substance use disorders

Offspring of parents with substance use disorders (SUD) discount future rewards at a steeper rate on the monetary delay discounting task (DD) than typically developing youth. However, brain activation during DD has yet to be studied in drug naïve youth with a family history (FH) of SUD. Here, we investigate brain activation differences in high-risk youth during DD. We recruited substance naïve youth, aged 11–12, into three groups to compare brain activation during DD: (1) High-risk youth (n = 35) with a FH of SUD and externalizing psychiatric disorders, (2) psychiatric controls (n = 25) who had no FH of SUD, but with equivalent externalizing psychiatric disorders as high-risk youth, and (3) a healthy control group (n = 24) with no FH of SUD and minimal psychopathology. A whole-brain voxel wise analysis of the [Delay > Baseline], [Immediate > Baseline], and [Control > Baseline] contrasts identified functional regions of interest, from which extracted parameter estimates were tested for significant group differences. Relative to control youth, high-risk youth showed stronger activation in the left posterior insula and thalamus when making delayed choices, and stronger activation of the parahippocampal gyrus when making both delayed and control choices (ps < 0.05). Activation in the left posterior insula negatively correlated with both subscales of the Emotion Regulation Checklist, and positively correlated with the Stroop interference effect (ps < 0.05). Our findings suggest possible heritable SUD risk neural markers that distinguish drug naïve high-risk youth from psychiatric and healthy controls.

China’s millennial car travelers’ mode shift responses under congestion pricing and reward policies: A case study in Beijing

This paper investigates the potential impacts of congestion pricing and reward policies on migrant and resident millennial car travelers’ morning commute mode shift responses in China. A stated-preference survey developed for this study was conducted in 2017 among millennial car travelers living or working in Beijing’s inner district, from which approximately 2000 responses were collected. Separate random parameters bivariate ordered probit models were estimated for migrant and resident millennial car travelers to capture the differences between their mode shift responses, unobserved heterogeneity, and correlation between their mode shift responses to congestion pricing and reward policies. Sociodemographic characteristics, travel behavior and needs, residential location accessibility to and by transit, attitudes towards travel and congestion pricing and reward policies were found to affect millennial car travelers’ mode shift responses. In addition, the impacts of the contributing factors on the mode shift responses were very different between the congestion pricing and reward policies and between the migrant and resident millennial car travelers. The findings from the model estimation and descriptive statistics suggest that the implementation of pricing and reward policies, along with other unique regional (e.g., rapid increase in residential property price) and institutional (e.g., household registration system) characteristics, may potentially bring new challenges to migrant car travelers that may add to their economic burden and reduce their quality of life, particularly under congestion pricing policies. This study also provided insights for the design of future congestion pricing and reward policies and complementary measures that could address the travel needs of all travelers.

Qualitative Study in Identifying Reason that Motivate Malaysian Muslims in Writing Wasiyyah

While there has been increasing research interest in the Islamic estate planning areas, few studies have yielded detailed insights into the views and factors which motivate consumer in patronising wasiyyah. The qualitative research describes in this paper aimed to explore the views of Malaysian Muslims towards wasiyyah and to identify the reason that motivate Muslims in writing wasiyyah services in Malaysia. The study was framed by using the Self-Determination Theory (SDT) approach. This enabled us to examine the factors of motivation in two ways: intrinsic motivation and extrinsic motivation. The research participants for the current study were recruited on the basis of a consumer who has made wasiyyah in any institutions in Malaysia. Data analysis was undertaken using a framework approach and was informed by SDT. Consumers’ intrinsic motivation has been identified into three items: interest, religion, and family. Meanwhile, consumers’ extrinsic motivation also come out with three items: future rewards, awareness, and financial management. All these factors are the main thing that should be focused among marketers to come out with better strategies to increase more Muslims in writing wasiyyah as well as to avoid rising inward in frozen assets. Therefore, more research is required to examine how to raise awareness of Malaysian Muslims on the importance of wasiyyah.

Serotonergic projections to the orbitofrontal and medial prefrontal cortices differentially modulate waiting for future rewards.

Optogenetic activation of serotonergic neurons in the dorsal raphe nucleus (DRN) enhances patience when waiting for future rewards, and this effect is maximized by both high probability and high timing uncertainty of reward. Here, we explored which serotonin projection areas contribute to these effects using optogenetic axon terminal stimulation. We found that serotonin stimulation in the orbitofrontal cortex (OFC) is nearly as effective as that in the DRN for promoting waiting, while in the nucleus accumbens, it does not promote waiting. We also found that serotonin stimulation in the medial prefrontal cortex (mPFC) promotes waiting only when the timing of future rewards is uncertain. Our Bayesian decision model of waiting assumed that the OFC and mPFC calculate the posterior probability of reward delivery separately. These results suggest that serotonin in the mPFC affects evaluation of time committed, while serotonin in the OFC is responsible for overall valuation of delayed rewards.

Behavioral Activation Therapy on Reward Seeking Behaviors in Depressed People: An Experimental study.

Introduction: Behavioral activation therapy (BAT) is designed to help individuals’ approach and access sources of positive reinforcement in their life, which can serve a natural antidepressant function and efforts to help depressed people reengage in their life through focused activation strategies.Methods: In this study, 60 individuals were selected and randomly assigned to intervention and control groups. The intervention group received behavioral activation treatment, including eight treatment sessions and 5 weeks later, a follow-up study was conducted. The data were collected, using a Beck Depression Inventory-II and behavioral activation system (BIS)/ behavioral inhibition system (BAS) Carver and White questionnaires, before the intervention and after the intervention and five weeks after the intervention. SPSS 23 and analysis of covariance (ANCOVA) was used for data analysis.Results: Results showed a significant increase in the two components of the BAS including reward seeking and response to reward in the intervention group, which indicates an increase in positive affect and appetitive motivation for reward seeking and decreases the risk of depression. Also, the results showed a significant decrease in the BIS and depression in the intervention group, which indicates a decline in experiencing negative emotions.Conclusion: The implementation of BAT will cause depressed people to try to maximize future rewards and it’s effective in improving the reward seeking and reward response in depressed people because this treatment will increase the positive reinforcement and lead to learning cues that predict possible rewards in environments.