Future Rewards Research Articles

How firms can create delightful customer experience? Contrasting roles of future reward uncertainty

The recent emergence of customer experience management has sparked renewed interest in customer delight. Building upon social comparison theory and the uncertainty literature, this research examines a dual-path model (surprise and exclusivity) for creating delightful customer experience through bolstering self-enhancement and a contrasting moderating effect of future reward uncertainty in reward programs. We posit that a surprise (vs. exclusive) reward has a positive effect on customer delight through self-enhancement, but only when perceived uncertainty of receiving a similar reward in the future is high (vs. low). We verify our predictions with data from a laboratory experiment and a field survey, supplemented with multiple waves of customers’ actual behavioral loyalty data and confirm the impacts of customer delight on attitudinal (repeat and WOM intentions) and behavioral (both short- and long-term) loyalty. Our findings offer companies actionable managerial guidelines for building customer loyalty through creating delightful customer experience.

The associations between posttraumatic stress disorder and delay discounting, future orientation, and reward availability: A behavioral economic model.

The theoretical framework of behavioral economics, a metatheory that integrates operant learning and economic theory, has only recently been applied to posttraumatic stress disorder (PTSD). A behavioral economic theory of PTSD reflects an expansion of prior behavioral conceptualization of PTSD, which described PTSD in terms of respondent and operant conditioning. In the behavioral economic framework of PTSD, negatively reinforced avoidance behavior is overvalued, in part due to deficits in environmental reward, and may be conceptualized as a form of reinforcer pathology (i.e., excessive preference for and valuation of an immediate reinforcer). We investigated cross-sectional relationships between PTSD severity and several constructs rooted in this behavioral economic framework, including future orientation, reward availability, and delay discounting in a sample of 110 military personnel/veterans (87.2% male) who had served combat deployments following September 11, 2001. Total PTSD severity was inversely related to environmental reward availability, β = -.49, ΔR2 = 0.24, p < .001; hedonic reward availability, β = -.32, ΔR2 = 0.10, p = .001; and future orientation, β = -.20, ΔR2 = 0.04, p = .032, but not delay discounting, r = -.05, p = .633. An examination of individual symptom clusters did not suggest that avoidance symptoms were uniquely associated with these behavioral economic constructs. The findings offer support for a behavioral economic model of PTSD in which there is a lack of positive reinforcement as well as a myopic focus on the present.

Structure-function coupling within the reward network in preschool children predicts executive functioning in later childhood

Early differences in reward behavior have been linked to executive functioning development. The nucleus accumbens (NAc) and orbitofrontal cortex (OFC) are activated by reward-related tasks and identified as key nodes of the brain circuit that underlie reward processing. We aimed to investigate the relation between NAc-OFC structural and functional connectivity in preschool children, as well as associations with future reward sensitivity and executive function. We showed that NAc-OFC structural and functional connectivity were not significantly associated in preschool children, but both independently predicted sensitivity to reward in males in a left-lateralized manner. Moreover, significant NAc-OFC structure-function coupling was only found in individuals who performed poorly on executive function tasks in later childhood, but not in the middle- and high-performing groups. As structure-function coupling is proposed to measure functional specialization, this finding suggests premature functional specialization within the reward network, which may impede dynamic communication with other regions, affects executive function development. Our study also highlights the utility of multimodal imaging data integration when studying the effects of reward network functional flexibility in the preschool age, a critical period in brain and executive function development.

Planning for potential: efficient safe reinforcement learning

Deep reinforcement learning (DRL) has shown remarkable success in artificial domains and in some real-world applications. However, substantial challenges remain such as learning efficiently under safety constraints. Adherence to safety constraints is a hard requirement in many high-impact application domains such as healthcare and finance. These constraints are preferably represented symbolically to ensure clear semantics at a suitable level of abstraction. Existing approaches to safe DRL assume that being unsafe leads to low rewards. We show that this is a special case of symbolically constrained RL and analyze a generic setting in which total reward and being safe may or may not be correlated. We analyze the impact of symbolic constraints and identify a connection between expected future reward and distance towards a goal in an automaton representation of the constraints. We use this connection in an algorithm for learning complex behaviors safely and efficiently. This algorithm relies on symbolic reasoning over safety constraints to improve the efficiency of a subsymbolic learner with a symbolically obtained measure of progress. We measure sample efficiency on a grid world and a conversational product recommender with real-world constraints. The so-called Planning for Potential algorithm converges quickly and significantly outperforms all baselines. Specifically, we find that symbolic reasoning is necessary for safety during and after learning and can be effectively used to guide a neural learner towards promising areas of the solution space. We conclude that RL can be applied both safely and efficiently when combined with symbolic reasoning.

I’m not sure what to think about them: Confronting naive present bias in a dynamic threshold public goods game

This paper explores the implications of naive present bias in a simple, dynamic threshold public goods game. Our analysis demonstrates how equilibrium behavior and outcomes can be highly dependent on the “direction” with which players reassess their naive prior beliefs (including higher-order beliefs) regarding both players’ present biases. In some situations, a naive player may actually become more inclined to exert effort in pursuit of a future reward upon discovering their own present bias.

Joint Beamforming, Power Allocation, and Splitting Control for SWIPT-Enabled IoT Networks with Deep Reinforcement Learning and Game Theory.

Future wireless networks promise immense increases on data rate and energy efficiency while overcoming the difficulties of charging the wireless stations or devices in the Internet of Things (IoT) with the capability of simultaneous wireless information and power transfer (SWIPT). For such networks, jointly optimizing beamforming, power control, and energy harvesting to enhance the communication performance from the base stations (BSs) (or access points (APs)) to the mobile nodes (MNs) served would be a real challenge. In this work, we formulate the joint optimization as a mixed integer nonlinear programming (MINLP) problem, which can be also realized as a complex multiple resource allocation (MRA) optimization problem subject to different allocation constraints. By means of deep reinforcement learning to estimate future rewards of actions based on the reported information from the users served by the networks, we introduce single-layer MRA algorithms based on deep Q-learning (DQN) and deep deterministic policy gradient (DDPG), respectively, as the basis for the downlink wireless transmissions. Moreover, by incorporating the capability of data-driven DQN technique and the strength of noncooperative game theory model, we propose a two-layer iterative approach to resolve the NP-hard MRA problem, which can further improve the communication performance in terms of data rate, energy harvesting, and power consumption. For the two-layer approach, we also introduce a pricing strategy for BSs or APs to determine their power costs on the basis of social utility maximization to control the transmit power. Finally, with the simulated environment based on realistic wireless networks, our numerical results show that the two-layer MRA algorithm proposed can achieve up to 2.3 times higher value than the single-layer counterparts which represent the data-driven deep reinforcement learning-based algorithms extended to resolve the problem, in terms of the utilities designed to reflect the trade-off among the performance metrics considered.

The Resting-State Neural Network of Delay Discounting.

Delay discounting is a common phenomenon in daily life, which refers to the subjective value of a future reward decreasing as a function of time. Previous studies have identified several cortical regions involved in delay discounting, but the neural network constructed by the cortical regions of delay discounting is less clear. In this study, we employed resting-state functional magnetic resonance imaging (RS-fMRI) to measure the spontaneous neural activity in a large sample of healthy young adults and used the Monetary Choice Questionnaire to directly measure participants’ level of delay discounting. To identify the neural network of delay discounting at rest, we used an individual difference approach to explore brain regions whose spontaneous activities were related to delay discounting across the whole brain. Then, these brain regions served as seeds to identify the neural network of delay discounting. We found that the fractional amplitude of low-frequency fluctuations (fALFF) of the left insula were positively correlated to delay discounting. More importantly, its connectivity to the anterior cingulate cortex was read out for participants’ behavioral performance in the task of delay discounting. In short, our study provides empirical evidence that insula-anterior cingulate cortex connectivity may serve as a part of the neural network for delay discounting.

Present bias and externalities: Can government intervention raise welfare?

AbstractQuasi‐hyperbolic discounted preferences imply that consumers overemphasize immediate current rewards and overlook future ones (they have a “bias for the present”). Within this context the literature has emphasized that the misalignment between immediate and future rewards can be rectified by government policy. Importantly, it has also been shown that intervention by a government that shares the same biased intertemporal preferences with consumers does not deliver welfare improvements. Focusing on the latter, this paper identifies conditions under which, in the presence of quasi‐hyperbolic preferences and a market imperfection (which takes the form of a negative externality), intervention by a present‐biased government is welfare enhancing. This is the case if the market imperfection is sufficiently strong or the consumers' bias for the present is weak.

Self-Optimizing Data Offloading in Mobile Heterogeneous Radio-Optical Networks: A Deep Reinforcement Learning Approach

In addition to the exploration of more spectrum at high-frequency bands, next-generation wireless networks will witness an intelligent convergence of radio frequency (RF) and non-RF links such as optical and visible light communication. Optical attocell (OAC) networks provide an additional layer to RF-based wireless networks with gigabit-per-second data transmission rate and centimeter-level location accuracy. However, the directionality, line-of-sight constraints, as well as strong sensitivity to the location and orientation of user terminals challenge the stringent requirements for throughput and latency. In this article, we consider mobile heterogeneous networks (HetNets) incorporating indoor OAC with femtocells and macrocells to provide a low-cost and energy-efficient solution. The HetNets solution satisfies diverse service requirements in terms of user-experienced data rate, mobility, latency, accuracy, and security in the Internet of Things. To support seamless connectivity and optimal resource allocation in the proposed HetNets with mobility awareness, handover in dynamic environments needs to be addressed efficiently. Incorporating rich environmental parameters into such a decision making problem facilitates the self-optimization process, but extensively expands the state space. To achieve a fast convergence speed, a deep reinforcement learning approach is proposed to optimize the handover parameters (e.g., time-to-trigger and hysteresis margin). This is a model-free and off-policy reinforcement setting that trains and employs a deep neural network to predict future rewards for successions of states and actions. Thus, the optimal parameters are obtained by selecting the best actions to take. Through numerical simulation and performance analysis, we discover the gain from enriching the state space and the adaptability of the system to dynamic environments.

Temporal discounting across adulthood: A systematic review and meta-analysis.

A number of developmental theories have been proposed that make differential predictions about the links between age and temporal discounting, or the devaluation of future rewards. Most empirical studies examining adult age differences in temporal discounting have relied on economic intertemporal choice tasks, which pit choosing a smaller, sooner monetary reward against choosing a larger, later one. Although initial studies using these tasks suggested older adults discount less than younger adults, follow-up studies provided heterogeneous, and thus inconclusive, results. Using an open science approach, we test the replicability of adult age differences in temporal discounting by conducting a preregistered systematic literature search and meta-analysis of adult age differences in intertemporal choice tasks. Across 37 cross-sectional studies (Total N = 104,737), a planned meta-analysis found no sizeable relation between age and temporal discounting, r = -0.068, 95% CI [-0.170, 0.035]. We also found little evidence of publication bias or p-hacking. Exploratory analyses of moderators found no effect of research design (e.g., extreme-group vs. continuous age), incentives (hypothetical vs. real rewards), duration of delay (e.g., days, weeks, months, or years), or quantification of discounting behavior (e.g., proportion of immediate choices vs. parameters from computational modeling). Additional analyses of 12 participant-level data sets found little support for a nonlinear relation between age and temporal discounting across adulthood. Overall, the results suggest that younger, middle-aged, and older adults show similar preferences for smaller, sooner over larger, later rewards. We provide recommendations for future empirical work on temporal discounting across the adult life span. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

Percent framing attenuates the magnitude effect in a preference-matching task of intertemporal choice.

Research in intertemporal decisions shows that people value future gains less than equivalent but immediate gains by a factor known as the discount rate (i.e., people want a premium for waiting to receive a reward). A robust phenomenon in intertemporal decisions is the finding that the discount rate is larger for small gains than for large gains, termed the magnitude effect. However, the psychological underpinnings of this effect are not yet fully understood. One explanation proposes that intertemporal choices are driven by comparisons of features of the present and future choice options (e.g., information on rewards). According to this explanation, the hypothesis is that the magnitude effect is stronger when the absolute difference between present and future rewards is emphasized, compared to when their relative difference is emphasized. However, this hypothesis has only been tested using one task (the two-choice paradigm) and only for gains (i.e., not losses). It's therefore unclear whether the findings that support the hypothesis can be generalized to different methodological paradigms (e.g., preference matching) and to the domain of losses. To address this question, we conducted experiments using the preference-matching method whereby the premium amounts that people could ask for were framed in terms of either currencies (emphasizing absolute differences) or percentages (emphasizing relative differences). We thus tested the robustness of the evidence in support of the hypothesis that percent framing, relative to currency framing, attenuates the magnitude effect in the domain of gains (Studies 1, 2, and 3) and in the domain of losses (Study 1, 3, and 4). The data were heavily skewed and the assumption of equal variances was violated. Therefore, in place of parametric statistical tests, we calculated and interpreted parametric and nonparametric standardized and unstandardized effect size estimates and their confidence intervals. Overall, the results support the hypothesis.

Too much or too little information: how unknown uncertainty fuels time inconsistency

Under uncertainty, there is considerable heterogeneity in expectations of results, and the outcome of each choice is a reflection of those expectations. This study aims to understand the role of subjective probabilistic inference in updating information for decision-making procedures under uncertainty. We show that adding uncertainty of trade-offs in decision-making criteria induces more inconsistent present preferences. We find that subjective probabilistic inference results in different levels of information acquisition, which plays a central role in many everyday cases of forecasting. The result of forecasting exerts substantive constraints on cognitive processes and shapes a type of restriction or stimulus in decision-making procedures. As uncertainty increases, generated fear of losses turns into an obstacle to the information acquisition process, and especially participants with low probabilistic inference tend to overestimate or underestimate future unknown rewards. In addition, our experiment shows that risk preference does not play a key role in decision-making procedures under unknown uncertainty. This finding is an experimental manifestation of Knight’s argument (Risk, uncertainty, and profit, Houghton Mifflin, Boston, 1921), which explains unknown uncertainty, and shows the relationship between cognitive ability and time inconsistency.

Prenatal opioid exposure reprograms the behavioural response to future alcohol reward.

As the opioid crisis has continued to grow, so has the number of infants exposed to opioids during the prenatal period. A growing concern is that prenatal exposure to opioids may induce persistent neurological changes that increase the propensity for future addictions. Although alcohol represents the most likely addictive substance that the growing population of prenatal opioid exposed will encounter as they mature, no studies to date have examined the effect of prenatal opioid exposure on future sensitivity to alcohol reward. Using a recently developed mouse model of prenatal methadone exposure (PME), we investigated the rewarding properties of alcohol and alcohol consumption in male and female adolescent PME and prenatal saline exposed (PSE) control animals. Conditioned place preference to alcohol was disrupted in PME offspring in a sex-dependent manner with PME males exhibiting resistance to the rewarding properties of alcohol. Repeated injections of alcohol revealed enhanced sensitivity to the locomotor-stimulating effects of alcohol specific to PME females. PME males consumed significantly more alcohol over 4 weeks of alcohol access relative to PSE males and exhibited increased resistance to quinine-adulterated alcohol. Further, a novel machine learning model was developed to employ measured differences in alcohol consumption and drinking microstructure to reliably predict prenatal exposure. These findings indicate that PME alters the sensitivity to alcohol reward in adolescent mice in a sex-specific manner and suggests prenatal opioid exposure may induce persistent effects on reward neurocircuitry that can reprogram offspring behavioural response to alcohol later in life.

Trial-wise exposure to visual emotional cues increases physiological arousal but not temporal discounting.

Humans and many animals devalue future rewards as a function of time (temporal discounting). Increased discounting has been linked to various psychiatric conditions, including substance-use-disorders, behavioral addictions, and obesity. Despite its high intra-individual stability, temporal discounting is partly under contextual control. One prominent manipulation that has been linked to increases in discounting is the exposure to highly arousing appetitive cues. However, results from trial-wise cue exposure studies appear highly mixed, and changes in physiological arousal were not adequately controlled. Here we tested the effects of appetitive (erotic), aversive, and neutral visual cues on temporal discounting in 35 healthy male participants. The contribution of single-trial physiological arousal was assessed using comprehensive monitoring of autonomic activity (pupil size, heart rate, electrodermal activity). Physiological arousal was elevated following aversive and in particular erotic cues. In contrast to our pre-registered hypothesis, steepness of temporal discounting was not significantly affected by emotional cues of either valence. Aversive cues tended to increase decision noise. Computational modeling revealed that trial-wise arousal only accounted for minor variance over and above aversive and erotic condition effects, arguing against a general effect of physiological arousal on temporal discounting.

Investiture rules and the formation and type of government in Israel and Italy

ABSTRACT This paper investigates the seemingly puzzling situation in which legislators vote against a government of which they will become a part. Why are legislators from the coalition parties not fully disciplined in the investiture vote? Is it because of the type of government when the characteristics of the investiture vote is similar (a simple majority and an ex-post vote)? Using Israel and Italy as case studies, we demonstrate that when the type of government is an oversized coalition and the investiture consists of a simple majority and an ex-post vote (as a constant), legislators will be willing to vote against their party. We argue that they do so to express their disappointment with the positions they are offered and/or the policies that the coalition is adopting. Given the simple majority and oversized coalition, they are risking little and potentially reaping future rewards from their actions.

Social trust and the discounting of gains and losses in prolonged grief

Prolonged grief (PG) is a potentially debilitating consequence of bereavement. Previous studies have associated PG severity with relative deficits in key future-oriented cognitive processes, including the discounting of future rewards. Individual differences in the rate at which one devalues the future have been associated with a range of important economic and health outcomes. To extend understanding of the mechanisms underlying discounting behaviour in PG, this study investigated discounting behaviour using an online paradigm where future consequences were made salient at the time of the decision. Ninety-four bereaved participants made choices to accept or reject offers that involved either an immediate gain and a long-term loss, or an immediate loss and a longer-term gain. Lower levels of social trust and higher PG severity were associated with greater discounting of future losses. Social trust but not PG severity was related to discounting of future gains. Results replicate and extend on previous findings in PG, and suggest potentially modifiable factors to reduce the biases in decision making processes that may contribute to ongoing dysfunction in PG.

A Deep Reinforcement Learning Approach to Sensor Placement under Uncertainty

Optimal sensor placement is critical for enhancing the effectiveness of monitoring dynamical systems. Deterministic solutions do not reflect the effects of input and parameter uncertainty on the sensor placement. Using a Markov decision process (MDP) and a sensor placement agent, this study proposes a stochastic approach to maximize the gain from placing a fixed number of sensors within the system. Utilizing Deep Reinforcement Learning (DRL), the agent is trained by collecting interactive samples within the environment, which uses an information-theoretic reward function that is a measure, based on Shannon entropy, of the identifiability of the model parameters. The goal of the agent is to maximize its expected future reward by selecting, at each step, the action (placing a sensor) that provides the most information. This framework is validated using a synthetic model of a base-isolated structure. To consider the existing uncertainty in the parameters, a prior probability distribution is chosen (e.g., based on expert judgement or preliminary study) for each model parameter. Further, a probabilistic model for the input is used to reflect input variability. In a Deep Q-network, a type of DRL algorithm, the agent learns a mapping from states (i.e., sensor configurations) to the ”quality” of each action at that state, called ”Q-values”. This network is trained using samples of state, action, and reward by interacting with the environment. The modular property of the framework and the function approximation used in this study makes it scalable to complex real-world applications of sensor placement problems in the presence of uncertainties.

Reinforced optimal control

Least squares Monte Carlo methods are a popular numerical approximation method for solving stochastic control problems. Based on dynamic programming, their key feature is the approximation of the conditional expectation of future rewards by linear least squares regression. Hence, the choice of basis functions is crucial for the accuracy of the method. Earlier work by some of us [Belomestny, Schoenmakers, Spokoiny, Zharkynbay. Commun.~Math.~Sci., 18(1):109-121, 2020] proposes to \emph{reinforce} the basis functions in the case of optimal stopping problems by already computed value functions for later times, thereby considerably improving the accuracy with limited additional computational cost. We extend the reinforced regression method to a general class of stochastic control problems, while considerably improving the method's efficiency, as demonstrated by substantial numerical examples as well as theoretical analysis.

A meta-analysis of the effects of episodic future thinking on delay discounting

Delay discounting (DD) refers to the phenomenon in which the subjective value of future rewards is reduced over time. There are individual differences in the DD rate, and increased discounting has been observed in those with various psychiatric disorders. Episodic future thinking (EFT) is the act of vividly imagining events that may happen in the future. Studies have shown that EFT could reduce DD, although inconsistent results have been reported. The aim of this meta-analysis was to clarify the efficacy with which EFT reduces DD and to identify potential moderators. Forty-seven studies (including 63 contrasts) were included in the final analysis. EFT was found to significantly reduce DD (Hedges’ g = 0.52). Moderator analysis showed that positive EFT ( g = 0.64) was more effective in reducing DD than EFT with the valence not specifically mentioned ( g = 0.28) and EFT with neutral or negative valence ( g = –0.03). In addition, several factors related to the control task and DD task were related to the efficacy of EFT to reduce DD. These findings have implications for using EFT to reduce DD in the future.

Observational learning promotes hippocampal remote awake replay toward future reward locations

The neural circuit mechanisms underlying observational learning, learning through observing the behavior of others, are poorly understood. Hippocampal place cells are important for spatial learning, and awake replay of place cell patterns is involved in spatial decisions. Here we show that, in observer rats learning to run a maze by watching a demonstrator's spatial trajectories from a separate nearby observation box, place cell patterns during self-running in the maze are replayed remotely in the box. The contents of the remote awake replay preferentially target the maze's reward sites from both forward and reverse replay directions and reflect the observer's future correct trajectories in the maze. In contrast, under control conditions without a demonstrator, the remote replay is significantly reduced, and the preferences for reward sites and future trajectories disappear. Our results suggest that social observation directs the contents of remote awake replay to guide spatial decisions in observational learning.