Future Rewards Research Articles

Reinforcement Learning With Data Envelopment Analysis and Conditional Value-At-Risk for the Capacity Expansion Problem

The capacity expansion problem is solved by accurately measuring the existing demand-supply mismatch and controlling the emissions output, considering multiple objectives, specific constraints, resource diversity, and resource allocation. This article proposes a reinforcement learning (RL) framework embedded with data envelopment analysis (DEA) to generate the optimal policy and guide the productivity improvement. The proposed framework uses DEA to evaluate efficiency and effectiveness for reward estimation in RL, and also assesses conditional value-at-risk to characterize the risk-averse capacity decision. Instead of focusing on short-term fluctuations in demand, RL optimizes the expected future reward with sequential capacity decisions over time. An empirical study of U.S. power generation validates the proposed framework and provides the managerial implications to policy makers. The results show that the RL agent can successfully learn the optimal policy through observing the interactions between the agent and the environment, and suggest the capacity adjustment that can improve efficiency by 8.3% and effectiveness by 0.9%. We conclude that RL complements productivity analysis, and emphasizes ex-ante planning over ex-post evaluation.

Affect-congruent attention modulates generalized reward expectations.

Positive and negative affective states are respectively associated with optimistic and pessimistic expectations regarding future reward. One mechanism that might underlie these affect-related expectation biases is attention to positive- versus negative-valence features (e.g., attending to the positive reviews of a restaurant versus its expensive price). Here we tested the effects of experimentally induced positive and negative affect on feature-based attention in 120 participants completing a compound-generalization task with eye-tracking. We found that participants' reward expectations for novel compound stimuli were modulated in an affect-congruent way: positive affect induction increased reward expectations for compounds, whereas negative affect induction decreased reward expectations. Computational modelling and eye-tracking analyses each revealed that these effects were driven by affect-congruent changes in participants' allocation of attention to high- versus low-value features of compounds. These results provide mechanistic insight into a process by which affect produces biases in generalized reward expectations.

OLDER ADULTS’ PERCEPTIONS OF HOPE REGARDING THEIR ABILITY TO MAKE DIET AND EXERCISE CHANGES

Abstract Successful aging is facilitated by healthy diet and exercise levels; however, older adults may face barriers to obtaining these. One key motivator for improving older adults’ diet and exercise may be perceptions of hope. Persuasive Hope Theory (PHT) posits that hope motivates behavior by focusing peoples’ thoughts on opportunities for actions that lead to future rewards. Research suggests hope is particularly motivational for older adults, yet little work has examined factors that lead older adults to feel hope. In line with PHT, this project examined whether older adults’ perceptions that increasing fruit and vegetable intake (FVI) and exercise are important, possible, congruent with their goals (i.e. goal congruence), and will lead to a better future (i.e., future expectation), influence feelings of hope. A convenience sample of participants (N=58) aged 57-87 (M=71.98, SD =8.74) living in urban Alaska completed a questionnaire as part of a larger study. A multiple regression analysis revealed that importance (M=4.40, SD=0.67) and future expectation (M=4.47, SD=0.66) for increasing FVI significantly predicted feeling hopeful about increasing FVI (M=4.40, SD=0.75), but perceptions that increasing FVI is possible (M=4.26, SD=0.74) and goal congruent (M=4.34, SD=0.64) did not (F(4, 53)=11.29, p<.05, R2=.46). The PHT variables for increasing exercise did not significantly predict feeling hopeful about increasing exercise. These results suggest PHT, specifically importance and future expectation, may be used to strategically design messages to increase older adults’ hopeful feelings for increasing FVI; however, more research is needed due to the limitations of the small convenience sample.

Exposure to immersive virtual environments decreases present bias

In a Cave Automatic Virtual Environment, we conducted an experiment to investigate the influence of future self-continuity on the delay of immediate monetary gratification. By utilizing immersive virtual environments, we aimed to accentuate the role of environmental cues in shaping behavior. Our findings revealed that these environmental stimuli had a significant impact, leading to a higher likelihood of selecting a larger delayed option. This effect was attributed to the enhanced desirability of future gratification and a reduction in the influence of present bias on future rewards. These results underscore the utility of virtual environments as a valuable tool for both testing and elucidating intertemporal choice behavior.

Effort Foraging Task reveals positive correlation between individual differences in the cost of cognitive and physical effort in humans

Effort-based decisions, in which people weigh potential future rewards against effort costs required to achieve those rewards involve both cognitive and physical effort, though the mechanistic relationship between them is not yet understood. Here, we use an individual differences approach to isolate and measure the computational processes underlying effort-based decisions and test the association between cognitive and physical domains. Patch foraging is an ecologically valid reward rate maximization problem with well-developed theoretical tools. We developed the Effort Foraging Task, which embedded cognitive or physical effort into patch foraging, to quantify the cost of both cognitive and physical effort indirectly, by their effects on foraging choices. Participants chose between harvesting a depleting patch, or traveling to a new patch that was costly in time and effort. Participants’ exit thresholds (reflecting the reward they expected to receive by harvesting when they chose to travel to a new patch) were sensitive to cognitive and physical effort demands, allowing us to quantify the perceived effort cost in monetary terms. The indirect sequential choice style revealed effort-seeking behavior in a minority of participants (preferring high over low effort) that has apparently been missed by many previous approaches. Individual differences in cognitive and physical effort costs were positively correlated, suggesting that these are perceived and processed in common. We used canonical correlation analysis to probe the relationship of task measures to self-reported affect and motivation, and found correlations of cognitive effort with anxiety, cognitive function, behavioral activation, and self-efficacy, but no similar correlations with physical effort.

OFC neurons do not represent the negative value of a conditioned inhibitor

The orbitofrontal cortex (OFC) is often proposed to function as a value integrator; however, alternative accounts focus on its role in representing associative structures that specify the probability and sensory identity of future outcomes. These two accounts make different predictions about how this area should respond to conditioned inhibitors of reward, since in the former, neural activity should reflect the negative value of the inhibitor, whereas in the latter, it should track the estimated probability of a future reward based on all cues present. Here, we assessed these predictions by recording from small groups of neurons in the lateral OFC of rats during training in a conditioned inhibition design. Rats showed negative summation when the inhibitor was compounded with a novel excitor, suggesting that they learned to respond to the conditioned inhibitor appropriately. Against this backdrop, we found unit and population responses that scaled with expected reward value on excitor + inhibitor compound trials. However, the responses of these neurons did not differentiate between the conditioned inhibitor and a neutral cue when both were presented in isolation. Further, when the ensemble patterns were analyzed, activity to the conditioned inhibitor did not classify according to putative negative value. Instead, it classified with a same-modality neutral cue when presented alone and as a unique item when presented in compound with a novel excitor. This pattern of results supports the notion that OFC encodes a model of the causal structure of the environment rather than either the modality or the value of cues.

1110. Why has Utilization of Novel Gram-negative Antibiotics been Sluggish in U.S. Hospitals? A Retrospective Cohort Study to Inform Future Antibiotic Development and Market Entry Rewards

Abstract Background Sluggish uptake of novel gram-negative antibiotics has jeopardized their future development and supply potentially risking patient lives. Understanding underutilization patterns might better inform unmet needs and market entry rewards. We studied patterns and drivers of use of recently U.S. FDA-approved novel gram-negative agents in patients with gram-negative infections (GNI) displaying difficult-to-treat resistance (DTR; resistance to all first line B-lactams including carbapenems, and fluoroquinolones). Methods Using the all-payer PINC-AI database, quarterly percent change in the post-approval use of seven gram-negative agents and secular trends in colistin use were calculated. For inpatients with DTR GNI at microbiology reporting hospitals, we compared novel vs older/reserve agent use by infection site and pathogen. At hospitals that use novel agents, generalized linear mixed models with weighting identified patient and hospital factors associated with the use of any novel (vs older/reserve) agents as targeted therapy against DTR GNI. Results At 619 hospitals, while colistin use declined between 2016Q1 and 2021Q2, post-approval use grew variably for each novel agent (Fig1); compared to the first novel anti–Klebsiella pneumoniae carbapenemase (KPC) agent ceftazidime-avibactam (approved 2015), usage was relatively sluggish for subsequently approved anti-KPC-agents meropenem-vaborbactam and imipenem-cilastatin-relebactam and zero for plazomicin. At 302 microbiology reporting hospitals, 44.4% of 2830 overall DTR cases including 70.2% of 627 DTR-A. baumannii cases received only older/reserve agents (Fig2). The odds of use of novel (vs older/novel) agents were greater in patients displaying hemodynamic instability and higher comorbidity burden, but similar across race categories and insurance status, and lower at Western(vs Midwestern) hospitals (Table1). Conclusion Nearly half the patients in US hospital with DTR-GNIs are still receiving targeted therapy with older/reserve agents. While the value of novel agents is indicated by clinicians preferring them over older/reserve agents in sicker patients, future agents with distinctly novel mechanisms and activity against DTR-A. baumannii are needed to bridge the utilization gap. Disclosures Morgan Walker, MD, Cytovale: Advisor/Consultant

Feedback-related brain activity in individual decision: evidence from a gambling EEG study.

In this study, based on scalp electroencephalogram (EEG), we conducted cortical source localization and functional network analyses to investigate the underlying mechanism explaining the decision processes when individuals anticipate maximizing gambling benefits, particularly in situations where the decision outcomes are inconsistent with the profit goals. The findings shed light on the feedback monitoring process, wherein incongruity between outcomes and gambling goals triggers a more pronounced medial frontal negativity and activates the frontal lobe. Moreover, long-range theta connectivity is implicated in processing surprise and uncertainty caused by inconsistent feedback conditions, while middle-range delta coupling reflects a more intricate evaluation of feedback outcomes, which subsequently modifies individual decision-making for optimizing future rewards. Collectively, these findings deepen our comprehension of decision-making under circumstances where the profit goals are compromised by decision outcomes and provide electrophysiological evidence supporting adaptive adjustments in individual decision strategies to achieve maximum benefit.

A Novel Blockchain-Reinforcement Learning Framework for Securing Wireless Sensor Networks with Energy Efficiency

This paper introduces a novel approach to Reinforcement Learning (RL), focusing on the development and implementation of a Q-learning based algorithm. Reinforcement Learning, a critical branch of machine learning, enables agents to make decisions by interacting with their environment and learning from the consequences of their actions. Our study emphasizes the Q-learning model, a popular, model-free, off-policy algorithm that offers a robust framework for agents to learn optimal strategies in diverse settings. By iteratively updating the action-value function (Q-function) based on observed rewards and future reward estimations, our algorithm aims to achieve efficient learning and decision-making. This work contributes to the field by providing a detailed algorithmic structure, complete with mathematical formulations, that facilitates a deeper understanding of the Q-learning process and its practical applications in various domains.

Reframing dopamine: A controlled controller at the limbic-motor interface.

Pavlovian influences notoriously interfere with operant behaviour. Evidence suggests this interference sometimes coincides with the release of the neuromodulator dopamine in the nucleus accumbens. Suppressing such interference is one of the targets of cognitive control. Here, using the examples of active avoidance and omission behaviour, we examine the possibility that direct manipulation of the dopamine signal is an instrument of control itself. In particular, when instrumental and Pavlovian influences come into conflict, dopamine levels might be affected by the controlled deployment of a reframing mechanism that recasts the prospect of possible punishment as an opportunity to approach safety, and the prospect of future reward in terms of a possible loss of that reward. We operationalize this reframing mechanism and fit the resulting model to rodent behaviour from two paradigmatic experiments in which accumbens dopamine release was also measured. We show that in addition to matching animals' behaviour, the model predicts dopamine transients that capture some key features of observed dopamine release at the time of discriminative cues, supporting the idea that modulation of this neuromodulator is amongst the repertoire of cognitive control strategies.

Blockchain and International Trade: An Empirical Analysis

Blockchain has great potential in international trade because it has the possibility to disrupt bureaucracy, simplify the task of monitoring commodities by removing vast paper trails, and reduce losses due to delayed payments (Morabito, 2017). By analysing time series data between 2012 and 2021, this study will analyse the deployment and application of distributed electronic ledgers in global trade. The study topic is justified since there is a limited knowledge on how blockchain may affect international trade. Furthermore, there is a lack of understanding regarding its relevance in facilitating trade finance. To put it another way, blockchain technology has a significant impact on global trade. Additionally, we demonstrate that per capita GDP has a significant positive impact on international trade. Our findings are congruent with those of Derindag et al. (2020), Keller (2021), and Yu (2021). This study advises decision-makers, notably executives and customers, to validate blockchain's potential advantages and embrace its application for international trade objectives, resulting in higher global trade flows. Companies who commit to embracing blockchain technology will reap huge future rewards

PENGARUH PENGETAHUAN INVESTASI, MOTIVASI INVESTASI, LOCUS OF CONTROL, DAN RISK TOLERANCE TERHADAP KEPUTUSAN INVESTASI MAHASISWA ADMINISTRASI BISNIS UNIVERSITAS 17 AGUSTUS 1945 SURABAYA

Investing entails laying away a particular amount of money in the anticipation of future financial reward. Some of the factors that impact students' investing decisions include investment motivation, investment knowledge, locus of control, and risk tolerance. The goal of this study is to see if investing knowledge, investment motive, locus of control, and risk tolerance impact the investment decisions of Business Administration students at the University of 17 August 1945 Surabaya. This study employed the quantitative research approach. This research included 109 Business Administration students from the University of 17 August 1945 Surabaya. In this investigation, random sampling was used. According to the findings of the study, financial knowledge, locus of control, and risk tolerance all have a substantial partial influence on investment decisions. However, investing motive has little impact on investment decisions. As a result, investment knowledge, investment motive, locus of control, and risk tolerance all have a large influence on investing decisions.

Cephalopod learning and memory

Cephalopod molluscs are renowned for their unique central nervous system - a donut-shaped brain organised around the oesophagus. This brain supports sophisticated learning and memory abilities. Between the 1950s and 1980s, these cognitive abilities were extensively studied in octopus (Figure 1A) - a now leading model for the study of memory and its neural substrates (approximately 200 papers during this period). The focus on octopus learning and memory was mainly due to their curious nature and the fact that they adapt to laboratory-controlled conditions, making them easy to test and maintain in captivity. Research on cephalopod cognition began to widen in the late 20th century, when scientists started focusing on other coleoid cephalopods (i.e., cuttlefish and squid) (Figure 1B,C), and not just on associative learning and memory per se, but other more complex aspects of cognition such as episodic-like memory (the ability to remember the what, where, and when of a past event), source memory (the retrieval of contextual details from a memory), and self-control (the ability to inhibit an action in the present to gain a more valuable future reward). Attention broadened further over the last two decades to focus on the shelled cephalopods - the nautiloids (Figure 1D). The nautiloids have relatively primitive brains compared to their soft-bodied cousins (octopus, cuttlefish, and squid) but research shows that they are still able to comparatively succeed in some cognitive tasks. In this primer, we will provide a general description of the types of memory studied in cephalopods, and discuss learning and memory experiments that address the main challenges cephalopods face during their daily lives: navigation, timing, and food selection. Determining the type of information cephalopods learn and remember and whether they use such information to overcome ecological challenges will highlight why these invertebrates evolved large and sophisticated brains.

Selective encoding of reward predictions and prediction errors by globus pallidus subpopulations

Basal ganglia (BG) circuits help guide and invigorate actions using predictions of future rewards (values). Within the BG, the globus pallidus pars externa (GPe) may play an essential role in aggregating and distributing value information. We recorded from the GPe in unrestrained rats performing both Pavlovian and instrumental tasks to obtain rewards and distinguished neuronal subtypes by their firing properties across the wake/sleep cycle and optogenetic tagging. In both tasks, the parvalbumin-positive (PV+), faster-firing "prototypical" neurons showed strong, sustained modulation by value, unlike other subtypes, including the "arkypallidal" cells that project back to striatum. Furthermore, we discovered that a distinct minority (7%) of GP cells display slower, pacemaker-like firing and encode reward prediction errors (RPEs) almost identically to midbrain dopamine neurons. These cell-specific forms of GPe value representation help define the circuit mechanisms by which the BG contribute to motivation and reinforcement learning.

Emergence of belief-like representations through reinforcement learning.

To behave adaptively, animals must learn to predict future reward, or value. To do this, animals are thought to learn reward predictions using reinforcement learning. However, in contrast to classical models, animals must learn to estimate value using only incomplete state information. Previous work suggests that animals estimate value in partially observable tasks by first forming "beliefs"-optimal Bayesian estimates of the hidden states in the task. Although this is one way to solve the problem of partial observability, it is not the only way, nor is it the most computationally scalable solution in complex, real-world environments. Here we show that a recurrent neural network (RNN) can learn to estimate value directly from observations, generating reward prediction errors that resemble those observed experimentally, without any explicit objective of estimating beliefs. We integrate statistical, functional, and dynamical systems perspectives on beliefs to show that the RNN's learned representation encodes belief information, but only when the RNN's capacity is sufficiently large. These results illustrate how animals can estimate value in tasks without explicitly estimating beliefs, yielding a representation useful for systems with limited capacity.

Draw on advantages and avoid disadvantages by making a multi-step prediction

Reinforcement learning learns about the environment through the process of exploration, and the sufficient information collected during the interaction helps the agent to predict the future situation. However, uncontrolled exploration may cause the agent to stray into dangerous regions of the environment, leading to bad decisions and impairing the agent’s performance. In order to address the issue, a framework, referred to as the policy guided by multi-step prediction (PGMP), is proposed. PGMP utilizes a curiosity mechanism based on multi-step prediction errors to stimulate exploration. To encourage the agent to explore safe or task-relevant areas, a safety bonus model is designed to determine whether the exploration area is safe or not by predicting the possible reward that can be gained. The combination of two intrinsic rewards serves as a curiosity model to give high returns to unknown states and possible safe actions. In addition, to avoid possible dangers in a limited number of future steps during exploration, a looking-ahead model is introduced to predict future multi-step states, actions, and rewards, respectively. Then, future information is combined with the policy network and included in the loss function of the policy update, allowing the agent to optimize its policy for predicted future states. Experiments on several tasks demonstrated that the proposed PGMP framework significantly improves the agent’s performance.

Different topological patterns in structural covariance networks between high and low delay discounters.

People prefer immediate over future rewards because they discount the latter's value (a phenomenon termed "delay discounting," used as an index of impulsivity). However, little is known about how the preferences are implemented in brain in terms of the coordinated pattern of large-scale structural brain networks. To examine this question, we classified high discounting group (HDG) and low discounting group (LDG) in young adults by assessing their propensity for intertemporal choice. We compared global and regional topological properties in gray matter volume-based structural covariance networks between two groups using graph theoretical analysis. HDG had less clustering coefficient and characteristic path length over the wide sparsity range than LDG, indicating low network segregation and high integration. In addition, the degree of small-worldness was more significant in HDG. Locally, HDG showed less betweenness centrality (BC) in the parahippocampal gyrus and amygdala than LDG. These findings suggest the involvement of structural covariance network topology on impulsive choice, measured by delay discounting, and extend our understanding of how impulsive choice is associated with brain morphological features.

Dual credit assignment processes underlie dopamine signals in a complex spatial environment

Animals frequently make decisions based on expectations of future reward ("values"). Values are updated by ongoing experience: places and choices that result in reward are assigned greater value. Yet, the specific algorithms used by the brain for such credit assignment remain unclear. We monitored accumbens dopamine as rats foraged for rewards in a complex, changing environment. We observed brief dopamine pulses both at reward receipt (scaling with prediction error) and at novel path opportunities. Dopamine also ramped up as rats ran toward reward ports, in proportion to the value at each location. By examining the evolution of these dopamine place-value signals, we found evidence for two distinct update processes: progressive propagation of value along taken paths, as in temporal difference learning, and inference of value throughout the maze, using internal models. Our results demonstrate that within rich, naturalistic environments dopamine conveys place values that are updated via multiple, complementary learning algorithms.

Erotic cue exposure increases neural reward responses without modulating temporal discounting

Abstract Humans prefer smaller sooner over larger later rewards, a tendency denoted as temporal discounting. Discounting of future rewards is increased in multiple maladaptive behaviors and clinical conditions. Although temporal discounting is stable over time, it is partly under contextual control. Appetitive (erotic) cues might increase preferences for immediate rewards, although evidence to date remains mixed. Reward circuit activity was hypothesized to drive increases in temporal discounting following cue exposure, yet this was never tested directly. We examined erotic vs. neutral cue exposure effects on subsequent temporal discounting in a preregistered within-subjects study in healthy male participants (n = 38). Functional magnetic resonance imaging assessed neural cue-reactivity, value-computations, and choice-related effects. We replicated previous findings of value-coding in ventromedial prefrontal cortices, striatum, and cingulate cortex. Likewise, as hypothesized, lateral prefrontal cortex activity increased during delayed reward choices, potentially reflecting cognitive control. Erotic cue exposure was associated with increased activity in attention and reward circuits. Contrary to preregistered hypotheses, temporal discounting was unaffected by cue exposure, and cue responses in reward circuits did not reliably predict changes in behavior. Our results raise doubts on the hypothesis that upregulation of (dopaminergic) reward systems following erotic cue exposure is sufficient to drive myopic approach behavior towards immediate rewards.

Cautious Bayesian MPC: Regret Analysis and Bounds on the Number of Unsafe Learning Episodes

This paper investigates the combination of model predictive control (MPC) concepts and posterior sampling techniques and proposes a simple constraint tightening technique to introduce cautiousness during explorative learning episodes. The provided theoretical analysis in terms of cumulative regret focuses on previously stated sufficient conditions of the resulting ‘Cautious Bayesian MPC’ algorithm and shows Lipschitz continuity of the future reward function in the case of linear MPC problems. In the case of nonlinear MPC problems, it is shown that commonly required assumptions for nonlinear MPC optimization techniques provide sufficient criteria for model-based RL using posterior sampling. Furthermore, it is shown that the proposed constraint tightening implies a bound on the expected number of unsafe learning episodes in the linear and nonlinear case using a soft-constrained MPC formulation. The efficiency of the method is illustrated using numerical examples.