Reward Rate Research Articles

Task goals shape the relationship between decision and movement speed.

The speed at which we move is linked to the speed at which we decide to make these movements. Yet, the principles guiding such relationship remain unclear: while some studies point towards a shared invigoration process boosting decision and movement speed jointly, others rather indicate a tradeoff between both levels of control, with slower movements accompanying faster decisions. Here, we aimed (1) at further investigating the existence of a shared invigoration process linking decision and movement and (2) at testing the hypothesis that such a link is masked when detrimental to the reward rate. To this aim, we tested 62 subjects who performed the tokens task in two experiments (separate sessions): Experiment 1 evaluated how changing decision speed affects movement speed while Experiment 2 assessed how changing movement speed affects decision speed. In the latter experiment, subjects were either encouraged to favor decision speed (fast decision group) or decision accuracy (slow decision group). Various mixed model analyses revealed a coregulation of decision (urgency) and movement speed in Experiment 1 and in the fast decision group of Experiment 2, but not in the slow decision group despite the fact that these same subjects displayed a coregulation effect in Experiment 1. Altogether, our findings support the idea that coregulation occurs as a default mode but that this form of control is diminished or supplanted by a tradeoff relationship, contingent on reward rate maximization. Drawing from these behavioral observations, we propose that multiple processes contribute to shaping the speed of decisions and movements.

A reward self-bias leads to more optimal foraging for ourselves than others.

People are self-biased for rewards. We place a higher value on rewards if we receive them than if other people do. However, existing work has ignored one of the most powerful theorems from behavioural ecology of how animals seek resources in everyday life, the Marginal Value Theorem (MVT), which accounts for optimal behaviour for maximising resources intake rate. Does this self-bias help humans maximise rewards when foraging for their own benefit compared to foraging for the benefit of others? Participants had to decide when to leave patches where reward intake was gradually depleting, in environments with different average reward rates. Half of the time participants foraged for themselves, and in the other half they collected rewards for an anonymous stranger. The optimal MVT derived solution states people should leave when the instantaneous reward intake in a patch equals the average rate in an environment. Across two studies, people were more optimal when foraging for self, showing a reduced sensitivity to instantaneous rewards when foraging for other. Autistic traits were linked to reduced sensitivity to reward rates when foraging for self but not for other. These results highlight that the self-bias may be adaptive, helping people maximise reward intake.

A dynamical systems approach to optimal foraging

Foraging for resources in an environment is a fundamental activity that must be addressed by any biological agent. Modelling this phenomenon in simulations can enhance our understanding of the characteristics of natural intelligence. In this work, we present a novel approach to model foraging in-silico using a continuous coupled dynamical system. The dynamical system is composed of three differential equations, representing the position of the agent, the agent’s control policy, and the environmental resource dynamics. Crucially, the control policy is implemented as a parameterized differential equation which allows the control policy to adapt in order to solve the foraging task. Using this setup, we show that when these dynamics are coupled and the controller parameters are optimized to maximize the rate of reward collected, adaptive foraging emerges in the agent. We further show that the internal dynamics of the controller, as a surrogate brain model, closely resemble the dynamics of the evidence accumulation mechanism, which may be used by certain neurons of the dorsal anterior cingulate cortex region in non-human primates, for deciding when to migrate from one patch to another. We show that by modulating the resource growth rates of the environment, the emergent behaviour of the artificial agent agrees with the predictions of the optimal foraging theory. Finally, we demonstrate how the framework can be extended to stochastic and multi-agent settings.

On Waterfall's cryptoeconomic forecast

This work presents a system dynamics modeling approach to evaluate critical economic metrics on the Waterfall platform, a decentralized public network. The constructed model serves as a significant tool for understanding the intricate economic processes within the platform and supports informed decision-making by stakeholders. By using input parameters such as the number of network validators and transactions per second (TPS) over time, the model projects key metrics like the inflation rate and total supply of coins, providing insights into the platform's economic stability. The Waterfall platform is designed as a scalable smart contract ecosystem, underpinned by cryptoeconomics, to ensure a self- sustaining and self-regulating environment that maximizes stakeholder benefits. The platform's mainnet, launched in June 2024, has shown consistency between testnet results and theoretical calculations, demonstrating the robustness of the underlying economic model. However, as with any economic system, forecasting is essential for decision-making and improving efficiency. The system dynamics model allows for the exploration of different scenarios, considering various growth assumptions for validators and TPS, to project future economic conditions of the platform. Three scenarios – pessimistic, realistic, and optimistic – are considered, each with distinct growth trajectories for the number of validators and TPS. The outcomes highlight that while the initial stages of the network are characterized by high reward rates due to a smaller number of validators, these rates decline as more validators join, affecting the inflation rate over time. The relationship between the minting of new coins (as validator rewards) and the burning of transaction fees determines the system's inflationary dynamics. Under certain conditions, an increase in TPS could lead to deflation, enhancing the platform's economic sustainability. The results suggest that Waterfall's cryptoeconomic model has a strong potential for long-term stability and growth. The model's flexibility allows for adjustments based on real-time data and expert judgments, making it a valuable tool for both internal stakeholders and external analysts. This approach could also be applied to studying the cryptoeconomics of other decentralized networks, providing a broader understanding of their economic stability and long-term viability.

Rational inattention in neural coding for economic choice.

Mental operations like computing the value of an option are computationally expensive. Even before we evaluate options, we must decide how much attentional effort to invest in the evaluation process. More precise evaluation will improve choice accuracy, and thus reward yield, but the gain may not justify the cost. Rational Inattention theories provide an accounting of the internal economics of attentionally effortful economic decisions. To understand this process, we examined choices and neural activity in several brain regions in six macaques making risky choices. We extended the rational inattention framework to incorporate the foraging theoretic understanding of local environmental richness or reward rate, which we predict will promote attentional effort. Consistent with this idea, we found local reward rate positively predicted choice accuracy. Supporting the hypothesis that this effect reflects variations in attentional effort, richer contexts were associated with increased baseline and evoked pupil size. Neural populations likewise showed systematic baseline coding of reward rate context. During increased reward rate contexts, ventral striatum and orbitofrontal cortex showed both an increase in value decodability and a rotation in the population geometries for value. This confluence of these results suggests a mechanism of attentional effort that operates by controlling gain through using partially distinct population codes for value. Additionally, increased reward rate accelerated value code dynamics, which have been linked to improved signal-to-noise. These results extend the theory of rational inattention to static and stationary contexts and align theories of rational inattention with specific costly, neural processes.

B - 62 Cognitive Training Targeting Reward Responsiveness Promotes more Efficient Affective Processing in an Interference Go/NoGo Task

Abstract Objective Abnormalities in emotional processing are commonly seen in depression and anxiety, disorders also associated with decreased reward sensitivity i.e., anhedonia. We tested a noninvasive, cognitive training program aimed at boosting reward responsiveness and gaged if it may also improve affective processing efficiency. Methods Individuals (N = 50; 54% Female; 18–65 yo) with low hedonic tone (Positive and Negative Affect Scale/PANAS score < 25) and diagnostic levels of depression and/or anxiety (Patient Health Questionnaire-8/PHQ-8 or General Anxiety Disorder-7/GAD-7 score ≥ 10) who responded to community-posted flyers completed an affective go/no-go (AGN) task before and after completing an exploratory reward learning task, i.e., a multi-arm bandit (MAB). Participants were randomly assigned to a high-reward MAB training with either high variance (HV) or low variance (LV) in reward rates, HV being expected to boost reward maximization. In the AGN, participants saw negative and positive pictures and had to respond as fast as possible to pictures of a given target category (positive or negative). Results Following the HV vs LV training MAB, participants were more accurate (higher likelihood of hits), which was more evident in negative target blocks relative to positive target blocks (odds ratio/OR:0.61, 95% CI = [0.38,0.98]; z = −2.045, p = 0.041). Participants exhibited stronger decreases in hits response times following the HV vs LV training MAB, but across all target blocks (B:-37 ms, 95% CI = [−56,-18]; t = −3.806, p < 0.001). Conclusions A novel cognitive training program aimed at boosting reward sensitivity may help improve patients’ processing of emotionally taxing stimuli. Targeting reward sensitivity is a critical component of improving clinical outcomes in depressed and anxious patients..

Edge computing in Internet of Vehicles: A federated learning method based on Stackelberg dynamic game

With the development of Intelligent Transportation Systems (ITS), data on the Internet of Vehicles (IoV) is increasing day by day. To alleviate computing pressure in IoV, Vehicle Edge Computing (VEC) is being widely used as a new computing paradigm. Moreover, to address the serious problem of privacy leakage in VEC, Federated Learning (FL) is increasingly considered for analyzing big data in VEC. However, in actual VEC, problems such as data heterogeneity and poor training often occur. To address this set of problems, we introduce a two-stage Stackelberg game structure for FL training, choosing the Cloud Server (CS) as the leader and the Roadside Unit (RSU) as the follower. Then, we define the utility functions of CS and RSU and obtain the optimal reward rate and local accuracy for each iteration. To address the problem of inefficiency during learning process, we separate high-dimensional data features into global features and personalized features based on feature separation, and use them to capture historical information. Next, vehicle federated learning with historical information based on dynamic Stackelberg game (VFLHI-DSG) was proposed. Finally, we conducted a comprehensive comparative experiment, results show that VFLHI-DSG has excellent performance in different scenarios.

Economic foraging in a floral marketplace: asymmetrically dominated decoy effects in bumblebees.

While most models of decision-making assume that individuals assign options absolute values, animals often assess options comparatively, violating principles of economic rationality. Such 'irrational' preferences are especially common when two rewards vary along multiple dimensions of quality and a third, 'decoy' option is available. Bumblebees are models of decision-making, yet whether they are subject to decoy effects is unknown. We addressed this question using bumblebees (Bombus impatiens) choosing between flowers that varied in their nectar concentration and reward rate. We first gave bees a choice between two flower types, one higher in concentration and the other higher in reward rate. Bees were then given a choice between these flowers and either a 'concentration' or 'rate' decoy, designed to be asymmetrically dominated on each axis. The rate decoy increased bees' preference in the expected direction, while the concentration decoy did not. In a second experiment, we manipulated choices along two single reward dimensions to test whether this discrepancy was explained by differences in how concentration versus reward rate were evaluated. We found that low-concentration decoys increased bees' preference for the medium option as predicted, whereas low-rate decoys had no effect. Our results suggest that both low- and high-value flowers can influence pollinator preferences in ways previously unconsidered.

Random interval schedule of reinforcement influences punishment resistance for cocaine in rats

In an animal model of compulsive drug use, a subset of rats continues to self-administer cocaine despite footshock consequences and is considered punishment resistant. We recently found that punishment resistance is associated with habits that persist under conditions that typically encourage a transition to goal-directed control. Given that random ratio (RR) and random interval (RI) schedules of reinforcement influence whether responding is goal-directed or habitual, we investigated the influence of these schedules on punishment resistance for cocaine or food. Male and female Sprague Dawley rats were trained to self-administer either intravenous cocaine or food pellets on a seeking-taking chained schedule of reinforcement, with the seeking lever requiring completion of either an RR20 or RI60 schedule. Rats were then given four days of punishment testing with footshock administered at the completion of seeking on a random one-third of trials. For cocaine-trained rats, the RI60 schedule led to greater punishment resistance (i.e., more trials completed) than the RR20 schedule in males and females. For food-trained rats, the RI60 schedule led to greater punishment resistance (i.e., higher reward rates) than the RR20 schedule in female rats, although male rats showed punishment resistance on both RR20 and RI60 schedules. For both cocaine and food, we found that seeking responses were suppressed to a greater degree than reward rate with the RI60 schedule, whereas response rate and reward rate were equally suppressed with the RR20 schedule. This dissociation between punishment effects on reward rate and response rate with the RI60 schedule can be explained by the nonlinear relation between these variables on RI schedules, but it does not account for the enhanced resistance to punishment. Overall, the results show greater punishment resistance with the RI60 schedule as compared to the RR20 schedule, indicating that schedules of reinforcement are an influencing factor on resistance to negative consequences.

Flexible recommendation for optimizing the debt collection process based on customer risk using deep reinforcement learning

Finance sector loss can be minimized by reducing the number of defaulters who often miss payments during debt collection. Most research focused on the credit risk analysis before approving the loan to the customer, and few papers related to predicting the defaulters. Certain works are focused on minimizing the number of calls made by debt collectors during the collection process. We have developed a Flexible Recommender System using Deep Reinforcement Learning (FRS-DRL) to provide Flexible recommendations based on the risk category of the customers. Initially, we categorized customers into defaulters and non-defaulters. The defaulters are again classified into Very Low, Low, Medium, High, and Very High-Risk customers. Based on this category, we recommend suitable actions to the Field Collection Agent (FCA) to increase the collection rate. The suggestions include how frequently the customer should be contacted, statements, and the dates and times of visits. At the same time, the statements define instructions to FCA for collecting money. Managers are traditionally in charge of this type of allocation; they must understand each customer personally. Our proposed recommendation method eliminates the need for manual mapping. We build a Hybrid Actor-Critic algorithm with an Adaptive Trust region Policy Optimization algorithm for classifying the customer category and providing the appropriate recommendations to reduce the defaulter rate, thereby increasing the collection efficiency and reducing the human effort. This FRS-DRL is tested in a real-time environment for three months and provides an improved reward rate with and without transfer learning at 73.22 % and 62.44 %. The collection percentage improved by 20.34 % in five different regions, with a 16.30 % reduction in the number of visits made by the FCA.

Regulation of reinforcement learning parameters captures long-term changes in rat behaviour.

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

User-centric Markov reward model for state-dependent Erlang loss systems

Markov reward models are commonly used in the analysis of systems by integrating a reward rate to each system state. Typically, rewards are defined based on system states and reflect the system’s perspective. From a user’s point of view, it is important to consider the changing system conditions and dynamics while the user consumes a service. The key contributions of this paper are proper definitions for (i) system-centric reward and (ii) user-centric reward of the Erlang loss model M/M/n-0 and M/M(x)/n with state-dependent service rates, as well as (iii) the analysis of the relationships between those metrics. Our key result allows a simple computation of the user-centric rewards. The differences between the system-centric and the user-centric rewards are demonstrated for a real-world cloud gaming use case. To the best of our knowledge, this is the first analysis showing the relationship between user-centric rewards and system-centric rewards. This work gives relevant and important insights in how to integrate the user’s perspective in the analysis of Markov reward models and is a blueprint for the analysis of other services beyond cloud gaming while also considering user engagement.

Dual-tasking modulates movement speed but not value-based choices during walking.

Value-based decision-making often occurs in multitasking scenarios relying on both cognitive and motor processes. Yet, laboratory experiments often isolate these processes, thereby neglecting potential interactions. This isolated approach reveals a dichotomy: the cognitive process by which reward influences decision-making is capacity-limited, whereas the influence of motor cost is free of such constraints. If true, dual-tasking should predominantly impair reward processing but not affect the impact of motor costs. To test this hypothesis, we designed a decision-making task in which participants made choices to walk toward targets for rewards while navigating past an obstacle. The motor cost to reach these rewards varied in real-time. Participants either solely performed the decision-making task, or additionally performed a secondary pitch-recall task. Results revealed that while both reward and motor costs influenced decision-making, the secondary task did not affect these factors. Instead, dual-tasking slowed down participants' walking, thereby reducing the overall reward rate. Hence, contrary to the prediction that the added cognitive demand would affect the weighing of reward or motor cost differentially, these processes seem to be maintained at the expense of slowing down the motor system. This slowdown may be indicative of interference at the locomotor level, thereby underpinning motor-cognitive interactions during decision-making.

Attention deficits linked with proclivity to explore while foraging.

All mobile organisms forage for resources, choosing how and when to search for new opportunities by comparing current returns with the average for the environment. In humans, nomadic lifestyles favouring exploration have been associated with genetic mutations implicated in attention deficit hyperactivity disorder (ADHD), inviting the hypothesis that this condition may impact foraging decisions in the general population. Here we tested this pre-registered hypothesis by examining how human participants collected resources in an online foraging task. On every trial, participants chose either to continue to collect rewards from a depleting patch of resources or to replenish the patch. Participants also completed a well-validated ADHD self-report screening assessment at the end of sessions. Participants departed resource patches sooner when travel times between patches were shorter than when they were longer, as predicted by optimal foraging theory. Participants whose scores on the ADHD scale crossed the threshold for a positive screen departed patches significantly sooner than participants who did not meet this criterion. Participants meeting this threshold for ADHD also achieved higher reward rates than individuals who did not. Our findings suggest that ADHD attributes may confer foraging advantages in some environments and invite the possibility that this condition may reflect an adaptation favouring exploration over exploitation.

Striatal and Behavioral Responses to Reward Vary by Socioeconomic Status in Adolescents.

Disparities in socioeconomic status (SES) lead to unequal access to financial and social support. These disparities are believed to influence reward sensitivity, which in turn are hypothesized to shape how individuals respond to and pursue rewarding experiences. However, surprisingly little is known about how SES shapes reward sensitivity in adolescence. Here, we investigated how SES influenced adolescent responses to reward, both in behavior and the striatum-a brain region that is highly sensitive to reward. We examined responses to both immediate reward (tracked by phasic dopamine) and average reward rate fluctuations (tracked by tonic dopamine) as these distinct signals independently shape learning and motivation. Adolescents (n = 114; 12-14 years; 58 female) performed a gambling task during functional magnetic resonance imaging. We manipulated trial-by-trial reward and loss outcomes, leading to fluctuations between periods of reward scarcity and abundance. We found that a higher reward rate hastened behavioral responses, and increased guess switching, consistent with the idea that reward abundance increases response vigor and exploration. Moreover, immediate reward reinforced previously rewarding decisions (win-stay, lose-switch) and slowed responses (postreward pausing), particularly when rewards were scarce. Notably, lower-SES adolescents slowed down less after rare rewards than higher-SES adolescents. In the brain, striatal activations covaried with the average reward rate across time and showed greater activations during rewarding blocks. However, these striatal effects were diminished in lower-SES adolescents. These findings show that the striatum tracks reward rate fluctuations, which shape decisions and motivation. Moreover, lower SES appears to attenuate reward-driven behavioral and brain responses.

Dopamine-independent effect of rewards on choices through hidden-state inference

Dopamine is implicated in adaptive behavior through reward prediction error (RPE) signals that update value estimates. There is also accumulating evidence that animals in structured environments can use inference processes to facilitate behavioral flexibility. However, it is unclear how these two accounts of reward-guided decision-making should be integrated. Using a two-step task for mice, we show that dopamine reports RPEs using value information inferred from task structure knowledge, alongside information about reward rate and movement. Nonetheless, although rewards strongly influenced choices and dopamine activity, neither activating nor inhibiting dopamine neurons at trial outcome affected future choice. These data were recapitulated by a neural network model where cortex learned to track hidden task states by predicting observations, while basal ganglia learned values and actions via RPEs. This shows that the influence of rewards on choices can stem from dopamine-independent information they convey about the world’s state, not the dopaminergic RPEs they produce.

Inventory model for livestock farm under quantity discount, power demand, prepayment and carbon rules

Government-enforced regulations and incentives are primarily intended to promote sustainability. Consumers’ growing concern about the environmental impact of the products they consume, including meat and dairy products, further drives this shift. As a result, livestock farming companies are consequently prioritizing environmental sustainability efforts. Strict environmental restrictions in Europe, Canada, and the USA make it challenging for industrial livestock farms to abide by them without jeopardizing their profitability. This work illustrates the business of a livestock farm operated under environmental regulations, including cap-and-price, cap-and-trade, cap-and-offset, and carbon tax policies. Under a quantity-based all-units and prepayment scheme, the company buys a single kind of newborn-growing items from a supplier and sells them after slaughter when they have gained the desired weight via feeding. The whole targeted live weight is not consumable after slaughtering due to instant weight loss during bleeding and the removal of various non-edible components from the growing items. Incorporating this attribute, the current work derives sustainable inventory and pricing decisions for a livestock farm for the first time. Furthermore, the consumption rate of the edible components of growing items is adopted as a power form of both selling price per weight unit and storage duration. To determine whether a single or a series of prepayment installment strategies is more cost-effective for the company, important criteria are defined. Finally, several salient recommendations for the farm are extracted to not only gain economically but also preserve the environment by lowering its carbon footprint. When the carbon tariff rate is higher than the reward rate, the cap-and-trade policy outperforms the cap-and-price, cap-and-offset, and carbon tax mechanisms in terms of profit. However, both cap-and-trade and carbon tax regulations perform equally better than the cap-and-price and cap-and-offset regulations in terms of released emissions.

Effort Foraging Task reveals positive correlation between individual differences in the cost of cognitive and physical effort in humans

Effort-based decisions, in which people weigh potential future rewards against effort costs required to achieve those rewards involve both cognitive and physical effort, though the mechanistic relationship between them is not yet understood. Here, we use an individual differences approach to isolate and measure the computational processes underlying effort-based decisions and test the association between cognitive and physical domains. Patch foraging is an ecologically valid reward rate maximization problem with well-developed theoretical tools. We developed the Effort Foraging Task, which embedded cognitive or physical effort into patch foraging, to quantify the cost of both cognitive and physical effort indirectly, by their effects on foraging choices. Participants chose between harvesting a depleting patch, or traveling to a new patch that was costly in time and effort. Participants’ exit thresholds (reflecting the reward they expected to receive by harvesting when they chose to travel to a new patch) were sensitive to cognitive and physical effort demands, allowing us to quantify the perceived effort cost in monetary terms. The indirect sequential choice style revealed effort-seeking behavior in a minority of participants (preferring high over low effort) that has apparently been missed by many previous approaches. Individual differences in cognitive and physical effort costs were positively correlated, suggesting that these are perceived and processed in common. We used canonical correlation analysis to probe the relationship of task measures to self-reported affect and motivation, and found correlations of cognitive effort with anxiety, cognitive function, behavioral activation, and self-efficacy, but no similar correlations with physical effort.

Effects of reward and effort history on decision making and movement vigor during foraging.

During foraging, animals explore a site and harvest reward and then abandon that site and travel to the next opportunity. One aspect of this behavior involves decision making, and the other involves movement control. These two aspects of behavior may be linked via an underlying desire to maximize a single normative utility: the sum of all rewards acquired, minus all efforts expended, divided by time. According to this theory, the history of rewards, and not just its immediate availability, should dictate how long one should stay and harvest reward and how vigorously one should travel to the next opportunity. We tested this theory in a series of experiments in which humans used their hand to harvest tokens at a reward patch and then used their arm to reach toward another patch. After a history of high rewards, the subjects not only shortened their harvest duration but also moved more vigorously toward the next reward opportunity. In contrast, after a history of high effort they lengthened their harvest duration but reduced their movement vigor, reaching more slowly to the next reward site. Thus, a history of high reward or low effort biased decisions by promoting early abandonment of the reward site and biased movements by promoting vigor.NEW & NOTEWORTHY Much of life is spent foraging. Whereas previous work has focused on the decision regarding time spent harvesting from a reward patch, here we test the idea that both decision making and movement control are tuned to optimize the net rate of reward in an environment. Our results show that movement patterns reflect not just immediate expectations but also past experiences in the environment, providing fundamental insight into the factors governing volitional control of arm movements.

Energy analysis using semi‐Markov modeling for the base station in 5G networks

SummaryWith the increasing prevalence of wireless sensor networks, the uninterrupted operation of these networks becomes essential. To ensure continuous functionality, wireless networks rely on available base stations (BSs). However, the persistent operation of BSs comes at the cost of substantial energy consumption. Consequently, given the surge in wireless network traffic and the necessity for uninterrupted BS availability, energy efficiency within the BS becomes a concerning issue. To address this, the study employs a semi‐Markov model to depict the availability of the BS, with states corresponding to the failures of its components (baseband unit, remote unit, and software module). The analysis yields a steady‐state solution, with reward rates assigned to each state based on the energy consumption of individual BS components. This approach enables the determination of the expected energy consumption within this model. Additionally, the BS's throughput is assessed using an M/G/1 queueing model with server breakdown.