Nonlinear Utility Research Articles

A systematic review of unique methods for measuring discount rates

AbstractDiscount rates play a pivotal role in various fields, capturing the trade-off between present and future, influencing predictions of unhealthy behaviors. Despite substantial variations in discount rates across studies, methods, and individuals, our goal was to systematically overview diverse methods for measuring discount rates. We conducted a comprehensive literature review across multiple databases, targeting English studies introducing and utilizing discounting measurement methods in human subjects. Two authors screened titles and abstracts, with full text review split between them. Extracted data encompassed bibliographic details, theoretical aspects (e.g., discount function), and operational features (e.g., elicitation procedure). After deduplication, 4976, 218, and 83 records underwent title and abstract screening, full text screening, and inclusion, respectively. A total of 86 unique methods were identified. Predominantly, methods were devised for money (75%) and health (22%) discounting. Network analysis on citations indicated limited cross-disciplinary overlap. Only about one fourth, one third, and one third of methods possessed theoretically desirable traits: i) allowance for negative discount rates, ii) application of multiple discount functions, and iii) correction for non-linear utility of outcomes. Diverse discounting measurement methods exist, showcasing varied theoretical and operational characteristics. These differences may stem from the isolation of development in fields like psychology, experimental economics, and health economics. Our systematic review aids readers in choosing methods aligned with their priorities when measuring discount rates.

Analyzing Queueing Problems via Bandits With Linear Reward & Nonlinear Workload Fairness

Queueing models serve as important building blocks in many networking applications such as task scheduling in mobile edge computing nodes, traffic scheduling in networks, congestion control in Internet, etc. However, queueing theory often needs to make strong assumptions about the arrival process or service rewards at each queue. In addition, fairness in serving workload among all queues is of great importance in many applications. In this paper, we address how to optimize resource allocation among multiple queues with a fairness guarantee and without any a priori knowledge of these queues' parameters. To characterize queues with unknown parameters and the fairness requirement, we formulate an online learning model with a varying and continuous action space, as well as a nonlinear utility objective. We design an online learning algorithm to tackle the problem. We prove that our algorithm has a regret upper bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$O(\sqrt{T}\log T)$</tex-math></inline-formula> and our model has a regret lower bound of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\Omega (\sqrt{T})$</tex-math></inline-formula> , where <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$T$</tex-math></inline-formula> stands for the number of decision rounds. The asymptotic closeness of upper and lower bounds guarantees their near tightness and our algorithm's near optimality. We discuss our model's real-world applications in mobile edge computing, wireless networks, and crowdsourcing, and conduct simulations to validate our algorithm's effectiveness.

Empowering revealed preference survey with a supplementary stated preference survey: demonstration of willingness-to-pay estimation within a mode choice case

Mode choice models play a pivotal role in transport demand modelling and help transport planners, engineers and researchers with policy and infrastructure investment evaluation. Recent mode choice studies primarily use revealed preference (RP) data to reflect individuals’ true behaviour. However, this may not be the best practice, given the lack of information in RP data. This study uses a nonlinear utility specification for a multinomial logit mode choice model development using high-quality travel data collected by a GPS-based smartphone application complemented by stated preference (SP) data. The model results highlight the impact of sociodemographic variables on mode choice behaviour and individuals’ willingness-to-pay (WTP) when the model is jointly developed compared to stand-alone SP and RP models. The main message of this study is that in addition to collecting RP, which is a reliable and unbiased source of data, collecting complementary SP data is beneficial as it provides information that is not otherwise available in RP data. This may include a proper variation in the public transport cost variable as demonstrated in this study. Moreover, to better understand the travellers' behaviour regarding the trade-off between time and cost a mixed multinomial logit (MMNL) model in the willingness to pay space is developed on the SP data. capturing the unobserved heterogeneity within the estimated WTPs, the MMNL model outputs reveal a higher variation in WTP of car in-vehicle travel time compared to bus in-vehicle travel time.

Actor-critic multi-objective reinforcement learning for non-linear utility functions

Actor-critic multi-objective reinforcement learning for non-linear utility functions

Monte Carlo tree search algorithms for risk-aware and multi-objective reinforcement learning

In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from a single execution of a policy. In these settings, making decisions based on the average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. Making decisions using just the expected future returns–known in reinforcement learning as the value–cannot account for the potential range of adverse or positive outcomes a decision may have. Therefore, we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time by taking both the future and accrued returns into consideration. In this paper, we propose two novel Monte Carlo tree search algorithms. Firstly, we present a Monte Carlo tree search algorithm that can compute policies for nonlinear utility functions (NLU-MCTS) by optimising the utility of the different possible returns attainable from individual policy executions, resulting in good policies for both risk-aware and multi-objective settings. Secondly, we propose a distributional Monte Carlo tree search algorithm (DMCTS) which extends NLU-MCTS. DMCTS computes an approximate posterior distribution over the utility of the returns, and utilises Thompson sampling during planning to compute policies in risk-aware and multi-objective settings. Both algorithms outperform the state-of-the-art in multi-objective reinforcement learning for the expected utility of the returns.

Magnitude-sensitive reaction times reveal non-linear time costs in multi-alternative decision-making.

Optimality analysis of value-based decisions in binary and multi-alternative choice settings predicts that reaction times should be sensitive only to differences in stimulus magnitudes, but not to overall absolute stimulus magnitude. Yet experimental work in the binary case has shown magnitude sensitive reaction times, and theory shows that this can be explained by switching from linear to multiplicative time costs, but also by nonlinear subjective utility. Thus disentangling explanations for observed magnitude sensitive reaction times is difficult. Here for the first time we extend the theoretical analysis of geometric time-discounting to ternary choices, and present novel experimental evidence for magnitude-sensitivity in such decisions, in both humans and slime moulds. We consider the optimal policies for all possible combinations of linear and geometric time costs, and linear and nonlinear utility; interestingly, geometric discounting emerges as the predominant explanation for magnitude sensitivity.

A Distributional Framework for Risk-Sensitive End-to-End Planning in Continuous MDPs

Recent advances in efficient planning in deterministic or stochastic high-dimensional domains with continuous action spaces leverage backpropagation through a model of the environment to directly optimize action sequences. However, existing methods typically do not take risk into account when optimizing in stochastic domains, which can be incorporated efficiently in MDPs by optimizing a nonlinear utility function of the return distribution. We bridge this gap by introducing Risk-Aware Planning using PyTorch (RAPTOR), a novel unified framework for risk-sensitive planning through end-to-end optimization of commonly-studied risk-sensitive utility functions such as entropic utility, mean-variance optimization and CVaR. A key technical difficulty of our approach is that direct optimization of general risk-sensitive utility functions by backpropagation is impossible due to the presence of environment stochasticity. The novelty of RAPTOR lies in leveraging reparameterization of the state distribution, leading to a unique distributional perspective of end-to-end planning where the return distribution is utilized for sampling as well as optimizing risk-aware objectives by backpropagation in a unified framework. We evaluate and compare RAPTOR on three highly stochastic MDPs, including nonlinear navigation, HVAC control, and linear reservoir control, demonstrating the ability of RAPTOR to manage risk in complex continuous domains according to different notions of risk-sensitive utility.

General Deep Multinomial Logit Model

Multinomial logit model (MNL) is by far the most widely used discrete choice model that is widely used to explain or predict a choice from a set of two or more discrete alternatives. MNL operates within a framework of the random utility model (RUM) in which the utility of an alternative perceived by an individual consists of two components: systematic component and random component. The systematic component is usually defined as a linear function. However, practical decision processes involve complex considerations regarding various aspects of the alternatives and individual which cannot be adequately represented by simple linear models. To overcome the weakness of linear utility model and improve the performance of MNL, in this paper, we propose a general deep multinomial logit model (GDMNL) that takes advantage of both traditional MNL and deep learning. In this model, deep neural networks are applied to extend MNL by learning different nonlinear utility functions of various alternatives. The empirical study in the domain of transit route choice analysis demonstrates the validity and superiority of the proposed model.

A learning mechanism shaping risk preferences and a preliminary test of its relationship with psychopathic traits

People tend to avoid risk in the domain of gains but take risks in the domain of losses; this is called the reflection effect. Formal theories of decision-making have provided important perspectives on risk preferences, but how individuals acquire risk preferences through experiences remains unknown. In the present study, we used reinforcement learning (RL) models to examine the learning processes that can shape attitudes toward risk in both domains. In addition, relationships between learning parameters and personality traits were investigated. Fifty-one participants performed a learning task, and we examined learning parameters and risk preference in each domain. Our results revealed that an RL model that included a nonlinear subjective utility parameter and differential learning rates for positive and negative prediction errors exhibited better fit than other models and that these parameters independently predicted risk preferences and the reflection effect. Regarding personality traits, although the sample sizes may be too small to test personality traits, increased primary psychopathy scores could be linked with decreased learning rates for positive prediction error in loss conditions among participants who had low anxiety traits. The present findings not only contribute to understanding how decision-making in risky conditions is influenced by past experiences but also provide insights into certain psychiatric problems.

Normative Principles for Decision-Making in Natural Environments.

The decisions we make are shaped by a lifetime of learning. Past experience guides the way that we encode information in neural systems for perception and valuation, and determines the information we retrieve when making decisions. Distinct literatures have discussed how lifelong learning and local context shape decisions made about sensory signals, propositional information, or economic prospects. Here, we build bridges between these literatures, arguing for common principles of adaptive rationality in perception, cognition, and economic choice. We discuss how a single common framework, based on normative principles of efficient coding and Bayesian inference, can help us understand a myriad of human decision biases, including sensory illusions, adaptive aftereffects, choice history biases, central tendency effects, anchoring effects, contrast effects, framing effects, congruency effects, reference-dependent valuation, nonlinear utility functions, and discretization heuristics. We describe a simple computational framework for explaining these phenomena.

Why do people pursue goals sequentially when they try to balance cost and utility?

ABSTRACT While research in Cognitive Psychology has investigated people’s ability to use feedback to pursue a single goal, little research has addressed their ability to use feedback to pursue multiple goals. We investigated the reasons for the sequential goal pursuit observed in previous research using a multiple-cue-probability learning task aiming at teaching energy efficiency, specifically, if it derives from cognitive constraints. The task was to balance utility (comfort) against cost, with an explicit budget for both variables (Experiment 1); with utility expressed in an accessible unit (Experiment 2); and with utility represented by a linear function (Experiment 3). The results suggest that the sequential goal pursuit behaviour is driven by limits on cognitive capacity that are little affected by training and goal phrasing. One cognitive constraint was the difficulty with interpreting the effect of actions on the nonlinear utility, which reinforced an initial priority assigned to actions on the linear cost.

Input-dependent noise can explain magnitude-sensitivity in optimal value-based decision-making

AbstractRecent work has derived the optimal policy for two-alternative value-based decisions, in which decision-makers compare the subjective expected reward of two alternatives. Under specific task assumptions — such as linear utility, linear cost of time and constant processing noise — the optimal policy is implemented by a diffusion process in which parallel decision thresholds collapse over time as a function of prior knowledge about average reward across trials. This policy predicts that the decision dynamics of each trial are dominated by the difference in value between alternatives and are insensitive to the magnitude of the alternatives (i.e., their summed values). This prediction clashes with empirical evidence showing magnitude-sensitivity even in the case of equal alternatives, and with ecologically plausible accounts of decision making. Previous work has shown that relaxing assumptions about linear utility or linear time cost can give rise to optimal magnitude-sensitive policies. Here we question the assumption of constant processing noise, in favour of input-dependent noise. The neurally plausible assumption of input-dependent noise during evidence accumulation has received strong support from previous experimental and modelling work. We show that including input-dependent noise in the evidence accumulation process results in a magnitude-sensitive optimal policy for value-based decision-making, even in the case of a linear utility function and a linear cost of time, for both single (i.e., isolated) choices and sequences of choices in which decision-makers maximise reward rate. Compared to explanations that rely on non-linear utility functions and/or non-linear cost of time, our proposed account of magnitude-sensitive optimal decision-making provides a parsimonious explanation that bridges the gap between various task assumptions and between various types of decision making.

An association between prediction errors and risk-seeking: Theory and behavioral evidence.

Reward prediction errors (RPEs) and risk preferences have two things in common: both can shape decision making behavior, and both are commonly associated with dopamine. RPEs drive value learning and are thought to be represented in the phasic release of striatal dopamine. Risk preferences bias choices towards or away from uncertainty; they can be manipulated with drugs that target the dopaminergic system. Based on the common neural substrate, we hypothesize that RPEs and risk preferences are linked on the level of behavior as well. Here, we develop this hypothesis theoretically and test it empirically. First, we apply a recent theory of learning in the basal ganglia to predict how RPEs influence risk preferences. We find that positive RPEs should cause increased risk-seeking, while negative RPEs should cause risk-aversion. We then test our behavioral predictions using a novel bandit task in which value and risk vary independently across options. Critically, conditions are included where options vary in risk but are matched for value. We find that our prediction was correct: participants become more risk-seeking if choices are preceded by positive RPEs, and more risk-averse if choices are preceded by negative RPEs. These findings cannot be explained by other known effects, such as nonlinear utility curves or dynamic learning rates.

Multiple discrete-continuous choice models with bounds on consumptions

This paper derives a multiple discrete–continuous (MDC) choice model formulation with constraints that specify upper bounds on consumption. To do so, considering the conventional utility maximization problem of a consumer, the Karush-Kuhn-Tucker (KKT) conditions are laid out for the MDC model with a general set of linear constraints that include inequalities. Subsequently, we derive a model with constraints that accommodate upper bounds on consumptions and an additive utility structure that accommodates lower bounds on consumptions. The likelihood expression for the proposed model takes a closed form. Furthermore, we extend the formulation to impose bounds on an MDC choice model with activity episode-level choice alternatives that accommodates a logical ordering among different episodes of an activity. The proposed models are derived for two different specifications of the outside good utility – (1) nonlinear utility with respect to consumption and (2) linear utility with respect to consumption. The proposed models are applied to an empirical context to analyze activity-level as well as episode-level activity participation and time allocation while considering bounds on time allocations. Empirical results suggest that the models that consider upper bounds on consumption offer a better fit to data, avoid predictions of unrealistically large time allocations, and result in overall better predictions than those from models without bounds. The proposed models are useful in situations, such as microsimulation models of travel demand, where it is crucial to avoid unrealistically large predictions.

Opponent learning awareness and modelling in multi-objective normal form games

Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with nonlinear utilities. Specifically, we consider two-player multi-objective normal form games with nonlinear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e. learning while considering the impact of one’s policy when anticipating the opponent’s learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.

Double-Auction Mechanisms for Resource Trading Markets

We consider a double-auction mechanism, which was recently proposed in the context of rate allocation in mobile data-offloading markets; our mechanism is also applicable to the problem of bandwidth allocation in network slicing markets. Network operators (users) derive benefit from offloading their traffic to third party WiFi or femtocell networks (link-suppliers). Link-suppliers experience costs for the additional capacity that they provide. Users and link-suppliers (collectively referred to as agents) have their pay-offs and cost functions as private knowledge. A network-manager decomposes the problem into a network problem (with surrogate pay-offs and surrogate cost functions) and agent problems (one per agent). The surrogate pay-offs and cost functions are modulated by the agents' bids. Agents' payoffs and costs are then determined by the allocations and prices set by the network-manager. Under this design, so long as the agents do not anticipate the effect of their actions on the prices set by the network-manager (i.e., price-taking agents), a competitive equilibrium exists as a solution to the network and agent problems, and this equilibrium optimizes the sum utility of all agents. However, this design fails when the agents (including the link-supplier) are all strategic (price-anticipating). Specifically, the presence of a strategic link-supplier drives the system to an undesirable equilibrium with zero participation resulting in an efficiency loss of 100%. This is in stark contrast to an earlier setting where the users alone are strategic but the link-supplier is not - the efficiency loss is known to be at most 34%. The paper then proposes the following Stackelberg game modification with asymmetric information structures for link-supplier and users in order to alleviate the efficiency-loss problem: the network-manager first announces the allocation and payment functions; he then invites the link-supplier to announce its bid, following which the users are invited to respond with their bids. The resulting Stackelberg games' efficiency losses can be characterized in terms of the link-supplier's cost function when the users' pay-off functions are linear. Specifically, when the link-supplier's cost function is quadratic, the worst case efficiency loss is 25%. Further, the loss in efficiency improves for polynomial cost functions of higher degree. For non-linear utility functions (e.g., α-fair and log utilities), we demonstrate the efficacy of the proposed mechanism via. a detailed numerical study.

Optimal Economic Growth Models with Nonlinear Utility Functions

We study a class of finite horizon optimal economic growth problems with nonlinear utility functions and linear production functions. By using a maximum principle in the optimal control theory and employing the special structure of the problems, we are able to explicitly describe the unique solution via input parameters. Economic interpretations of the obtained results and an open problem about the case where the total factor productivity falls into a bounded open interval defined by the growth rate of labor force, the real interest rate, and the exponent of the utility function are also expressed.

Random ambiguity

We introduce a model of random ambiguity aversion. Choice is stochastic due to unobserved shocks to both information and ambiguity aversion. This is modeled as a random set of beliefs in the maxmin expected utility model of Gilboa and Schmeidler (1989). We characterize the model and show that the distribution of ambiguity aversion can be uniquely identified from binary choices. A novel stochastic order on random sets is introduced that characterizes greater uncertainty aversion under stochastic choice. If the set of priors is the Aumann expectation of the random set, then choices satisfy dynamic consistency. This corresponds to an agent who knows the distribution of signals but is uncertain about how to interpret signal realizations. More broadly, the analysis of stochastic properties of random ambiguity attitudes provides a theoretical foundation for the study of other random nonlinear utility models.

On complex dynamic investigations of a piecewise smooth nonlinear duopoly game

Abstract In this paper, we introduce a nonlinear duopoly game which is modelled by a piecewise smooth map. The inverse demand functions of the game are derived from a proposed nonlinear utility function. In the game, we consider that the second firm remanufactures returned products and then sell them again in the market. This means that the second firm receives returned products after the first firm presents its products in the market. A discrete-time piecewise smooth map is used to describe this game based on bounded rationality mechanism. The phase space of the map is divided into two regions separated by a border curve. The map’s fixed points in both regions are obtained. Their conditions of stability are discussed. We carry out numerical experiments to confirm the local stability of these fixed points and identify types of bifurcations by which they become unstable. Through simulation we highlight on periodic cycles and chaotic attractors that jump from region to other or cross the border curve in the phase space.

A Bayesian Joint Model for Risk-Taking and Momentary Mood Reveals the Importance of Subjective, Non-Linear Utility Curves

Mood is one of the most common terms used in psychiatry, yet poorly defined. Here we propose a novel computational approach that links mood and reaction time to underlying cognitive processes using data from a monetary task that was conducted under closed-loop mood manipulation.