Nonlinear Utility Functions Research Articles

Actor-critic multi-objective reinforcement learning for non-linear utility functions

Actor-critic multi-objective reinforcement learning for non-linear utility functions

Monte Carlo tree search algorithms for risk-aware and multi-objective reinforcement learning

In many risk-aware and multi-objective reinforcement learning settings, the utility of the user is derived from a single execution of a policy. In these settings, making decisions based on the average future returns is not suitable. For example, in a medical setting a patient may only have one opportunity to treat their illness. Making decisions using just the expected future returns–known in reinforcement learning as the value–cannot account for the potential range of adverse or positive outcomes a decision may have. Therefore, we should use the distribution over expected future returns differently to represent the critical information that the agent requires at decision time by taking both the future and accrued returns into consideration. In this paper, we propose two novel Monte Carlo tree search algorithms. Firstly, we present a Monte Carlo tree search algorithm that can compute policies for nonlinear utility functions (NLU-MCTS) by optimising the utility of the different possible returns attainable from individual policy executions, resulting in good policies for both risk-aware and multi-objective settings. Secondly, we propose a distributional Monte Carlo tree search algorithm (DMCTS) which extends NLU-MCTS. DMCTS computes an approximate posterior distribution over the utility of the returns, and utilises Thompson sampling during planning to compute policies in risk-aware and multi-objective settings. Both algorithms outperform the state-of-the-art in multi-objective reinforcement learning for the expected utility of the returns.

A Distributional Framework for Risk-Sensitive End-to-End Planning in Continuous MDPs

Recent advances in efficient planning in deterministic or stochastic high-dimensional domains with continuous action spaces leverage backpropagation through a model of the environment to directly optimize action sequences. However, existing methods typically do not take risk into account when optimizing in stochastic domains, which can be incorporated efficiently in MDPs by optimizing a nonlinear utility function of the return distribution. We bridge this gap by introducing Risk-Aware Planning using PyTorch (RAPTOR), a novel unified framework for risk-sensitive planning through end-to-end optimization of commonly-studied risk-sensitive utility functions such as entropic utility, mean-variance optimization and CVaR. A key technical difficulty of our approach is that direct optimization of general risk-sensitive utility functions by backpropagation is impossible due to the presence of environment stochasticity. The novelty of RAPTOR lies in leveraging reparameterization of the state distribution, leading to a unique distributional perspective of end-to-end planning where the return distribution is utilized for sampling as well as optimizing risk-aware objectives by backpropagation in a unified framework. We evaluate and compare RAPTOR on three highly stochastic MDPs, including nonlinear navigation, HVAC control, and linear reservoir control, demonstrating the ability of RAPTOR to manage risk in complex continuous domains according to different notions of risk-sensitive utility.

General Deep Multinomial Logit Model

Multinomial logit model (MNL) is by far the most widely used discrete choice model that is widely used to explain or predict a choice from a set of two or more discrete alternatives. MNL operates within a framework of the random utility model (RUM) in which the utility of an alternative perceived by an individual consists of two components: systematic component and random component. The systematic component is usually defined as a linear function. However, practical decision processes involve complex considerations regarding various aspects of the alternatives and individual which cannot be adequately represented by simple linear models. To overcome the weakness of linear utility model and improve the performance of MNL, in this paper, we propose a general deep multinomial logit model (GDMNL) that takes advantage of both traditional MNL and deep learning. In this model, deep neural networks are applied to extend MNL by learning different nonlinear utility functions of various alternatives. The empirical study in the domain of transit route choice analysis demonstrates the validity and superiority of the proposed model.

Normative Principles for Decision-Making in Natural Environments.

The decisions we make are shaped by a lifetime of learning. Past experience guides the way that we encode information in neural systems for perception and valuation, and determines the information we retrieve when making decisions. Distinct literatures have discussed how lifelong learning and local context shape decisions made about sensory signals, propositional information, or economic prospects. Here, we build bridges between these literatures, arguing for common principles of adaptive rationality in perception, cognition, and economic choice. We discuss how a single common framework, based on normative principles of efficient coding and Bayesian inference, can help us understand a myriad of human decision biases, including sensory illusions, adaptive aftereffects, choice history biases, central tendency effects, anchoring effects, contrast effects, framing effects, congruency effects, reference-dependent valuation, nonlinear utility functions, and discretization heuristics. We describe a simple computational framework for explaining these phenomena.

Input-dependent noise can explain magnitude-sensitivity in optimal value-based decision-making

AbstractRecent work has derived the optimal policy for two-alternative value-based decisions, in which decision-makers compare the subjective expected reward of two alternatives. Under specific task assumptions — such as linear utility, linear cost of time and constant processing noise — the optimal policy is implemented by a diffusion process in which parallel decision thresholds collapse over time as a function of prior knowledge about average reward across trials. This policy predicts that the decision dynamics of each trial are dominated by the difference in value between alternatives and are insensitive to the magnitude of the alternatives (i.e., their summed values). This prediction clashes with empirical evidence showing magnitude-sensitivity even in the case of equal alternatives, and with ecologically plausible accounts of decision making. Previous work has shown that relaxing assumptions about linear utility or linear time cost can give rise to optimal magnitude-sensitive policies. Here we question the assumption of constant processing noise, in favour of input-dependent noise. The neurally plausible assumption of input-dependent noise during evidence accumulation has received strong support from previous experimental and modelling work. We show that including input-dependent noise in the evidence accumulation process results in a magnitude-sensitive optimal policy for value-based decision-making, even in the case of a linear utility function and a linear cost of time, for both single (i.e., isolated) choices and sequences of choices in which decision-makers maximise reward rate. Compared to explanations that rely on non-linear utility functions and/or non-linear cost of time, our proposed account of magnitude-sensitive optimal decision-making provides a parsimonious explanation that bridges the gap between various task assumptions and between various types of decision making.

Opponent learning awareness and modelling in multi-objective normal form games

Many real-world multi-agent interactions consider multiple distinct criteria, i.e. the payoffs are multi-objective in nature. However, the same multi-objective payoff vector may lead to different utilities for each participant. Therefore, it is essential for an agent to learn about the behaviour of other agents in the system. In this work, we present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with nonlinear utilities. Specifically, we consider two-player multi-objective normal form games with nonlinear utility functions under the scalarised expected returns optimisation criterion. We contribute novel actor-critic and policy gradient formulations to allow reinforcement learning of mixed strategies in this setting, along with extensions that incorporate opponent policy reconstruction and learning with opponent learning awareness (i.e. learning while considering the impact of one’s policy when anticipating the opponent’s learning step). Empirical results in five different MONFGs demonstrate that opponent learning awareness and modelling can drastically alter the learning dynamics in this setting. When equilibria are present, opponent modelling can confer significant benefits on agents that implement it. When there are no Nash equilibria, opponent learning awareness and modelling allows agents to still converge to meaningful solutions that approximate equilibria.

Double-Auction Mechanisms for Resource Trading Markets

We consider a double-auction mechanism, which was recently proposed in the context of rate allocation in mobile data-offloading markets; our mechanism is also applicable to the problem of bandwidth allocation in network slicing markets. Network operators (users) derive benefit from offloading their traffic to third party WiFi or femtocell networks (link-suppliers). Link-suppliers experience costs for the additional capacity that they provide. Users and link-suppliers (collectively referred to as agents) have their pay-offs and cost functions as private knowledge. A network-manager decomposes the problem into a network problem (with surrogate pay-offs and surrogate cost functions) and agent problems (one per agent). The surrogate pay-offs and cost functions are modulated by the agents' bids. Agents' payoffs and costs are then determined by the allocations and prices set by the network-manager. Under this design, so long as the agents do not anticipate the effect of their actions on the prices set by the network-manager (i.e., price-taking agents), a competitive equilibrium exists as a solution to the network and agent problems, and this equilibrium optimizes the sum utility of all agents. However, this design fails when the agents (including the link-supplier) are all strategic (price-anticipating). Specifically, the presence of a strategic link-supplier drives the system to an undesirable equilibrium with zero participation resulting in an efficiency loss of 100%. This is in stark contrast to an earlier setting where the users alone are strategic but the link-supplier is not - the efficiency loss is known to be at most 34%. The paper then proposes the following Stackelberg game modification with asymmetric information structures for link-supplier and users in order to alleviate the efficiency-loss problem: the network-manager first announces the allocation and payment functions; he then invites the link-supplier to announce its bid, following which the users are invited to respond with their bids. The resulting Stackelberg games' efficiency losses can be characterized in terms of the link-supplier's cost function when the users' pay-off functions are linear. Specifically, when the link-supplier's cost function is quadratic, the worst case efficiency loss is 25%. Further, the loss in efficiency improves for polynomial cost functions of higher degree. For non-linear utility functions (e.g., α-fair and log utilities), we demonstrate the efficacy of the proposed mechanism via. a detailed numerical study.

Optimal Economic Growth Models with Nonlinear Utility Functions

We study a class of finite horizon optimal economic growth problems with nonlinear utility functions and linear production functions. By using a maximum principle in the optimal control theory and employing the special structure of the problems, we are able to explicitly describe the unique solution via input parameters. Economic interpretations of the obtained results and an open problem about the case where the total factor productivity falls into a bounded open interval defined by the growth rate of labor force, the real interest rate, and the exponent of the utility function are also expressed.

On complex dynamic investigations of a piecewise smooth nonlinear duopoly game

Abstract In this paper, we introduce a nonlinear duopoly game which is modelled by a piecewise smooth map. The inverse demand functions of the game are derived from a proposed nonlinear utility function. In the game, we consider that the second firm remanufactures returned products and then sell them again in the market. This means that the second firm receives returned products after the first firm presents its products in the market. A discrete-time piecewise smooth map is used to describe this game based on bounded rationality mechanism. The phase space of the map is divided into two regions separated by a border curve. The map’s fixed points in both regions are obtained. Their conditions of stability are discussed. We carry out numerical experiments to confirm the local stability of these fixed points and identify types of bifurcations by which they become unstable. Through simulation we highlight on periodic cycles and chaotic attractors that jump from region to other or cross the border curve in the phase space.

A Deep Choice Model for Hiring Outcome Prediction in Online Labor Markets

A key challenge faced by online labor market researchers and practitioners is to understand how employers make hiring decisions from many job bidders with distinct attributes. This study investigates employer hiring behavior in one of the largest online labor markets by building a datadriven hiring decision prediction model. With the limitation of traditional discrete choice model (conditional logit model), we develop a novel deep choice model to simulate the hiring behavior from 722,339 job posts. The deep choice model extends the classical conditional logit model by learning a non-linear utility function identically for each bidder within of the job posts via a pointwise convolutional neural network. This non-linear mapping can be straightforwardly optimized using stochastic gradient approach. We test the model on 12 categories of job posts in the dataset. Results show that our deep choice model outperforms the linear-utility conditional logit model in predicting hiring preferences. By analyzing the model using dimensionality reduction and sensitivity analysis, we highlight the nonlinear combination of bidders’ features in impacting employers’ hiring decisions.

A Remanufacturing Duopoly Game Based on a Piecewise Nonlinear Map: Analysis and Investigations

Abstract A remanufacturing Cournot duopoly game is introduced based on a nonlinear utility function in this paper. What we mean by remanufacturing here is that the second firm in this game receives used products and remanufacture them and then sell them again in the market. The bounded rationality mechanism is used to form a piecewise system that describes this game in discrete time periods. This piecewise system depends on five parameters and is defined on two regions separated by a border curve. The fixed points of this system in each region are calculated and their stability is discussed. Numerical simulations for this system exhibit the occurrence of different types of multiple attractors. We also give examples of different stable periodic cycles and chaotic attractors that are separated by the border curve or passing through it.

Mixed Logit Model Based on Improved Nonlinear Utility Functions: A Market Shares Solution Method of Different Railway Traffic Modes

In recent years, with the development of high-speed railway in China, the railway operating mileages and passenger transport capacity have increased rapidly. Due to the high density of trains and the limited capacity of railways, it is necessary to solve market shares of different railway traffic modes in order to adjust the operation plans appropriately and run railway passenger transport products in line with passenger demand. Therefore, the purpose of this paper is to calculate market shares by formulating a mixed logit model based on improved nonlinear utility functions taking different factors into consideration, such as seat grades, fares, running time, passenger income levels and so on. Firstly according to maximum likelihood estimation, the likelihood function of this mixed logit model is proposed to maximize utility of all passenger groups. After that, we propose two improved algorithms based on the simulated annealing algorithm (ISAA-CC and ISAA-SS) to estimate the unknown parameters and solve the optimal solution of this model in order to enhance the computational efficiency. Finally, a real-world instance with related data of Beijing–Tianjin corridor, is implemented to demonstrate the performance and effectiveness of the proposed approaches. In addition, by performing this numerical experiment and comparing these two improved algorithms with the traditional Newton method, the ant colony algorithm and the simulated annealing algorithm, we prove that the improved algorithms we developed are superior to others in the optimal solution.

A utility-based analysis of equilibria in multi-objective normal-form games

Abstract In multi-objective multi-agent systems (MOMASs), agents explicitly consider the possible trade-offs between conflicting objective functions. We argue that compromises between competing objectives in MOMAS should be analyzed on the basis of the utility that these compromises have for the users of a system, where an agent’s utility function maps their payoff vectors to scalar utility values. This utility-based approach naturally leads to two different optimization criteria for agents in a MOMAS: expected scalarized returns (ESRs) and scalarized expected returns (SERs). In this article, we explore the differences between these two criteria using the framework of multi-objective normal-form games (MONFGs). We demonstrate that the choice of optimization criterion (ESR or SER) can radically alter the set of equilibria in a MONFG when nonlinear utility functions are used.

Determining mode shift elasticity based on household income and travel cost

The present study focuses on changes in the behavior of travelers belonging to various monthly income groups from a transit deficient metropolitan city, when the travel cost of present mode changes. Socio-economic, demographic, and travel characteristics were collected using a well-structured questionnaire. To capture the mode shift behavior, perception of travelers (willingness to shift) for different travel cost change scenarios were also recorded. Households were categorized into four income group levels based on the statistical Post-Hoc technique to corroborate the effect of income levels on mode shift behavior. After that, travel demand elasticity analysis was carried out with variation in travel cost using Shrinkage Ratio Technique, to comprehend the sensitivity of travelers in different income groups towards mode choice due to changes in travel cost. The results indicated that the low-income group (LIG) travelers are more sensitive to price change compared to the high income group (HIG) travelers. Subsequently, the shift behavior from the existing private modes to public transport was modeled using Binary Logistic Regression for different income groups, to comprehend the combined effect of monthly income and changes in travel cost on mode shift behavior. It was observed that concerning low-income group (LIG), other three income groups enjoyed higher satisfaction (higher utility) for personalized travel mode at a specific value of travel cost and hence, a lesser shift towards public transportation was observed. Interestingly, identical shift behavior was observed among different income groups for a certain range of travel cost values. Further, sensitivity analyses of the developed mode shift model are performed by considering the changes in In-Vehicle Travel time (IVT) and Travel Cost (TC) of present private mode. Indifference curves with travel cost and travel time were then developed based on a non-linear utility function for different income groups. The curves explain the level-of-service and utility difference of available travel modes based on travel cost and travel time.

Bandits with Global Convex Constraints and Objective

Multiarmed bandit (MAB) is a classic model for capturing the exploration–exploitation trade-off inherent in many sequential decision-making problems. The classic MAB framework, however, only allows “local” constraints on decisions and “sum of rewards” as objective. In many real-world applications, there are multiple complex constraints on resources that are consumed during the entire decision process, and performance may be evaluated through nonlinear utility functions on aggregate rewards. This article presents a new MAB framework that allows such “global” convex constraints and concave objective functions along with new algorithmic techniques with provably near-optimal performance bounds. The authors discuss applications in several domains, such as network revenue management, crowdsourcing, and pay-per-click advertising, which benefit from the new more general framework by admitting richer models and more efficient risk-averse solutions.

Measuring and Achieving Equity in Multiperiod Emergency Material Allocation.

Emergency material allocation is an important part of postdisaster emergency logistics that is significant for improving rescue effectiveness and reducing disaster losses. However, the traditional single-period allocation model often causes local surpluses or shortages and high cost, and prevents the system from achieving an equitable or optimal multiperiod allocation. To achieve equitable allocation of emergency materials in the case of serious shortages relative to the demand by victims, this article introduces a multiperiod model for allocation of emergency materials to multiple affected locations (using an exponential utility function to reflect the disutility loss due to material shortfalls), and illustrates the relationship between equity of allocations and the cost of emergency response. Finally, numerical examples are presented to demonstrate both the feasibility and the usefulness of the proposed model for achieving multiperiod equitable allocation of emergency material among multiple disaster locations. The results indicate that the introduction of a nonlinear utility function to reflect the disutility of large shortfalls can make the material allocation fairer, and minimize large losses due to shortfalls. We found that achieving equity has a significant but not unreasonable impact on emergency costs. We also illustrate that using differing utility functions for different types of materials adds an important dimension of flexibility.

An Analysis of Decision under Risk in Rats

In 1979, Daniel Kahneman and Amos Tversky published a ground-breaking paper titled "Prospect Theory: An Analysis of Decision under Risk," which presented a behavioral economic theory that accounted for the ways in which humans deviate from economists' normative workhorse model, Expected Utility Theory [1, 2]. For example, people exhibit probability distortion (they overweight low probabilities), loss aversion (losses loom larger than gains), and reference dependence (outcomes are evaluated as gains or losses relative to an internal reference point). We found that rats exhibited many of these same biases, using a task in which rats chose between guaranteed and probabilistic rewards. However, prospect theory assumes stable preferences in the absence of learning, an assumption at odds with alternative frameworks such as animal learning theory and reinforcement learning [3-7]. Rats also exhibited trial history effects, consistent with ongoing learning. A reinforcement learning model in which state-action values were updated by the subjective value of outcomes according to prospect theory reproduced rats' nonlinear utility and probability weighting functions and also captured trial-by-trial learning dynamics.

COMPARING BEHAVIORAL MODELS USING DATA FROM EXPERIMENTAL CENTIPEDE GAMES

The centipede game posits one of the most well‐known paradoxes of backward induction in the literature of experimental game theory. Given that deviations from the unique subgame perfect Nash equilibrium generates a Pareto improvement, several theoretical models have been employed in order to rationalize this kind of behavior in this social dilemma. The available explanations range from social preferences including fairness, altruism or cooperation motives, errors in playing, inability to perform backward induction or different depths of reasoning. In the present study, we use the Blavatskyy's theoretical contribution, and relax the assumptions of Expected Utility maximization and risk‐neutral attitudes, to test an alternative explanation. We compare various probabilistic decision theory models in terms of their descriptive (in‐sample) and predictive (out‐of‐sample fit) performance, using data from experimental centipede games. We find that introducing non‐Expected Utility preferences to the Quantal Response Equilibrium model, along with a nonlinear utility function, provides a better explanation compared to alternative specifications such as the Level‐k or the Quantal Response Equilibrium model with altruistic motives. (JEL C72, C92, D81, D82)

Forecasting household consumption of fuels: A multiple discrete-continuous approach

This paper uses a multiple discrete–continuous extreme value (MDCEV) model with perfect and imperfect substitutes to study residential energy demand. A non-linear utility function is employed within a Kuhn-Tucker multiple-discrete economic model of consumer demand, estimated on Italian expenditure data. The simulation algorithm measures demand elasticity with respect to price variations and the marginal effects of other covariates. Results show that household energy demand (space and water heating and transportation) is relatively inelastic with respect to prices (−0.55 and −0.67, respectively), meaning that pricing policies can induce a reduction in the demand for fuels less than proportional to the price variation. The model also allows to forecast the energy demand for space and water heating within a global warming scenario: an increase of 2 degrees Celsius would lead, for example, to a reduction in household energy consumption of 4.07%.