Risk-sensitive Criterion Research Articles

Risk-sensitive benchmarked portfolio optimization under non-linear market dynamics

We discuss a continuous-time portfolio optimization problem to beat a stochastic benchmark. The model considers non-linear stochastic differential equations (SDEs) to model the dynamics of assets and economic factors. Unlike existing literature on risk-sensitive criteria, the proposed framework allows the model to capture the non-linearity in assets and factors dynamics. This article contributes to the two essential aspects of the problem; first, the existence and uniqueness of optimal investment strategy, which we prove using stochastic control theory and shows optimal strategies remain unchanged for the finite and infinite time horizon problems. Second, we use forecasting to compare investment performance under different economic factors. We analyze returns for the proposed model for three years in two important financial markets; S&P 100 and Dow 30. Risk versus return analysis indicates the importance of choosing relevant economic factors and their model for better portfolio performance. The results suggest an excellent bet to select the non-linear model over the linear one.

Covariance-analytic performance criteria, Hardy-Schatten norms and Wick-like ordering of cascaded systems

This paper is concerned with robust performance analysis of linear stochastic systems acting as an integral operator on a standard Wiener process at the input and producing a stationary Gaussian random process at the output. We propose a performance criterion which uses the trace of an analytic function of the spectral density of the output process. This class of “covariance-analytic” cost functionals includes the usual mean square and risk-sensitive criteria as particular cases. Due to the “cost-shaping” analytic function, the covariance-analytic performance criterion depends on higher-order Hardy-Schatten norms of the system transfer function. We discuss the links of these norms with the asymptotic behaviour of cumulants of finite-horizon quadratic functionals of the system output and their variational properties pertaining to system robustness to statistically uncertain inputs which differ from the standard Wiener process. In the case of strictly proper finite-dimensional systems, governed in state space by linear stochastic differential equations, we develop a method for recursively computing the Hardy-Schatten norms through a recently proposed technique of rearranging cascaded linear systems, which resembles the Wick ordering of mixed products of noncommuting annihilation and creation operators in quantum mechanics. This computational procedure, which is one of the main results of the paper, involves a recurrence sequence of solutions to algebraic Lyapunov equations and represents the covariance-analytic cost as the squared H2-norm of an auxiliary cascaded system. These results are also compared with an alternative approach, which uses higher-order derivatives of stabilising solutions of parameter-dependent algebraic Riccati equations, and illustrated by a numerical example.

Risk sensitive filtering with randomly delayed measurements

Conventional Bayesian estimation requires an accurate stochastic model of a system. However, this requirement is not always met in many practical cases where the system is not completely known or may differ from the assumed model. For such a system, we consider a scenario where the measurements are transmitted to a remote location using a common communication network and due to which, a delay is introduced while receiving the measurements. The delay that we consider here is random and one step maximum at a given time instant. For such a scenario, this paper develops a robust estimator for a linear Gaussian system by minimizing the risk sensitive error criterion that is defined as an expectation of the accumulated exponential quadratic error. The criteria for the stability of the risk sensitive Kalman filter (RSKF) are derived and the results are used to study the stability of the developed filter. Further, it is assumed that the latency probability related to delay is not known and it is estimated by maximizing the likelihood function. Simulation results suggest that the proposed filter shows acceptable performance under the nominal conditions, and it performs better than the Kalman filter for randomly delayed measurements and the RSKF in presence of both the model uncertainty and random delays.

Continuous-time zero-sum games for Markov chains with risk-sensitive finite-horizon cost criterion

In this paper, we study the two-person zero-sum stochastic games for controlled continuous time Markov chains with risk-sensitive finite-horizon cost criterion. The transition and cost rates are possibly unbounded. For the zero-sum risk-sensitive stochastic game, we prove the existence of the value of the game and a Markov saddle-point equilibrium in the class of all history-dependent multi-strategies under the suitable conditions. We achieve our results by studying the corresponding risk-sensitive finite-horizon optimality equations.

Robust event-triggered state estimation: A risk-sensitive approach

In this work, we investigate a robust event-triggered remote state estimation problem for linear Gaussian systems with a stochastic event-triggering condition. The reference measure approach is used to obtain a robust event-triggered estimate that minimizes the so-called risk-sensitive criterion, which refers to the expectation of the exponential of the sum of the squared estimation error. We introduce the reference measure, under which, the measurements are identically independently distributed (i.i.d.) and independent of the states, and propose a map to link the “real-world” measure to the reference measure so that the recursions of the information states under the reference measure can be obtained. Based on these results, the risk-sensitive criteria are reformulated under the reference measure and closed-form expressions of the risk-sensitive event-triggered posterior and prior estimates are presented, which are shown to evolve in simple recursive Kalman-like structures. Moreover, two sufficient stability conditions for the proposed estimators are given, where the first requires the solution of a time-varying Riccati equation to be positive-definite and satisfy a specific inequality, which can be further extended to the scenario when the weighting matrix in the risk-sensitive criterion is time-variant; the second gives the range of values of the risk-sensitive parameter and covariance of the initial state for which the proposed estimators are stable. Comparative simulation results demonstrate that the proposed risk-sensitive event-triggered estimator is more robust to model uncertainties compared with a typical minimum mean squared error (MMSE) estimator with stochastic event-triggered sensor scheduling.

Probabilistically distorted risk-sensitive infinite-horizon dynamic programming

Historically, the study of risk-sensitive criteria has focused on their normative applications — i.e., what should be done. The classic example is expected utility functions which produce deterministic policies. More recently, the literature on dynamic coherent risk measures has broadened the choices for risk-sensitive performance evaluation. However, coherent risk measures must be convex. This paper presents an alternative to both the expected utility and coherent risk measure approaches. This new approach, inspired by cumulative prospect theory (CPT), is nonconvex and has substantial empirical evidence supporting its descriptive power for human decisions, i.e., what is actually done. A key unique feature of the CPT-based approach, essential for modeling human decisions, is probabilistic distortion. Hence, CPT should be used instead of both expected utility and coherent risk measures when modeling human decisions, which requires a higher level of expressiveness than allowed by previous work. In addition, although both coherent risk measures and CPT produce randomized policies, which are more robust against inaccurate probabilistic descriptions of systems, CPT generates policies that are significantly different from those of coherent risk measures.

Multi-point Gaussian States, Quadratic–Exponential Cost Functionals, and Large Deviations Estimates for Linear Quantum Stochastic Systems

This paper is concerned with risk-sensitive performance analysis for linear quantum stochastic systems interacting with external bosonic fields. We consider a cost functional in the form of the exponential moment of the integral of a quadratic polynomial of the system variables over a bounded time interval. Such functionals are related to more conservative behaviour and robustness of systems with respect to statistical uncertainty, which makes the challenging problems of their computation and minimization practically important. To this end, we obtain an integro-differential equation for the time evolution of the quadratic–exponential functional, which is different from the original quantum risk-sensitive performance criterion employed previously for measurement-based quantum control and filtering problems. Using multi-point Gaussian quantum states for the past history of the system variables and their first four moments, we discuss a quartic approximation of the cost functional and its infinite-horizon asymptotic behaviour. The computation of the asymptotic growth rate of this approximation is reduced to solving two algebraic Lyapunov equations. Further approximations of the cost functional, based on higher-order cumulants and their growth rates, are applied to large deviations estimates in the form of upper bounds for tail distributions. We discuss an auxiliary classical Gaussian–Markov diffusion process in a complex Euclidean space which reproduces the quantum system variables at the level of covariances but has different fourth-order cumulants, thus showing that the risk-sensitive criteria are not reducible to quadratic–exponential moments of classical Gaussian processes. The results of the paper are illustrated by a numerical example and may find applications to coherent quantum risk-sensitive control problems, where the plant and controller form a fully quantum closed-loop system, and other settings with nonquadratic cost functionals.

Parallel Nonstationary Direct Policy Search for Risk-Averse Stochastic Optimization

This paper presents an algorithmic strategy to nonstationary policy search for finite-horizon, discrete-time Markovian decision problems with large state spaces, constrained action sets, and a risk-sensitive optimality criterion. The methodology relies on modeling time-variant policy parameters by a nonparametric response surface model for an indirect parametrized policy motivated by Bellman’s equation. The policy structure is heuristic when the optimization of the risk-sensitive criterion does not admit a dynamic programming reformulation. Through the interpolating approximation, the level of nonstationarity of the policy, and consequently, the size of the resulting search problem can be adjusted. The computational tractability and the generality of the approach follow from a nested parallel implementation of derivative-free optimization in conjunction with Monte Carlo simulation. We demonstrate the efficiency of the approach on an optimal energy storage charging problem, and illustrate the effect of the risk functional on the improvement achieved by allowing a higher complexity in time variation for the policy. The online supplement is available at https://doi.org/10.1287/ijoc.2016.0733 .

Average cost criterion induced by the regular utility function for continuous-time Markov decision processes

In this paper we study the average cost criterion induced by the regular utility function (U-average cost criterion) for continuous-time Markov decision processes. This criterion is a generalization of the risk-sensitive average cost and expected average cost criteria. We first introduce an auxiliary risk-sensitive first passage optimization problem and obtain the properties of the corresponding optimal value function under the slight conditions. Then we show that the pair of the optimal value functions of the risk-sensitive average cost criterion and the risk-sensitive first passage criterion is a solution to the optimality equation of the risk-sensitive average cost criterion allowing the risk-sensitivity parameter to take any nonzero value. Moreover, we have that the optimal value function of the risk-sensitive average cost criterion is continuous with respect to the risk-sensitivity parameter. Finally, we give the connections between the U-average cost criterion and the average cost criteria induced by the identity function and the exponential utility function, and prove the existence of a U-average optimal deterministic stationary policy in the class of all randomized Markov policies.

Variance-constrained actor-critic algorithms for discounted and average reward MDPs

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms that operate on three timescales--a TD critic on the fastest timescale, a policy gradient (actor) on the intermediate timescale, and a dual ascent for Lagrange multipliers on the slowest timescale. In the discounted setting, we point out the difficulty in estimating the gradient of the variance of the return and incorporate simultaneous perturbation approaches to alleviate this. The average setting, on the other hand, allows for an actor update using compatible features to estimate the gradient of the variance. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application.

Long run risk sensitive portfolio with general factors

In the paper portfolio optimization over long run risk sensitive criterion is considered. It is assumed that economic factors which stimulate asset prices are ergodic but non necessarily uniformly ergodic. Solution to suitable Bellman equation using local span contraction with weighted norms is shown. The form of optimal strategy is presented and examples of market models satisfying imposed assumptions are shown.

Dynamic Limit Growth Indices in Discrete Time

We propose a new class of mappings, called Dynamic Limit Growth Indices, that are designed to measure the long-run performance of a financial portfolio in discrete time setup. We study various important properties for this new class of measures, and in particular, we provide necessary and sufficient condition for a Dynamic Limit Growth Index to be a dynamic assessment index. We also establish their connection with classical dynamic acceptability indices, and we show how to construct examples of Dynamic Limit Growth Indices using dynamic risk measures and dynamic certainty equivalents. Finally, we propose a new definition of time consistency, suitable for these indices, and we study time consistency for the most notable representative of this class—the dynamic analog of risk-sensitive criterion.

Probabilistic Planning with Risk-Sensitive Criterion

While probabilistic planning models have been extensively used by AI and Decision Theoretic communities for planning under uncertainty, the objective to minimize the expected cumulative cost is inappropriate for high-stake planning problems. With this motivation in mind, we revisit the Risk-Sensitive criterion (RS-criterion), where the objective is to find a policy that maximizes the probability that the cumulative cost is within some user-defined cost threshold. The overall scope of this research is to develop efficient and scalable algorithms to optimize the RS-criterion in probabilistic planning problems. In our recent paper (Hou, Yeoh, and Varakantham 2014), we formally defined Risk-Sensitive MDPs (RS-MDPs) and introduced new algorithms for RS-MDPs with non-negative costs. Next, my plan is to develop algorithm for RS-MDPs with negative cost cycles and for Risk-Sensitive POMDPs (RS-POMDPs).

Output‐feedback control for a class of Markovian jump non‐linear systems under a risk‐sensitive cost criterion with unknown transition probabilities

This study addresses the output-feedback control problem for a class of Markovian jump non-linear systems under a quadratic risk-sensitive cost function criterion. The transition probabilities of the Markov process are assumed to be completely unknown. By employing the high-gain scaling technique, common Lyapunov function method and backstepping technique, a control law is constructed that guarantees any arbitrary small risk-sensitive cost for a given risk-sensitive parameter. Moreover, the resulted closed-loop system solutions are bounded in probability. Compared with some previous results, this study does not require the uniform boundedness of the gain functions of the system noise, and the control law further achieves a zero risk-sensitive cost and asymptotically stable in the large for the closed-loop system solutions when the vector field of the disturbance vanishes at the origin. A numerical example is given to illustrate the proposed protocol.

More Risk-Sensitive Markov Decision Processes

We investigate the problem of minimizing a certainty equivalent of the total or discounted cost over a finite and an infinite horizon that is generated by a Markov decision process (MDP). In contrast to a risk-neutral decision maker this optimization criterion takes the variability of the cost into account. It contains as a special case the classical risk-sensitive optimization criterion with an exponential utility. We show that this optimization problem can be solved by an ordinary MDP with extended state space and give conditions under which an optimal policy exists. In the case of an infinite time horizon we show that the minimal discounted cost can be obtained by value iteration and can be characterized as the unique solution of a fixed-point equation using a “sandwich” argument. Interestingly, it turns out that in the case of a power utility, the problem simplifies and is of similar complexity than the exponential utility case, however has not been treated in the literature so far. We also establish the validity (and convergence) of the policy improvement method. A simple numerical example, namely, the classical repeated casino game, is considered to illustrate the influence of the certainty equivalent and its parameters. Finally, the average cost problem is also investigated. Surprisingly, it turns out that under suitable recurrence conditions on the MDP for convex power utility, the minimal average cost does not depend on the parameter of the utility function and is equal to the risk-neutral average cost. This is in contrast to the classical risk-sensitive criterion with exponential utility.

Variable risk control via stochastic optimization

We present new global and local policy search algorithms suitable for problems with policy-dependent cost variance (or risk), a property present in many robot control tasks. These algorithms exploit new techniques in non-parametric heteroscedastic regression to directly model the policy-dependent distribution of cost. For local search, the learned cost model can be used as a critic for performing risk-sensitive gradient descent. Alternatively, decision-theoretic criteria can be applied to globally select policies to balance exploration and exploitation in a principled way, or to perform greedy minimization with respect to various risk-sensitive criteria. This separation of learning and policy selection permits variable risk control, where risk-sensitivity can be flexibly adjusted and appropriate policies can be selected at runtime without relearning. We describe experiments in dynamic stabilization and manipulation with a mobile manipulator that demonstrate learning of flexible, risk-sensitive policies in very few trials.

Risk-Sensitive Control for the Parallel Server Model

A Markovian queueing model is considered in which servers of various types work in parallel to process jobs from a number of classes at rates $\mu_{ij}$ that depend on the class, $i$, and the type, $j$. The problem of dynamic resource allocation so as to minimize a risk-sensitive criterion is studied in a law-of-large-numbers scaling. Letting $X_i(t)$ denote the number of class-$i$ jobs in the system at time $t$, the cost is given by $E\exp\{n[\int_0^Th(\bar X(t))dt+g(\bar X(T))]\}$, where $T>0$, $h$ and $g$ are given functions satisfying regularity and growth conditions, and $\bar X=\bar X^n=n^{-1}X(n\cdot)$. It is well known in an analogous context of controlled diffusion, and has been shown for some classes of stochastic networks, that the limit behavior, as $n\to\infty$, is governed by a differential game (DG) in which the state dynamics is given by a fluid equation for the formal limit $\varphi$ of $\bar X$, while the cost consists of $\int_0^Th (\varphi(t))dt+g(\varphi(T))$ and an additional term th...

Risk-Sensitive Markov Control Processes

We introduce a general framework for measuring risk in the context of Markov control processes with risk maps on general Borel spaces that generalize known concepts of risk measures in mathematical finance, operations research and behavioral economics. Within the framework, applying weighted norm spaces to incorporate also unbounded costs, we study two types of infinite-horizon risk-sensitive criteria, discounted total risk and average risk, and solve the associated optimization problems by dynamic programming. For the discounted case, we propose a new discount scheme, which is different from the conventional form but consistent with the existing literature, while for the average risk criterion, we state Lyapunov-like stability conditions that generalize known conditions for Markov chains to ensure the existence of solutions to the optimality equation.

Expected Power-Utility Maximization Under Incomplete Information and with Cox-Process Observations

We consider the problem of maximization of expected terminal power utility (risk sensitive criterion). The underlying market model is a regime-switching diffusion model where the regime is determined by an unobservable factor process forming a finite state Markov process. The main novelty is due to the fact that prices are observed and the portfolio is rebalanced only at random times corresponding to a Cox process where the intensity is driven by the unobserved Markovian factor process as well. This leads to a more realistic modeling for many practical situations, like in markets with liquidity restrictions; on the other hand it considerably complicates the problem to the point that traditional methodologies cannot be directly applied. The approach presented here is specific to the power-utility. For log-utilities a different approach is presented in Fujimoto et al. (Preprint, 2012).

Quantifying the risk in age and block replacement policies

Preventive maintenance policies have been studied in the literature without considering the risk due to the cost variability. In this paper, we consider the two most popular preventive replacement policies, namely, age and block replacement policies under long-run average cost and expected unit time cost criteria. To quantify the risk in the preventive maintenance policies, we use the long-run variance of the accumulated cost over a time interval. We numerically derive the Risk-sensitive preventive replacement policies and study the impact of the Risk-sensitive optimality criterion on the managerial decisions. We also examine the performance of the expected unit time cost criterion as an alternative to the traditional long-run average cost criterion.