Risk-sensitive Control Research Articles

Risk Sensitive Control of the Lifetime Ruin Problem

We study a risk sensitive control version of the lifetime ruin probability problem. We consider a sequence of investments problems in Black-Scholes market that includes a risky asset and a riskless asset. We present a differential game that governs the limit behavior. We solve it explicitly and use it in order to find an asymptotically optimal policy.

Sufficient Stochastic Maximum Principle for the Optimal Control of Semi-Markov Modulated Jump-Diffusion with Application to Financial Optimization

The finite state semi-Markov process is a generalization over the Markov chain in which the sojourn time distribution is any general distribution. In this article, we provide a sufficient stochastic maximum principle for the optimal control of a semi-Markov modulated jump-diffusion process in which the drift, diffusion, and the jump kernel of the jump-diffusion process is modulated by a semi-Markov process. We also connect the sufficient stochastic maximum principle with the dynamic programming equation. We apply our results to finite horizon risk-sensitive control portfolio optimization problem and to a quadratic loss minimization problem.

Jump-diffusion asset–liability management via risk-sensitive control

In this paper, we use risk-sensitive control methods to solve a jump-diffusion asset–liability management (ALM) problem. We show that the ALM problem admits a unique classical (\(C^{1,2}\)) solution under two different sets of assumptions.

An Adversarial Interpretation of Information-Theoretic Bounded Rationality

Recently, there has been a growing interest in modeling planning with information constraints. Accordingly, an agent maximizes a regularized expected utility known as the free energy, where the regularizer is given by the information divergence from a prior to a posterior policy. While this approach can be justified in various ways, including from statistical mechanics and information theory, it is still unclear how it relates to decision-making against adversarial environments. This connection has previously been suggested in work relating the free energy to risk-sensitive control and to extensive form games. Here, we show that a single-agent free energy optimization is equivalent to a game between the agent and an imaginary adversary. The adversary can, by paying an exponential penalty, generate costs that diminish the decision maker's payoffs. It turns out that the optimal strategy of the adversary consists in choosing costs so as to render the decision maker indifferent among its choices, which is a definining property of a Nash equilibrium, thus tightening the connection between free energy optimization and game theory.

Linear Quadratic Optimal and Risk-Sensitive Control for Vehicle Active Suspensions

This paper is concerned with time-domain optimal control of active suspensions. The optimal control problem formulation is generalized by incorporating both road disturbances (ride quality) and a representation of driver inputs (handling quality) into the optimal control formulation. A regular optimal control problem as well as a risk-sensitive exponential optimal control performance index is considered. Emphasis is placed on practical considerations including the issue of state estimation in the presence of load disturbances (driver inputs).

Risk-sensitive control of continuous time Markov chains

We study risk-sensitive control of continuous time Markov chains taking values in discrete state space. We study both finite and infinite horizon problems. In the finite horizon problem we characterize the value function via Hamilton Jacobi Bellman equation and obtain an optimal Markov control. We do the same for infinite horizon discounted cost case. In the infinite horizon average cost case we establish the existence of an optimal stationary control under certain Lyapunov condition. We also develop a policy iteration algorithm for finding an optimal control.

Asynchronous Gossip for Averaging and Spectral Ranking

We consider two variants of the classical gossip algorithm. The first variant is a version of asynchronous stochastic approximation. We highlight a fundamental difficulty associated with the classical asynchronous gossip scheme, viz., that it may not converge to a desired average, and suggest an alternative scheme based on reinforcement learning that has guaranteed convergence to the desired average. We then discuss a potential application to a wireless network setting with simultaneous link activation constraints. The second variant is a gossip algorithm for distributed computation of the Perron-Frobenius eigenvector of a nonnegative matrix. While the first variant draws upon a reinforcement learning algorithm for an average cost controlled Markov decision problem, the second variant draws upon a reinforcement learning algorithm for risk-sensitive control. We then discuss potential applications of the second variant to ranking schemes, reputation networks, and principal component analysis.

Some Results on Optimal Control for a Partially Observed Linear Stochastic System with an Exponential Quadratic Cost

A control problem for a partially observed linear stochastic system with an exponential quadratic cost functional is formulated and explicitly solved. It is assumed given that the estimation of the state is described by the solution of the information filter which is known. This solution is a sufficient statistic for the unknown state based on the observations. In this paper an optimal control is determined explicitly in a simple, direct manner from this sufficient statistic. This approach does not use either the solution of a Hamilton-Jacobi-Bellman equation or a stochastic maximum principle with backward stochastic differential equations. This control problem is often called a linear partially observed risk sensitive control problem.

Pessimistic Optimal Choice for Risk-Averse Agents

We propose a general framework for the analysis of dynamic optimization with risk-averse agents, extending Whittle's (Whittle, 1990) formulation of risk-sensitive optimal control problems to accommodate time-discounting. We show how, within a Markovian set-up, optimal risk-averse behavior is identified via a pessimistic choice mechanism and described by simple recursive formulae. We apply this methodology to two distinct problems formulated respectively in discrete- and continuous-time. In the former, we extend Svensson's (Svensson, 1997) analysis of optimal monetary policy, showing that with a pessimistic central bank the inflation forecast is not longer an explicit intermediate target, the monetary authorities do not expect the inflation rate to mean revert to its target level and apply a more aggressive Taylor rule, while the inflation rate is less volatile. In the latter, we investigate the optimal production policy of a monopolistic entrepreneur which faces a demand schedule subject to stochastic shocks, once again showing that risk-aversion induces her to act more aggressively.

Definable Zero-Sum Stochastic Games

Definable zero-sum stochastic games involve a finite number of states and action sets, and reward and transition functions, that are definable in an o-minimal structure. Prominent examples of such games are finite, semi-algebraic, or globally subanalytic stochastic games. We prove that the Shapley operator of any definable stochastic game with separable transition and reward functions is definable in the same structure. Definability in the same structure does not hold systematically: we provide a counterexample of a stochastic game with semi-algebraic data yielding a non-semi-algebraic but globally subanalytic Shapley operator. Our definability results on Shapley operators are used to prove that any separable definable game has a uniform value; in the case of polynomially bounded structures, we also provide convergence rates. Using an approximation procedure, we actually establish that general zero-sum games with separable definable transition functions have a uniform value. These results highlight the key role played by the tame structure of transition functions. As particular cases of our main results, we obtain that stochastic games with polynomial transitions, definable games with finite actions on one side, and definable games with perfect information or switching controls have a uniform value. Applications to nonlinear maps arising in risk sensitive control and Perron-Frobenius theory are also given.

Risk-sensitive control for a class of nonlinear systems with multiplicative noise

In this paper, we consider the problem of optimal control for a class of nonlinear stochastic systems with multiplicative noise. The nonlinearity consists of quadratic terms in the state and control variables. The optimality criteria are of a risk-sensitive and generalised risk-sensitive type. The optimal control is found in an explicit closed-form by the completion of squares and the change of measure methods. As applications, we outline two special cases of our results. We show that a subset of the class of models which we consider leads to a generalised quadratic–affine term structure model (QATSM) for interest rates. We also demonstrate how our results lead to generalisation of exponential utility as a criterion in optimal investment.

Risk-Sensitive Control of Pure Jump Process on Countable Space with Near Monotone Cost

In this article, we study risk-sensitive control problem with controlled continuous time pure jump process on a countable space as state dynamics. We prove multiplicative dynamic programming principle, elliptic and parabolic Harnack’s inequalities. Using the multiplicative dynamic programing principle and the Harnack’s inequalities, we prove the existence and a characterization of optimal risk-sensitive control under the near monotone condition.

Second order backward stochastic differential equations with quadratic growth

We extend the well posedness results for second order backward stochastic differential equations introduced by Soner, Touzi and Zhang (2012) [31] to the case of a bounded terminal condition and a generator with quadratic growth in the z variable. More precisely, we obtain uniqueness through a representation of the solution inspired by stochastic control theory, and we obtain two existence results using two different methods. In particular, we obtain the existence of the simplest purely quadratic 2BSDEs through the classical exponential change, which allows us to introduce a quasi-sure version of the entropic risk measure. As an application, we also study robust risk-sensitive control problems. Finally, we prove a Feynman–Kac formula and a probabilistic representation for fully non-linear PDEs in this setting.

Closed-loop and robust control of quantum systems.

For most practical quantum control systems, it is important and difficult to attain robustness and reliability due to unavoidable uncertainties in the system dynamics or models. Three kinds of typical approaches (e.g., closed-loop learning control, feedback control, and robust control) have been proved to be effective to solve these problems. This work presents a self-contained survey on the closed-loop and robust control of quantum systems, as well as a brief introduction to a selection of basic theories and methods in this research area, to provide interested readers with a general idea for further studies. In the area of closed-loop learning control of quantum systems, we survey and introduce such learning control methods as gradient-based methods, genetic algorithms (GA), and reinforcement learning (RL) methods from a unified point of view of exploring the quantum control landscapes. For the feedback control approach, the paper surveys three control strategies including Lyapunov control, measurement-based control, and coherent-feedback control. Then such topics in the field of quantum robust control as H ∞ control, sliding mode control, quantum risk-sensitive control, and quantum ensemble control are reviewed. The paper concludes with a perspective of future research directions that are likely to attract more attention.

Jump-Diffusion Asset-Liability Management via Risk-Sensitive Control

In this paper, we use risk-sensitive control methods to solve a jump-diffusion Asset-Liability Management (ALM) problem. We show that the ALM problem admits a unique classical solution under two different sets of assumptions.

Forward–Backward Stochastic Differential Games and Stochastic Control under Model Uncertainty

We study optimal stochastic control problems with jumps under model uncertainty. We rewrite such problems as stochastic differential games of forward---backward stochastic differential equations. We prove general stochastic maximum principles for such games, both in the zero-sum case (finding conditions for saddle points) and for the nonzero sum games (finding conditions for Nash equilibria). We then apply these results to study robust optimal portfolio-consumption problems with penalty. We establish a connection between market viability under model uncertainty and equivalent martingale measures. In the case with entropic penalty, we prove a general reduction theorem, stating that a optimal portfolio-consumption problem under model uncertainty can be reduced to a classical portfolio-consumption problem under model certainty, with a change in the utility function, and we relate this to risk sensitive control. In particular, this result shows that model uncertainty increases the Arrow---Pratt risk aversion index.

Generalised Risk-Sensitive Control with Full and Partial State Observation

This paper generalises the risk-sensitive cost functional by introducing noise dependent penalties on the state and control variables. The optimal control problems for the full and partial state observation are considered. Using a change of probability measure approach, explicit closed-form solutions are found in both cases. This has resulted in a new risk-sensitive regulator and filter, which are generalisations of the well-known classical results.

Complete Statistical Characterization of Discrete-Time LQG and Cumulant Control

As the performance index of the linear-quadratic-Gaussian (LQG) problem is governed by the noncentral generalized Chi-square distribution, solely controlling the expected value of the performance index, as the traditional LQG theory aims at, is insufficient to deliver a satisfactory solution in some situations. While the risk sensitive control does control a specific weighting sum of various moments of the performance index, the single degree of freedom in adjusting the weighting coefficients in this specific weighting sum of various moments of the performance index often prevents the risk sensitive control from generating a desired pattern of high order moment-distribution. We achieve in this note the complete statistical characterization of the performance index for the discrete-time LQG formulation. More specifically, we derive a recursive relationship to obtain cumulants of various orders of the performance index successively. Parameterized in feedback gain, the optimal feedback control law can be computed off-line by solving a static polynomial optimization problem, thus serving two design goals: (i) To shape the probability density function (pdf) of the performance index to attain, at least approximately, a given desired pattern by regulating cumulants of various orders, and (ii) to improve the performance measure of an incumbent solution (generated by the risk sensitive control, for example) by adjusting the levels of cumulants of various orders.

Oja’s algorithm for graph clustering, Markov spectral decomposition, and risk sensitive control

Given a positive definite matrix M and an integer Nm≥1, Oja’s subspace algorithm will provide convergent estimates of the first Nm eigenvalues of M along with the corresponding eigenvectors. It is a common approach to principal component analysis. This paper introduces a normalized stochastic-approximation implementation of Oja’s subspace algorithm, as well as new applications to the spectral decomposition of a reversible Markov chain. Recall that this means that the stationary distribution satisfies the detailed balance equations (Meyn & Tweedie, 2009). Equivalently, the statistics of the process in steady state do not change when time is reversed. Stability and convergence of Oja’s algorithm are established under conditions far milder than that assumed in previous work. Applications to graph clustering, Markov spectral decomposition, and multiplicative ergodic theory are surveyed, along with numerical results.

Exploiting market fluctuations and price volatility through feedback control

We explore how feedback control can make chemical producers responsive to market forces through the use of dynamic operating policies. Using a toy model of a marginal chemical producer operating in a dynamic market, we examine two different control strategies for dealing with stochastic price fluctuations. The key contribution of this work is the derivation of switching rules under risk-neutral and risk-sensitive control formulations for problems where the dynamics arise from the market. These results provide a basis for exploring more complex control problems that include the effects of market forces.