Risk-sensitive Optimal Control Research Articles

Lyapunov’s stability analysis for first degree polynomial systems, subject to risk-sensitive control

This work presents a novel methodology to verify the stability of first-degree stochastic polynomial systems under a Risk-Sensitive (RS) optimal control using Lyapunov’s stability analysis theory. The so-called Lyapunov’s indirect method is applied to prove its stability when it is combined with the dynamics of Riccati’s gain equation. The proposed methodology guarantees both the exponential stability of deterministic systems and the robustness of stochastic systems. Simulation results demonstrate the robustness, the effectiveness and the feasibility of this proposal.

Discounted Risk-Sensitive Optimal Control of Switching Diffusions: Viscosity Solution and Numerical Approximation

This work considers the infinite horizon discounted risk-sensitive optimal control problem for the switching diffusions with a compact control space and controlled through the drift; thus, the the generator of the switching diffusions also depends on the controls. Note that the running cost of interest can be unbounded, so a decent estimation on the value function is obtained, under suitable conditions. To solve such a risk-sensitive optimal control problem, we adopt the viscosity solution methods and propose a numerical approximation scheme. We can verify that the value function of the optimal control problem solves the optimality equation as the unique viscosity solution. The optimality equation is also called the Hamilton–Jacobi–Bellman (HJB) equation, which is a second-order partial differential equation (PDE). Since, the explicit solutions to such PDEs are usually difficult to obtain, the finite difference approximation scheme is derived to approximate the value function. As a byproduct, the ϵ-optimal control of finite difference type is also obtained.

Risk-sensitive Optimal Control with Forward-looking Variables

Risk-sensitive Optimal Control with Forward-looking Variables

Risk-sensitive discounted cost criterion for continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton–Jacobi–Bellman equation. Also, we prove the existence of optimal risk-sensitive control in the class of Markov control and completely characterized the optimal control.

EFFECT OF RISK-SENSITIVE STOCHASTIC OPTIMAL CONTROL WITH TRACKING IN AN EVAPORATOR

This work presents an application of the Risk-Sensitive (R-S) control with tracking applied to a stochastic nonlinear system which models the operation of an electronic expansion valve (EEV) in a conventional evaporator. A novel dynamical stochastic equation represents the mathematical model of the evaporator system. The R-S stochastic optimal problem consists of the design of an optimal control u(t) such that the state reaches setpoint values (SP) and minimizes the exponential quadratic cost function. The presence of disturbances and errors in the sensor measurements is represented by Gauss white noise in the state equation, with the coefficient v(e/(2?^2 )) . One novel characteristic in this proposal is that the coefficient of the control into the state equation contains the state term. The error and exponential quadratic cost function show that the R-S control has a better performance versus the classical PID (Proportional, Integral Derivative) control. Key Words: Optimal Risk-Sensitive control with tracking, modelling of the evaporator.

RAT iLQR: A Risk Auto-Tuning Controller to Optimally Account for Stochastic Model Mismatch

Successful robotic operation stochastic environments relies on accurate characterization of the underlying probability distributions, yet this is often imperfect due to limited knowledge. This work presents a control algorithm that is capable of handling such distributional mismatches. Specifically, we propose a novel nonlinear MPC for distributionally robust control, which plans locally optimal feedback policies against a worst-case distribution within a given KL divergence bound from a Gaussian distribution. Leveraging mathematical equivalence between distributionally robust control and risk-sensitive optimal control, our framework also provides an algorithm to dynamically adjust the risk-sensitivity level online for risk-sensitive control. The benefits of the distributional robustness as well as the automatic risk-sensitivity adjustment are demonstrated in a dynamic collision avoidance scenario where the predictive distribution of human motion is erroneous.

The risk-sensitive maximum principle for controlled forward–backward stochastic differential equations

In this paper, we consider the risk-sensitive optimal control problem for forward–backward stochastic differential equations (FBSDEs). We consider two different cases: (Case 1) the drift and diffusion terms of the forward part are independent of the backward part (not fully-coupled FBSDE) and the running cost does not include the backward part; (Case 2) the fully-coupled FBSDE with terminal and initial costs. In both cases, the risk-sensitive objective functional is considered, the corresponding control domain is not necessarily convex, and the diffusion term of the forward part depends on control and state variables. In both cases, by using the nonlinear transformations of the equivalent risk-neutral problems, we obtain two (different) risk-sensitive maximum principles, which are characterized in terms of the variational inequalities. We show that the risk-sensitive maximum principle of (Case 1) consists of the two-coupled adjoint equations with an additional scalar adjoint equation, which are backward stochastic differential equations (BSDEs). The risk-sensitive maximum principle of (Case 2) consists of the two-coupled adjoint equations, which are coupled FBSDEs, with an additional scalar adjoint equation and optimality conditions. In both cases, the additional scalar adjoint equations are BSDEs, which are induced due to the nonlinear transformations of the adjoint processes in the equivalent risk-neutral problems. Through the application of results, we consider the linear–quadratic risk-sensitive optimal control problem for the linear FBSDE, and by using the risk-sensitive maximum principle of (Case 1), we characterize an explicit optimal solution.

Generalized Risk-Sensitive Optimal Control and Hamilton–Jacobi–Bellman Equation

In this article, we consider the generalized risk-sensitive optimal control problem, where the objective functional is defined by the controlled backward stochastic differential equation (BSDE) with quadratic growth coefficient. We extend the earlier results of the risk-sensitive optimal control problem to the case of the objective functional given by the controlled BSDE. Note that the risk-neutral stochastic optimal control problem corresponds to the BSDE objective functional with linear growth coefficient, which can be viewed as a special case of the article. We obtain the generalized risk-sensitive dynamic programming principle for the value function via the backward semigroup associated with the BSDE. Then we show that the corresponding value function is a viscosity solution to the Hamilton-Jacobi-Bellman equation. Under an additional parameter condition, the viscosity solution is unique, which implies that the solution characterizes the value function. We apply the theoretical results to the risk-sensitive European option pricing problem.

Measurement-based Feedback Control of Linear Quantum Stochastic Systems with Quadratic-Exponential Criteria

This paper is concerned with a risk-sensitive optimal control problem for a feedback connection of a quantum plant with a measurement-based classical controller. The plant is a multimode open quantum harmonic oscillator driven by a multichannel quantum Wiener process, and the controller is a linear time invariant system governed by a stochastic differential equation. The control objective is to stabilize the closed-loop system and minimize the infinite-horizon asymptotic growth rate of a quadratic-exponential functional (QEF) which penalizes the plant variables and the controller output. We combine a frequency-domain representation of the QEF growth rate, obtained recently, with variational techniques and establish first-order necessary conditions of optimality for the state-space matrices of the controller.

Risk-Sensitive Mean Field Games via the Stochastic Maximum Principle

In this paper, we consider risk-sensitive mean field games via the risk-sensitive maximum principle. The problem is analyzed through two sequential steps: (i) risk-sensitive optimal control for a fixed probability measure, and (ii) the associated fixed-point problem. For step (i), we use the risk-sensitive maximum principle to obtain the optimal solution, which is characterized in terms of the associated forward–backward stochastic differential equation (FBSDE). In step (ii), we solve for the probability law induced by the state process with the optimal control in step (i). In particular, we show the existence of the fixed point of the probability law of the state process determined by step (i) via Schauder’s fixed-point theorem. After analyzing steps (i) and (ii), we prove that the set of N optimal distributed controls obtained from steps (i) and (ii) constitutes an approximate Nash equilibrium or $$\epsilon $$-Nash equilibrium for the N player risk-sensitive game, where $$\epsilon \rightarrow 0$$ as $$N \rightarrow \infty $$ at the rate of $$O(\frac{1}{N^{1/(n+4)}})$$. Finally, we discuss extensions to heterogeneous (non-symmetric) risk-sensitive mean field games.

The maximum principles for partially observed risk-sensitive optimal controls of Markov regime-switching jump-diffusion system

ABSTRACTThis paper studies partially observed risk-sensitive optimal control problems with correlated noises between the system and the observation. It is assumed that the state process is governed by a continuous-time Markov regime-switching jump-diffusion process and the cost functional is of an exponential-of-integral type. By virtue of a classical spike variational approach, we obtain two general maximum principles for the aforementioned problems. Moreover, under certain convexity assumptions on both the control domain and the Hamiltonian, we give a sufficient condition for the optimality. For illustration, a linear-quadratic risk-sensitive control problem is proposed and solved using the main results. As a natural deduction, a fully observed risk-sensitive maximum principle is also obtained and applied to study a risk-sensitive portfolio optimization problem. Closed-form expressions for both the optimal portfolio and the corresponding optimal cost functional are obtained.

A risk-sensitive maximum principle for a Markov regime-switching jump-diffusion system and applications

In this paper, we derive a general stochastic maximum principle for a risk-sensitive type optimal control problem of Markov regime-switching jump-diffusion model. The results are obtained via a logarithmic transformation and the relationship between adjoint variables and the value function. We apply the results to study both a linear-quadratic optimal control problem and a risk-sensitive benchmarked asset management problem for Markov regime-switching models. In the latter case, the optimal control is of feedback form and is given in terms of solutions to a Markov regime-switching Riccatti equation and an ordinary Markov regime-switching differential equation.

Linear Quadratic Risk-Sensitive and Robust Mean Field Games

This paper considers two classes of large population stochastic differential games connected to optimal and robust decentralized control of large-scale multiagent systems. The first problem ( P1 ) is one where each agent minimizes an exponentiated cost function, capturing risk-sensitive behavior, whereas in the second problem ( P2 ) each agent minimizes a worst-case risk-neutral cost function, where the “worst case” stems from the presence of an adversary entering each agent’s dynamics characterized by a stochastic differential equation. In both problems, the individual agents are coupled through the mean field term included in each agent’s cost function, which captures the average or mass behavior of the agents. We solve both P1 and P2 via mean field game theory. Specifically, we first solve a generic risk-sensitive optimal control problem and a generic stochastic zero-sum differential game, where the corresponding optimal controllers are applied by each agent to construct the mean field systems of P1 and P2 . We then characterize an approximated mass behavior effect on an individual agent via a fixed-point analysis of the mean field system. For each problem, P1 and P2 , we show that the approximated mass behavior is in fact the best estimate of the actual mass behavior in various senses as the population size, $N$ , goes to infinity. Moreover, we show that for finite $N$ , there exist $\epsilon$ - Nash equilibria for both P1 and P2 , where the corresponding individual Nash strategies are decentralized in terms of local state information and the approximated mass behavior. We also show that $\epsilon$ can be taken to be arbitrarily small when $N$ is sufficiently large. We show that the $\epsilon$ - Nash equilibria of P1 and P2 are partially equivalent in the sense that the individual Nash strategies share identical control laws, but the approximated mass behaviors for P1 and P2 are different, since in P2 , the mass behavior is also affected by the associated worst-case disturbance. Finally, we prove that the Nash equilibria for P1 and P2 both feature robustness, and as the parameter characterizing this robustness becomes infinite, the two Nash equilibria become identical and equivalent to that of the risk-neutral case, as in the one-agent risk-sensitive and robust control theory.

Pontryagin’s Risk-Sensitive Stochastic Maximum Principle for Backward Stochastic Differential Equations with Application

This paper studies the risk-sensitive optimal control problem for a backward stochastic system. More precisely, we set up a necessary stochastic maximum principle for a risk-sensitive optimal control of this kind of equations. The control domain is assumed to be convex and the generator coefficient of such system is allowed to be depend on the control variable. As a preliminary step, we study the risk-neutral problem for which an optimal solution exists. This is an extension of initial control system to this type of problem, where the set of admissible controls is convex. An example to carried out to illustrate our main result of risk-sensitive control problem under linear stochastic dynamics with exponential quadratic cost function.

Risk-sensitive control of Markov jump linear systems: Caveats and difficulties

In this technical note, we revisit the risk-sensitive optimal control problem for Markov jump linear systems (MJLSs). We first demonstrate the inherent difficulty in solving the risk-sensitive optimal control problem even if the system is linear and the cost function is quadratic. This is due to the nonlinear nature of the coupled set of Hamilton-Jacobi-Bellman (HJB) equations, stemming from the presence of the jump process. It thus follows that the standard quadratic form of the value function with a set of coupled Riccati differential equations cannot be a candidate solution to the coupled HJB equations. We subsequently show that there is no equivalence relationship between the problems of risk-sensitive control and H∞ control of MJLSs, which are shown to be equivalent in the absence of any jumps. Finally, we show that there does not exist a large deviation limit as well as a risk-neutral limit of the risk-sensitive optimal control problem due to the presence of a nonlinear coupling term in the HJB equations.

Maximum principle for partially observed risk-sensitive optimal control problems of mean-field type

This paper deals with a class of partially observed risk-sensitive optimal control. By virtue of Girsanov׳s theorem and the standard spike variational technique, we establish the maximum principle for the partially observed control problems. Moreover, the sufficient condition is also obtained for the special case by using some concavity conditions. As an application, a linear–quadratic system is presented to demonstrate our results.

Pessimistic Optimal Choice for Risk-Averse Agents: The Continuous-Time Limit

We extend Hansen and Sargent's (Discounted linear exponential quadratic gaussian control, 1994, IEEE Trans Autom Control 40:968---971 1995, 2013) analysis of dynamic optimization with risk-averse agents in two directions. Firstly, following Whittle (Risk-sensitive optimal control, 1990), we show that the optimal risk-averse policy is identified via a pessimistic choice mechanism and described by simple recursive formulae. Secondly, we investigate the continuous-time limit and show that sufficient conditions for the existence of optimal solutions coincide with those which apply under risk-neutrality. Our analysis is conducted both under perfect and imperfect state observation. As an illustrative example, we analyze the optimal production policy of an entrepreneur running a monopolistic firm which faces a demand schedule subject to stochastic shocks, showing that risk-aversion induces her to act more aggressively.

Risk-Sensitive Ergodic Control of Continuous Time Markov Processes With Denumerable State Space

In this article, we study risk-sensitive control problem with controlled continuous time Markov chain state dynamics. Using multiplicative dynamic programming principle along with the atomic structure of the state dynamics, we prove the existence and a characterization of optimal risk-sensitive control under geometric ergodicity of the state dynamics along with a smallness condition on the running cost.

Synthesizing Anticipatory Haptic Assistance Considering Human Behavior Uncertainty

Intuitive and effective physical assistance is an essential requirement for robots sharing their workspace with humans. Application domains reach from manufacturing and service robotics via rehabilitation and mobility aids to education and training. In this context, assistance based on human behavior anticipation has shown superior performance in terms of human effort minimization. However, when a robot's expectations mismatch a human intentions, undesired interaction forces appear incurring safety risks and discomfort. Human behavior prediction is, therefore, a crucial issue: It enables effective anticipation but potentially produces disagreements when prediction errors occur. In this paper, we present a novel control scheme for anticipatory haptic assistance where robot behavior adapts to prediction uncertainty. Following a data-driven stochastic modeling approach, robot assistance is synthesized solving a risk-sensitive optimal control problem, where the cost function and plant dynamics are affected by model uncertainty. The proposed approach is objectively and subjectively evaluated in an experiment with human users. Results indicate that our method outperforms other assistive control approaches in terms of perceived helpfulness and human effort minimization.

Pessimistic Optimal Choice for Risk-Averse Agents

We propose a general framework for the analysis of dynamic optimization with risk-averse agents, extending Whittle's (Whittle, 1990) formulation of risk-sensitive optimal control problems to accommodate time-discounting. We show how, within a Markovian set-up, optimal risk-averse behavior is identified via a pessimistic choice mechanism and described by simple recursive formulae. We apply this methodology to two distinct problems formulated respectively in discrete- and continuous-time. In the former, we extend Svensson's (Svensson, 1997) analysis of optimal monetary policy, showing that with a pessimistic central bank the inflation forecast is not longer an explicit intermediate target, the monetary authorities do not expect the inflation rate to mean revert to its target level and apply a more aggressive Taylor rule, while the inflation rate is less volatile. In the latter, we investigate the optimal production policy of a monopolistic entrepreneur which faces a demand schedule subject to stochastic shocks, once again showing that risk-aversion induces her to act more aggressively.