Risk-sensitive Stochastic Control Research Articles

Risk-Sensitive Control of Finite State Machines on an Infinite Horizon I

In this paper we consider robust and risk-sensitive control of discrete time finite state systems on an infinite horizon. The solution of the state feedback robust control problem is characterized in terms of the value of an average cost dynamic game. The risk-sensitive stochastic optimal control problem is solved using the policy iteration algorithm, and the optimal rate is expressed in terms of the value of a stochastic dynamic game with average cost per unit time criterion. By taking a small noise limit, a deterministic dynamic game which is closely related to the robust control problem is obtained.

Connections between stochastic control and dynamic games

We consider duality relations between risk-sensitive stochastic control problems and dynamic games. They are derived from two basic duality results, the first involving free energy and relative entropy and resulting from a Legendre-type transformation, the second involving power functions. Our approach allows us to treat, in essentially the same way, continuous- and discrete-time problems, with complete and partial state observation, and leads to a very natural formal justification of the structure of the cost functional of the dual. It also allows us to obtain the solution of a stochastic game problem by solving a risk-sensitive control problem.

H/sup ∞/-0ptimal Control and Related Minimax Design Problems: A Dynamic Game Approach

One of the major concentrated activities of the past decade in control theory has been the development of the so-called control theory, which addresses the issue of worst-case controller design for linear plants subject to unknown disturbances and plant uncertainties. Among the different time-domain approaches to this class of worst-case design problems, the one that uses the framework of dynamic, differential game theory stands out to be the most natural. This is so because the original H-infinity control problem (in its equivalent time-domain formulation) is in fact a minimax optimization problem, and hence a zero-sum game, where the controller can be viewed as the minimizing player and disturbance as the maximizing player. Using this framework, the authors present in this book a complete theory that encompasses continuous-time as well as discrete-time systems, finite as well as infinite horizons, and several different measurement schemes, including closed loop perfect state, delayed perfect state, samples state, closed-loop imperfect state, delayed imperfect state and sampled imperfect state information patterns. They also discuss extensions of the linear theory to nonlinear systems, and derivation of the lower dimensional controller for systems with regularly and singularly perturbed dynamics. This is the second edition of a 1991 book with the same title, which, besides featuring a more streamlined presentation of the results included in the first edition, and at places under more refined conditions, also contains substantial new material, reflecting new developments in the field since 1991. Among these are the nonlinear theory; connections between H-infinity-optimal control and risk sensitive stochastic control problems; H-infinity filtering for linear and nonlinear systems; and robustness considerations in the presence of regular and singular perturbations. Also included are a rather detailed description of the relationship between frequency-and time-domain approaches to robust controller design, and a complete set of results on the existence of value and characterization of optimal policies in finite- and infinite-horizon LQ differential games. The authors believe that the theory is now at a stage where it can easily be incorporated into a second-level graduate course in a control curriculum, that would follow a basic course in linear control theory covering LQ and LQG designs. The framework adopted in this book makes such an ambitious plan possible. For the most part, the only prerequisite for the book is a basic knowledge of linear control theory. No background in differential games, or game theory in general, is required, as the requisite concepts and results have been developed in the book at the appropriate level. The book is written in such a way that makes it possible to follow the theory for the continuous- and discrete-time systems independently (and also in parallel).

NONLINEAR DISCRETE-TIME RISK-SENSITIVE OPTIMAL CONTROL

SUMMARY This paper is devoted to the study of the connections among risk-sensitive stochastic optimal control, dynamic game optimal control, risk-neutral stochastic optimal control and deterministic optimal control in a nonlinear, discrete-time context with complete state information. The analysis worked out sheds light on the profound links among these control strategies, which remain hidden in the linear context. In particular, it is shown that, under suitable parameterizations, risk-sensitive control can be regarded as a control methodology which combines features of both stochastic risk-neutral control and deterministic dynamic game control.

Uniqueness for Viscosity Solutions of Nonstationary Hamilton–Jacobi–Bellman Equations under Some a Priori Conditions (with Applications)

This paper extends the uniqueness results for viscosity solutions of nonstationary Hamilton--Jacobi--Bellman equations. The conditions for uniqueness which are obtained can involve a trade-off between the growth of the solution and the growth of the Hamiltonian. In particular, the result is valid for solutions which grow quadratically in the space variable and are associated with Hamiltonians which also grow quadratically. This particular class arises in the robust control limit of risk-sensitive stochastic control problems.

Risk-sensitive control and dynamic games for partially observed discrete-time nonlinear systems

Solves a finite-horizon partially observed risk-sensitive stochastic optimal control problem for discrete-time nonlinear systems and obtains small noise and small risk limits. The small noise limit is interpreted as a deterministic partially observed dynamic game, and new insights into the optimal solution of such game problems are obtained. Both the risk-sensitive stochastic control problem and the deterministic dynamic game problem are solved using information states, dynamic programming, and associated separated policies. A certainty equivalence principle is also discussed. The authors' results have implications for the nonlinear robust stabilization problem. The small risk limit is a standard partially observed risk-neutral stochastic optimal control problem. >