Isaacs Equation Research Articles

Representation formulas for solutions of Isaacs integro-PDE

We prove suband super-optimality inequalities of dynamic programming for viscosity solutions of Isaacs integro-PDE associated with two-player, zero-sum stochastic differential game driven by a Levy type noise. This implies that the lower and upper value functions of the game satisfy the dynamic programming principle and they are the unique viscosity solutions of the lower and upper Isaacs integro-PDE. We show how to regularize viscosity suband super-solutions of Isaacs equations to smooth suband super-solutions of slightly perturbed equations.

Robust utility maximization for a diffusion market model with misspecified coefficients

The paper studies the robust maximization of utility from terminal wealth in a diffusion financial market model. The underlying model consists of a tradable risky asset whose price is described by a diffusion process with misspecified trend and volatility coefficients, and a non-tradable asset with a known parameter. The robust functional is defined in terms of a utility function. An explicit characterization of the solution is given via the solution of the Hamilton–Jacobi–Bellman–Isaacs (HJBI) equation.

Value function of differential games without Isaacs conditions. An approach with nonanticipative mixed strategies

In the present paper we investigate the problem of the existence of a value for differential games without Isaacs condition. For this we introduce a suitable concept of mixed strategies along a partition of the time interval, which are associated with classical nonanticipative strategies (with delay). Imposing on the underlying controls for both players a conditional independence property, we obtain the existence of the value in mixed strategies as the limit of the lower as well as of the upper value functions along a sequence of partitions which mesh tends to zero. Moreover, we characterize this value in mixed strategies as the unique viscosity solution of the corresponding Hamilton–Jacobi–Isaacs equation.

Stochastic Homogenization for Some Nonlinear Integro-Differential Equations

In this note we prove the stochastic homogenization for a large class of fully nonlinear elliptic integro-differential equations in stationary ergodic random environments. Such equations include, but are not limited to, Bellman equations and the Isaacs equations for the control and differential games of some pure jump processes in a random, rapidly varying environment. The translation invariant and non-random effective equation is identified, and the almost everywhere in ω, uniform in x convergence of the family solutions of the original equations is obtained. Even in the linear case of the equations contained herein the results appear to be new.

Online solution of nonquadratic two‐player zero‐sum games arising in the H ∞ control of constrained input systems

SUMMARYIn this paper, we present an online learning algorithm to find the solution to the H ∞ control problem of continuous‐time systems with input constraints. A suitable nonquadratic functional is utilized to encode the input constraints into the H ∞ control problem, and the related H ∞ control problem is formulated as a two‐player zero‐sum game with a nonquadratic performance. Then, a policy iteration algorithm on an actor–critic–disturbance structure is developed to solve the Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game. That is, three NN approximators, namely, actor, critic, and disturbance, are tuned online and simultaneously for approximating the HJI solution. The value of the actor and disturbance policies is approximated continuously by the critic NN, and then on the basis of this value estimate, the actor and disturbance NNs are updated in real time to improve their policies. The disturbance tries to make the worst possible disturbance, whereas the actor tries to make the best control input. A persistence of excitation condition is shown to guarantee convergence to the optimal saddle point solution. Stability of the closed‐loop system is also guaranteed. A simulation on a nonlinear benchmark problem is performed to validate the effectiveness of the proposed approach. Copyright © 2012 John Wiley & Sons, Ltd.

On the homogenization of some non-coercive Hamilton--Jacobi--Isaacs equations

We study the homogenization of Hamilton-Jacobi equations with oscillating initial data and non-coercive Hamiltonian, mostly of the Bellman-Isaacs form arising in optimal control and differential games. We describe classes of equations for which pointwise homogenization fails for some data. We prove locally uniform homogenization for various Hamiltonians with some partial coercivity and some related restrictions on the oscillating variables, mostly motivated by the applications to differential games, in particular of pursuit-evasion type. The effective initial data are computed under some assumptions of asymptotic controllability of the underlying control system with two competing players.

On Lyapunov Inequalities and Subsolutions for Efficient Importance Sampling

In this article we explain some connections between Lyapunov methods and subsolutions of an associated Isaacs equation for the design of efficient importance sampling schemes. As we shall see, subsolutions can be derived by taking an appropriate limit of an associated Lyapunov inequality. They have been recently proposed in several works of Dupuis, Wang, and others and applied to address several important problems in rare-event simulation. Lyapunov inequalities have been used for testing the efficiency of state-dependent importance sampling schemes in heavy-tailed or discrete settings in a variety of works by Blanchet, Glynn, and others. While subsolutions provide an analytic criterion for the construction of efficient samplers, Lyapunov inequalities are useful for finding more precise information, in the form of bounds, for the behavior of the coefficient of variation of the associated importance sampling estimator in the prelimit. In addition, Lyapunov inequalities provide insight into the various mollification procedures that are often required in constructing associated subsolutions. Our aim is to demonstrate that applying Lyapunov inequalities for verification of efficiency can help both guide the selection of various mollification parameters and sharpen the information on the efficiency gain induced by the sampler.

Zero-Sum Risk-Sensitive Stochastic Differential Games

We study zero-sum risk-sensitive stochastic differential games on the infinite horizon with discounted and ergodic payoff criteria. Under certain assumptions, we establish the existence of values and saddle-point equilibria. We obtain our results by studying the corresponding Hamilton–Jacobi–Isaacs equations. Finally, we show that the value of the ergodic payoff criterion is a constant multiple of the maximal eigenvalue of the generators of the associated nonlinear semigroups.

Robust consumption-investment problems with random market coefficients

We consider consumption-investment problems in a financial market with general random coefficients where the market price of risk process is unknown. The investor tries to maximize his expected utility under the worst-case parameter configuration. To solve robust consumption-investment problems, we make use of stochastic Bellman–Isaac equations. These equations can be explicitly solved for power, exponential and logarithmic utility. This enables us to characterize a robust optimal consumption-investment strategy and a worst-case market price of risk process in terms of the solution of a backward stochastic differential equation.

Computationally efficient simultaneous policy update algorithm for nonlinear H∞ state feedback control with Galerkin's method

SUMMARYThe main bottleneck for the application of H∞ control theory on practical nonlinear systems is the need to solve the Hamilton–Jacobi–Isaacs (HJI) equation. The HJI equation is a nonlinear partial differential equation (PDE) that has proven to be impossible to solve analytically, even the approximate solution is still difficult to obtain. In this paper, we propose a simultaneous policy update algorithm (SPUA), in which the nonlinear HJI equation is solved by iteratively solving a sequence of Lyapunov function equations that are linear PDEs. By constructing a fixed point equation, the convergence of the SPUA is established rigorously by proving that it is essentially a Newton's iteration method for finding the fixed point. Subsequently, a computationally efficient SPUA (CESPUA) based on Galerkin's method, is developed to solve Lyapunov function equations in each iterative step of SPUA. The CESPUA is simple for implementation because only one iterative loop is included. Through the simulation studies on three examples, the results demonstrate that the proposed CESPUA is valid and efficient. Copyright © 2012 John Wiley & Sons, Ltd.

ℋ ∞ filtering for discrete-time affine non-linear descriptor systems

In this study, the authors consider the ℋ∞-filtering problem for discrete-time affine non-linear descriptor systems. Two types of filters are discussed; namely, (i) singular, (ii) normal and sufficient conditions for the solvability of the problem in terms of disrete-time Hamilton–Jacobi–Isaacs equations (DHJIEs) are presented. The results are also specialised to linear systems, in which case the DHJIEs reduce to a system of linear-matrix-inequalities (LMIs). Examples are also presented to illustrate the results.

The minmax principle and $W^{2,p}$ regularity for solutions of the simplest Isaacs equations

The minmax principle and $W^{2,p}$ regularity for solutions of the simplest Isaacs equations

Multigrid methods for two‐player zero‐sum stochastic games

SUMMARYWe present a fast numerical algorithm for large scale zero‐sum stochastic games with perfect information, which combines policy iteration and algebraic multigrid methods. This algorithm can be applied either to a true finite state space zero‐sum two‐player game or to the discretization of an Isaacs equation. We present numerical tests on discretizations of Isaacs equations or variational inequalities. We also present a full multilevel policy iteration, similar to full multigrid algorithm (FMG), which allows one to improve substantially the computation time for solving some variational inequalities. Copyright © 2012 John Wiley & Sons, Ltd.

Stochastic optimal control problems under G‐expectation

SUMMARYPeng first introduced the notion of G‐Brownian motion and G‐expectation and established the stochastic calculus with respect to G‐Brownian motion in 2006. In this paper, we investigate the stochastic optimal control problems under G‐expectation and obtain dynamic programming principle. The value function is proved to be a viscosity solution of a fully nonlinear second‐order partial differential equation. A particular case of this equation is the well‐known Hamilton–Jacobi–Bellman–Isaacs equation. Copyright © 2011 John Wiley & Sons, Ltd.

Analysis of a Splitting Estimator for Rare Event Probabilities in Jackson Networks

We consider a standard splitting algorithm for the rare-event simulation of overflow probabilities in any subset of stations in a Jackson network at level n, starting at a fixed initial position. It was shown in [8] that a subsolution to the Isaacs equation guarantees that a subexponential number of function evaluations (in n) suffices to estimate such overflow probabilities within a given relative accuracy. Our analysis here shows that in fact O(n2βV + 1) function evaluations suffice to achieve a given relative precision, where βV is the number of bottleneck stations in the subset of stations under consideration in the network. This is the first rigorous analysis that favorably compares splitting against directly computing the overflow probability of interest, which can be evaluated by solving a linear system of equations with O(nd) variables.

Stochastic differential games with reflection and related obstacle problems for Isaacs equations

In this paper we first investigate zero-sum two-player stochastic differential games with reflection, with the help of theory of Reflected Backward Stochastic Differential Equations (RBSDEs). We will establish the dynamic programming principle for the upper and the lower value functions of this kind of stochastic differential games with reflection in a straightforward way. Then the upper and the lower value functions are proved to be the unique viscosity solutions to the associated upper and the lower Hamilton-Jacobi-Bellman-Isaacs equations with obstacles, respectively. The method differs significantly from those used for control problems with reflection, with new techniques developed of interest on its own. Further, we also prove a new estimate for RBSDEs being sharper than that in the paper of El Karoui, Kapoudjian, Pardoux, Peng and Quenez (1997), which turns out to be very useful because it allows us to estimate the Lp-distance of the solutions of two different RBSDEs by the p-th power of the distance of the initial values of the driving forward equations. We also show that the unique viscosity solution to the approximating Isaacs equation constructed by the penalization method converges to the viscosity solution of the Isaacs equation with obstacle.

Stochastic minimax optimal time-delay state feedback control of uncertain quasi-integrable Hamiltonian systems

A stochastic minimax optimal time-delay state feedback control strategy for uncertain quasi-integrable Hamiltonian systems is proposed. First, a stochastic optimal state feedback control problem of uncertain quasi-integrable Hamiltonian system with time delay in feedback control subjected to Gaussian white noise is formulated. Then, the time-delayed state feedback control forces are approximated by the control forces without time delay and the original problem is converted into a stochastic optimal state feedback control problem of an uncertain system without time delay. After that, by following a procedure based on the stochastic averaging method and stochastic differential game, the worst-case disturbances and the optimal controls are obtained from solving a Hamiltonian–Jacobi–Isaacs (HJI) equation. As an example, the stochastic minimax optimal state feedback control of a Duffing oscillator with parametric disturbances and time-delayed feedback control is worked out in detail to illustrate the procedure and effectiveness of the proposed control strategy.

Static Solutions of the Hamilton-Jacobi-BellmanEquation for Circular Orbit Transfers

A LYAPUNOV control law was proposed by Chang et al. [1] for transfer between Keplerian orbits. For circular orbits, a Lyapunov function was chosen based on the energy E and the angularmomentumL, and a control lawwas derived in order tomake the derivative of the Lyapunov function negative-definite. However, the optimality of the control law was not analyzed. In this Note, the optimality of these transfers is shown by deriving a cost function for which these transfers are optimal. In particular, the link between the Lyapunov approach and the static solution of the Hamilton–Jacobi– Bellman equation (HJBE) for the two-body orbit transfer is shown. Moreover, it is shown that for long times of flight, constantpropulsive-acceleration transfers can be designed in feedback form with almost the same fuel cost in comparison with fuel-optimal transfers. Bonnard et al. [2] considered the geometric structure of timeoptimal control for spacecraft transfer in elliptic Keplerian orbits. A time-optimal control problem was proposed for constant specific impulse engines and the controllability of the system was analyzed. The methodology to compute the optimal control was based on the traditional indirect method, which was solved numerically by means of a shooting method. Freeman and Kokotovic [3] introduced the concept of a robust control Lyapunov function (rclf) and showed that every rclf satisfies the steady-state Hamilton–Jacobi–Isaacs equation associated with a meaningful game. Indeed, having a control Lyapunov function, it can be used for determination a stabilizing feedback control that is optimal in some sense. This technique is known as inverse optimality method. As an example, Bharadwaj et al. [4] used this method to design globally stabilizing attitude control laws. Primbs et al. [5] investigated the relation between optimal control problems and control Lyapunov functions from a receding-horizon perspective. In that work, it was shown that a broad class of Lyapunov controllers admit natural extensions to recedinghorizon formulations, which are locally optimal. However, these ideas were of a general character and applied only to a simple nonlinear system. In this Note, these ideas are applied specifically for mediumto low-thrust spacecraft orbital transfer in a two-body setting. In particular, a closed-form solution of an infinite-horizon HJBE for two-body, circular-orbit transfers is derived and is compared with fuel-optimal finite time transfers. The solution of the HJBE provides the optimal control at every point in phase space in feedback form. However, the nonlinearity of this partial differential equation (PDE)makes it extremely difficult to solve analytically. For linear systems subject to quadratic cost functions, it is possible to obtain an analytic form for the optimal cost [6], but for general nonlinear systems that are of practical interest, this is not the case in general. The numerical approach to solve this PDE is still challenging, as the solutions are in general not smooth and differentiable at every point. However, the great drawback of dynamic programming is, as Bellman himself calls it, “the curse of dimensionality.” Even recording the solution to a moderately sized problem involves an enormous amount of storage. If only an optimal path from a known initial point is desired, it is wasteful and tedious to find a whole field of extremals [7]. After a review of the main result obtained by Chang et al. [1], the static form of the HJBE is described and the optimality of the Lyapunov control law for a regulized fuel-optimal cost function is shown. The application of this control law is illustrated with a threedimensional transfer from a low Earth orbit (LEO) to the geostationary Earth orbit (GEO). The results are shown to be applicable to both variable specific impulse (VSI) and engines with constantpropulsive-acceleration capability. The examples illustrate the relation between the (infinite-horizon) transfers and actual finite time, fuel-optimal solutions. Finally, the application of thismethod to provide good initial guesses of the costates (relevant to the indirect formulation and solution of the optimal control problem) is explored.

Corrections to ‘ filtering for singularly perturbed nonlinear systems’

SUMMARYIn a recently published paper, the authors considered the ‐filtering problem for singularly perturbed (two time‐scale) nonlinear systems. Two types of filters were discussed; namely, decomposition and aggregate, and sufficient conditions for the solvability of the problem in terms of the Hamilton–Jacobi–Isaac's equations were presented. In this note, we provide a modification to the HJIEs that make them more formal and amenable to computations. Copyright © 2011 John Wiley & Sons, Ltd.

Online solution of nonlinear two‐player zero‐sum games using synchronous policy iteration

SUMMARYThe two‐player zero‐sum (ZS) game problem provides the solution to the bounded L2‐gain problem and so is important for robust control. However, its solution depends on solving a design Hamilton–Jacobi–Isaacs (HJI) equation, which is generally intractable for nonlinear systems. In this paper, we present an online adaptive learning algorithm based on policy iteration to solve the continuous‐time two‐player ZS game with infinite horizon cost for nonlinear systems with known dynamics. That is, the algorithm learns online in real time an approximate local solution to the game HJI equation. This method finds, in real time, suitable approximations of the optimal value and the saddle point feedback control policy and disturbance policy, while also guaranteeing closed‐loop stability. The adaptive algorithm is implemented as an actor/critic/disturbance structure that involves simultaneous continuous‐time adaptation of critic, actor, and disturbance neural networks. We call this online gaming algorithm ‘synchronous’ ZS game policy iteration. A persistence of excitation condition is shown to guarantee convergence of the critic to the actual optimal value function. Novel tuning algorithms are given for critic, actor, and disturbance networks. The convergence to the optimal saddle point solution is proven, and stability of the system is also guaranteed. Simulation examples show the effectiveness of the new algorithm in solving the HJI equation online for a linear system and a complex nonlinear system. Copyright © 2011 John Wiley & Sons, Ltd.