Hamilton Jacobi Bellman Partial Differential Equation Research Articles

ε-Nash Equilibrium of Pursuer–Evader–Defender Missile Navigation Dynamic Games

This research is dedicated to developing a min–max robust control strategy for a dynamic game involving pursuers, evaders, and defenders in a multiple-missile scenario. The approach employs neural dynamic programming, utilizing multiple continuous differential neural networks (DNNs). The competitive controller devised addresses the robust optimization of a joint cost function that relies on the trajectories of the pursuer–evader–defender system, accommodating an uncertain mathematical model while adhering to control restrictions. The dynamic programming min–max formulation facilitates robust control by accounting for bounded modeling uncertainties and external disturbances for each game component. The value function of the Hamilton–Jacobi–Bellman (HJB) equation is approximated by a DNN, enabling the estimation of the closed-loop formulation for the joint dynamic game with state restrictions. The controller’s design is grounded in estimating the state trajectory under the worst possible uncertainties and perturbations, providing a robustness factor through the robust neural controller. The learning law class for the time-varying weights in the DNN is generated by studying the HJB partial differential equation for the missile motion for each player in the dynamic game. The controller incorporates the solution of the obtained learning laws and a time-varying Riccati equation, offering an online solution to the control implementation. A recurrent algorithm, based on the Kiefer–Wolfowitz method, adjusts the initial conditions for the weights to satisfy the final condition of the given cost function for the dynamic game. A numerical example is presented to validate the proposed robust control methodology, confirming the optimization solution based on the DNN approximation for Bellman’s value function.

Lagrangian relaxation for continuous‐time optimal control of coupled hydrothermal power systems including storage capacity and a cascade of hydropower systems with time delays

AbstractThis work considers a short‐term, continuous time setting characterized by a coupled power supply system controlled exclusively by a single provider and comprising a cascade of hydropower systems (dams), fossil fuel power stations, and a storage capacity modeled by a single large battery. Cascaded hydropower generators introduce time‐delay effects in the state dynamics, which are modeled with differential equations, making it impossible to use classical dynamic programming. We address this issue by introducing a novel Lagrangian relaxation technique over continuous‐time constraints, constructing a nearly optimal policy efficiently. This approach yields a convex, nonsmooth optimization dual problem to recover the optimal Lagrangian multipliers, which is numerically solved using a limited memory bundle method. At each step of the dual optimization, we need to solve an optimization subproblem. Given the current values of the Lagrangian multipliers, the time delays are no longer active, and we can solve a corresponding nonlinear Hamilton–Jacobi–Bellman (HJB) Partial Differential Equation (PDE) for the optimization subproblem. The HJB PDE solver provides both the current value of the dual function and its subgradient, and is trivially parallelizable over the state space for each time step. To handle the infinite‐dimensional nature of the Lagrange multipliers, we design an adaptive refinement strategy to control the duality gap. Furthermore, we use a penalization technique for the constructed admissible primal solution to smooth the controls while achieving a sufficiently small duality gap. Numerical results based on the Uruguayan power system demonstrate the efficiency of the proposed mathematical models and numerical approach.

Differential neural network robust constrained controller using approximate dynamic programming

This study focuses on developing a continuous differential neural network (DNN) approximating min–max robust control by applying dynamic neural programming. The suggested controller is applied to a class of nonlinear perturbed systems providing satisfactory dynamics for a given cost function depending on both the trajectories of the perturbed system and the designed constrained control actions. The min–max formulation for dynamic programming offers reliable control for restricted modeling uncertainties and perturbations. The suggested design considers control norm restrictions in the optimization problem. DNN’s approximation of the worst (with respect to the admissible class of perturbations and uncertainties) value function of the Hamilton–Jacobi–Bellman (HJB) equation enables the estimation of the closed-loop formulation of the controller. The robust version of the HJB partial differential equation is studied to create the learning law class for the time-varying weights in the DNN. The controller employs a time-varying Lyapunov-like differential equation and the solution of the corresponding learning laws. A recurrent algorithm based on the Kiefer–Wolfowitz technique can be used by modifying the weights’ initial conditions to fulfill the specified cost function’s end requirements. A numerical example tests the robust control proposed in this study, validating the robust optimal solution based on the DNN approximation for Bellman’s value function.

A spectral iterative algorithm for solving constrained optimal control problems with nonquadratic functional

The numerical approximation of the solution to Hamilton-Jacobi-Bellman (HJB) partial differential equations (PDEs) in constrained optimal control problems (COCPs) with nonquadratic functional is studied in this investigation. Discretizing both of the space and time of HJB PDE by the Legendre collocation method, a nonlinear system of algebraic equations is obtained to find the expansion coefficients of the approximate solution. However, solving the system of nonlinear algebraic equations is generally relatively difficult and time-consuming. To improve the implementation of the method, we combine the Legendre collocation method with the policy iteration (PI) algorithm. This process leads to a sequence of linear systems of algebraic equations to find the expansion coefficients. The convergence results for the proposed method are provided. Moreover, it is shown that the PI algorithm for a class of constrained optimal control problems with odd nonquadratic functionals is mathematically equivalent to the quasi-Newton's iteration. Finally, the efficiency and accuracy of the method are demonstrated by solving a number of examples.

Computing geodesic paths encoding a curvature prior for curvilinear structure tracking

In this paper, we introduce an efficient method for computing curves minimizing a variant of the Euler-Mumford elastica energy, with fixed endpoints and tangents at these endpoints, where the bending energy is enhanced with a user-defined and data-driven scalar-valued term referred to as the curvature prior. In order to guarantee that the globally optimal curve is extracted, the proposed method involves the numerical computation of the viscosity solution to a specific static Hamilton-Jacobi-Bellman (HJB) partial differential equation (PDE). For that purpose, we derive the explicit Hamiltonian associated with this variant model equipped with a curvature prior, discretize the resulting HJB PDE using an adaptive finite difference scheme, and solve it in a single pass using a generalized fast-marching method. In addition, we also present a practical method for estimating the curvature prior values from image data, designed for the task of accurately tracking curvilinear structure centerlines. Numerical experiments on synthetic and real-image data illustrate the advantages of the considered variant of the elastica model with a prior curvature enhancement in complex scenarios where challenging geometric structures appear.

Min–Max Dynamic Programming Control for Systems with Uncertain Mathematical Models via Differential Neural Network Bellman’s Function Approximation

This research focuses on designing a min–max robust control based on a neural dynamic programming approach using a class of continuous differential neural networks (DNNs). The proposed controller solves the robust optimization of a proposed cost function that depends on the trajectories of a system with an uncertain mathematical model satisfying a class of non-linear perturbed systems. The dynamic programming min–max formulation enables robust control concerning bounded modelling uncertainties and disturbances. The Hamilton–Jacobi–Bellman (HJB) equation’s value function, approximated by a DNN, permits to estimate the closed-loop formulation of the controller. The controller design is based on an estimated state trajectory with the worst possible uncertainties/perturbations that provide the degree of robustness using the proposed controller. The class of learning laws for the time-varying weights in the DNN is produced by studying the HJB partial differential equation. The controller uses the solution of the obtained learning laws and a time-varying Riccati equation. A recurrent algorithm based on the Kiefer–Wolfowitz method leads to adjusting the initial conditions for the weights to satisfy the final condition of the given cost function. The robust control suggested in this work is evaluated using a numerical example confirming the optimizing solution based on the DNN approximate for Bellman’s value function.

Optimal Control on Finite-Time Consensus of the Leader-Following Stochastic Multiagent System With Heuristic Method

In this paper, the optimization of the finite-time consensus of a multiagent system is studied when the leader is disturbed by white noise. Different from the consensus problem itself, this paper is devoted to optimizing the performance of the consensus and the constructive method and martingale method are used to guarantee the effectiveness of the optimization. Another contribution is that the problem of the optimal control with the piecewise constant controller gains under a stochastic environment is solved, and the reformed dynamic principle and the reformed Hamilton–Jacobi–Bellman (HJB) partial differential equations (PDEs) are proposed by using the splitting method and Feynman–Kac formula. Furthermore, since the multiagent system often runs on a high-dimensional space, a heuristic method is proposed based on the derivation of the reformed HJB PDEs and the scalar of the computational complexity of the PDEs is equivalent to that of a system of ordinary differential equations with the same dimensions when the multiagent system can be described as a piecewise linear system. Finally, the heuristic method is compared with usual filtering methods for solving engineering problems by a numerical example and it is found that the heuristic method achieves higher precision in finite time.

PRICING AMERICAN OPTIONS WITH THE RUNGE–KUTTA–LEGENDRE FINITE DIFFERENCE SCHEME

This paper presents the Runge–Kutta–Legendre (RKL) finite difference scheme, allowing for an additional shift in its polynomial representation. A short presentation of the stability region, comparatively to the Runge–Kutta–Chebyshev scheme follows. We then explore the problem of pricing American options with the RKL scheme under the one factor Black–Scholes and the two factor Heston stochastic volatility models, as well as the pricing of butterfly spread and digital options under the uncertain volatility model, where a Hamilton–Jacobi–Bellman partial differential equation needs to be solved. We explore the order of convergence in these problems, as well as the option greeks stability, compared to the literature and popular schemes such as Crank–Nicolson, with Rannacher time-stepping.

Synthesis of control Lyapunov functions and stabilizing feedback strategies using exit‐time optimal control Part I: Theory

AbstractThis work studies the problem of constructing control Lyapunov functions (CLFs) and feedback stabilization strategies for deterministic nonlinear control systems described by ordinary differential equations. Many numerical methods for solving the Hamilton–Jacobi–Bellman partial differential equations specifying CLFs typically require dense state space discretizations and consequently suffer from the curse of dimensionality. A relevant direction of attenuating the curse of dimensionality concerns reducing the computation of the values of CLFs and associated feedbacks at any selected states to finite‐dimensional nonlinear programming problems. We propose to use exit‐time optimal control for that purpose. This article is the first part of a two‐part work. First, we state an exit‐time optimal control problem with respect to a sublevel set of an appropriate local CLF and establish that, under a number of reasonable conditions, the concatenation of the corresponding value function and the local CLF is a global CLF in the whole domain of asymptotic null‐controllability. We also investigate the formulated optimal control problem. A modification of these constructions for the case when one does not find a suitable local CLF is provided as well. Our developments serve as a theoretical basis for a curse‐of‐dimensionality‐free approach to feedback stabilization, that is presented in the second part Yegorov et al. (2021) of this work together with supporting numerical simulation results.

Actor-Critic Method for High Dimensional Static Hamilton--Jacobi--Bellman Partial Differential Equations based on Neural Networks

We propose a novel numerical method for high dimensional Hamilton--Jacobi--Bellman (HJB) type elliptic partial differential equations (PDEs). The HJB PDEs, reformulated as optimal control problems, are tackled by the actor-critic framework inspired by reinforcement learning, based on neural network parametrization of the value and control functions. Within the actor-critic framework, we employ a policy gradient approach to improve the control, while for the value function, we derive a variance reduced least-squares temporal difference method using stochastic calculus. To numerically discretize the stochastic control problem, we employ an adaptive step size scheme to improve the accuracy near the domain boundary. Numerical examples up to $20$ spatial dimensions including the linear quadratic regulators, the stochastic Van der Pol oscillators, the diffusive Eikonal equations, and fully nonlinear elliptic PDEs derived from a regulator problem are presented to validate the effectiveness of our proposed method.

Backward Reachability Analysis for Nonlinear Dynamical Systems via Pseudospectral Method

In this paper, we propose a new approach to solving the backward reachability problem for nonlinear dynamical systems. Previously, this class of problems has been studied within frameworks of optimal control and zero-sum differential games, where a backward reachable set can be expressed as the zero sublevel set of the value function that can be characterized by solving the Hamilton-Jacobi-Bellman (HJB) partial differential equation (PDE). In many cases, however, a high computational cost is incurred in numerically solving such HJB PDEs due to the curse of dimensionality. We use the pseudospectral method to convert the associated optimal control problem into nonlinear programs (NLPs). We then show that the zero sublevel set obtained by the optimal cost of the NLP is the corresponding backward reachable set. Note that our approach does not require solving complex HJB PDEs. Therefore, it can reduce computation time and handle high-dimensional dynamical systems, compared with the numerical software package developed by I. Mitchell, which has been used widely in the literature to obtain backward reachable sets by solving HJB equations. We provide several examples to validate the effectiveness of the proposed approach.

A new modified polynomial-based optimal control design approach

The main concern of this article is addressing a new modified approach to design a nonlinear optimal controller. The modification focuses on proposing a new approximate solution for the Hamilton–Jacobi–Bellman nonlinear partial differential equation. The introduced solution works based on the state-dependent power series expansion presentation of the involved functions in the Hamilton–Jacobi–Bellman partial differential equation. Applying this technique results in releasing a set of free state-dependent functions in the controller structure that can be adjusted to fulfill some special control missions in addition to the optimization objectives. They are formed based on the specific formulation of the candidate Lyapunov function. The proposed approach is exemplified for an intricate biological system, immunogenic tumor-immune cell interaction in the human body, to clarify the mechanism of designing the controller and adjusting the arrays of the free matrices. The closed-loop system by presented optimal state feedback controller meets the predefined optimization objectives without getting feedback from a hard-measurable state. It is achieved by adjusting the aforementioned released functions such that an optimal output feedback controller is obtained. To have some insights into the performance of the system and the effectiveness of the controller, the positiveness of the system’s states is proved and checked numerically by applying the differential transformation method to the system’s differential equations. Finally, to highlight the abilities of the proposed approach from different aspects, some simulations are carried out.

Optimal stopping time, consumption, labour, and portfolio decision for a pension scheme

In this work we solve in a closed form the problem of an agent who wants to optimise the inter-temporal recursive utility of both his consumption and leisure by choosing: (1) the optimal inter-temporal consumption, (2) the optimal inter-temporal labour supply, (3) the optimal share of wealth to invest in a risky asset, and (4) the optimal retirement age. The wage of the agent is assumed to be stochastic and correlated with the risky asset on the financial market. The problem is split into two sub-problems: the optimal consumption, labour, and portfolio problem is solved first, and then the optimal stopping time is approached. We compute the solution through both the so-called martingale approach and the solution of the Hamilton–Jacobi–Bellman partial differential equation. In the numerical simulations we compare two cases, with and without the opportunity, for the agent, to work after retirement, at a lower wage rate.

Nested Sparse Successive Galerkin Approximation for Nonlinear Optimal Control Problems

Solving the Hamilton-Jacobi-Bellman (HJB) equation for nonlinear optimal control problems usually suffers from the so-called curse of dimensionality. In this letter, a nested sparse successive Galerkin method is presented for HJB equations, and the computational cost only grows polynomially with the dimension. Based on successive approximation techniques, the nonlinear HJB partial differential equation (PDE) is transformed into a sequence of linear PDEs. Then the nested sparse grid methods are employed to solve the resultant linear PDEs. The designed method is sparse in two aspects. Firstly, the solution of the linear PDE is constructed based on sparse combinations of nested basis functions. Secondly, the multi-dimensional integrals in the Galerkin method are efficiently calculated using nested sparse grid quadrature rules. Once the successive approximation process is finished, the optimal controller can be analytically given based on the sparse basis functions and coefficients. Numerical results also demonstrate the accuracy and efficiency of the designed nested sparse successive Galerkin method.

HEDGE-FUND MANAGEMENT WITH LIQUIDITY CONSTRAINT

We propose a model for a manager of a hedge fund with a liquidity constraint, where he is seeking to optimize his utility of wealth, with one and multiple period horizons. By using stochastic control techniques, we state the corresponding multi-dimensional Hamilton–Jacobi–Bellman partial differential equation and we use a robust numerical approximation to obtain its unique viscosity solution. We examine the effects of the liquidity constraint on managerial trading decisions and optimal allocation, finding that the manager behaves in a less risky manner. We also calculate the cost of being at sub-optimal positions as the difference in the certainty equivalent payoff for the manager. Moreover, we compare the values of a benchmark hedge fund with another one having a risky asset with a higher rate of return but less liquidity, finding that higher rate of return with a liquidity constraint does not always lead to greater return.

Systemic Risk and Stochastic Games with Delay

We propose a model of inter-bank lending and borrowing which takes into account clearing debt obligations. The evolution of log-monetary reserves of banks is described by coupled diffusions driven by controls with delay in their drifts. Banks are minimizing their finite-horizon objective functions which take into account a quadratic cost for lending or borrowing and a linear incentive to borrow if the reserve is low or lend if the reserve is high relative to the average capitalization of the system. As such, our problem is a finite-player linear–quadratic stochastic differential game with delay. An open-loop Nash equilibrium is obtained using a system of fully coupled forward and advanced-backward stochastic differential equations. We then describe how the delay affects liquidity and systemic risk characterized by a large number of defaults. We also derive a closed-loop Nash equilibrium using a Hamilton–Jacobi–Bellman partial differential equation approach.

Credit portfolio selection with decaying contagion intensities

AbstractWe develop a fixed‐income portfolio framework capturing the exponential decay of contagious intensities between successive default events. We show that the value function of the control problem is the classical solution to a recursive system of second‐order uniformly parabolic Hamilton–Jacobi–Bellman partial differential equations. We analyze the interplay between risk premia, decay of default intensities, and their volatilities. Our comparative statics analysis finds that the investor chooses to go long only if he is capturing enough risk premia. If the default intensities deteriorate faster, the investor increases the size of his position if he goes short, or reduces the size of his position if he goes long.

Economically optimal strategies for medium-term recovery after a major nuclear reactor accident

Abstract The dynamic process of ground contamination after a major nuclear accident is modelled, and the system is then extended to include the transient equations describing the three broad countermeasures: food bans, remediation and population movement (relocation and repopulation). Countermeasures are assumed to be applicable once the deposition period has ended and surface contamination measurements have stabilised. A value function is constructed to account for the major economic factors, including allowance for the detrimental effect on human health of radiation exposure. The principle of optimality is then applied by requiring the value function to satisfy the Hamilton–Jacobi–Bellman partial differential equation, yielding an economically optimal combination of the countermeasures at any given moment of time within the recovery period. A classification into Broad Strategies is made in order to explore the similarities in structure of optimal strategies for wide ranges of economic parameter values. Population relocation forms no part of any optimal strategy in the Base Case (or Case I) as parameters are varied over a wide range. Strategies incorporating relocation have a low probability of being optimal even in the low-probability sensitivity studies of Case II, where relocation is imposed immediately the accident happens, and Case III, where the Base Case assumption is reversed of lower economic productivity awaiting those moving from the original to the new area. It is concluded that relocation is almost certain to be a less than optimal response after a great many large-scale nuclear accidents.

Numerical methods for dynamic Bertrand oligopoly and American options under regime switching

Bertrand oligopolies are competitive markets in which a small number of firms producing similar goods use price as their strategic variable. In particular, each firm wants to determine the optimal price that maximizes its expected discounted lifetime profit. The oligopoly problem can be modeled as nonzero-sum games which can be formulated as systems of Hamilton–Jacobi–Bellman (HJB) partial differential equations (PDEs). In this paper, we propose fully implicit, positive coefficient finite difference schemes that converge to the viscosity solution for the HJB PDE from dynamic Bertrand monopoly and the two-dimensional HJB system from dynamic Bertrand duopoly. Furthermore, we develop fast multigrid methods for solving these systems of discrete nonlinear HJB PDEs. The new multigrid methods are general and can be applied to other systems of HJB and HJB-Isaacs PDEs arising from American options under regime switching and American options with unequal lending/borrowing rates and stock borrowing fees under regime switching, respectively. We provide a theoretical analysis for the smoother, restriction and interpolation operators of the multigrid methods. Finally, we demonstrate the effectiveness of our method by numerical examples from the dynamic Bertrand problem and pricing American options under regime switching.

PORTFOLIO OPTIMIZATION AND STOCHASTIC VOLATILITY ASYMPTOTICS

We study the Merton portfolio optimization problem in the presence of stochastic volatility using asymptotic approximations when the volatility process is characterized by its timescales of fluctuation. This approach is tractable because it treats the incomplete markets problem as a perturbation around the complete market constant volatility problem for the value function, which is well understood. When volatility is fast mean‐reverting, this is a singular perturbation problem for a nonlinear Hamilton–Jacobi–Bellman partial differential equation, while when volatility is slowly varying, it is a regular perturbation. These analyses can be combined for multifactor multiscale stochastic volatility models. The asymptotics shares remarkable similarities with the linear option pricing problem, which follows from some new properties of the Merton risk tolerance function. We give examples in the family of mixture of power utilities and also use our asymptotic analysis to suggest a “practical” strategy that does not require tracking the fast‐moving volatility. In this paper, we present formal derivations of asymptotic approximations, and we provide a convergence proof in the case of power utility and single‐factor stochastic volatility. We assess our approximation in a particular case where there is an explicit solution.