Hamilton-Jacobi-Bellman Equation Research Articles

Singular HJB equations with applications to KPZ on the real line

This paper is devoted to studying Hamilton-Jacobi-Bellman equations with distribution-valued coefficients, which are not well-defined in the classical sense and are understood by using the paracontrolled distribution method introduced in (Gubinelli et al. in Forum Math Pi 3(6):1, 2015). By a new characterization of weighted Hölder spaces and Zvonkin’s transformation we prove some new a priori estimates, and therefore establish the global well-posedness for singular HJB equations. As applications, we obtain global well-posedness in polynomial weighted Hölder spaces for KPZ type equations on the real line, as well as modified KPZ equations for which the Cole–Hopf transformation is not applicable.

Optimal attitude consensus for multi rigid bodies network considered on the lie group SO(3) with connectivity preserving property

This paper proposes distributed optimal attitude consensus control for single-integrator multi rigid bodies with undirected network evolving on Special Orthogonal Group SO(3) while simultaneously guarantees the connectivity preservation property for agents using descent gradient algorithm. Since by Use of the Euclidean distance on Lie group as a measure of the energy of the state does not define and preserve the topology of SO(3); besides, solving the Hamilton–Jacobi–Bellman equation in optimal control problems shows difficulty implementing Euclidean distances and limits the results for SO(3) configuration state spaces. As a result, in this paper, the generic distance on SO(3) associated to the natural Riemannian metric structure is used. Using this structure, Firstly, a distributed potential function based consensus control law is applied to the system exploiting Riemannian distance on SO(3). Then, for relaxing some restrictive conditions, finite-time convergence, and increasing the speed of convergence the kinematic optimal control on SO(3) is considered. Referring to the proposed potential function designed in the previous section, an inverse optimal attitude consensus control problem is considered, which is solved by an inverse optimal control method. Finally, the designed method validates via two simulation examples.

Dual ML-ADHDP method for heterogeneous discrete-time nonlinear multi-agent systems with unknown dynamics and time delay

This paper develops a new dual ML-ADHDP method to solve the optimal consensus problem (OCP) of a class of heterogeneous discrete-time nonlinear multi-agent systems (MASs) with unknown dynamics and time delay. A hierarchical and distributed control strategy is used to transform the original problem into nonlinear model reference adaptive control (MRAC) problems and an OCP of virtual linear MASs. For the nonlinear MRAC problems, a new multi-layer action-dependent heuristic dynamic programming (ML-ADHDP) method is developed to overcome the unknown dynamics and neural network estimation errors, which has higher control accuracy. In order to solve the OCP of virtual linear MASs and improve the convergence speed, a new multi-layer performance index is proposed. Then the ML-ADHDP method is used to solve the coupled Hamiltonian–Jacobi–Bellman equation and obtain the optimal virtual control. Theoretical analysis proves that the original MASs can achieve Nash equilibrium, and simulation results show that the developed dual ML-ADHDP method ensures better convergence speed and higher control accuracy of original MASs.

Analytical and numerical solutions to ergodic control problems arising in environmental management

Environmental management optimizing a long‐run objective is an ergodic control problem whose resolution can be achieved by solving an associated non‐local Hamilton–Jacobi–Bellman (HJB) equation having an effective Hamiltonian. Focusing on sediment storage management as a modern engineering problem, we formulate, analyze, and compute a new ergodic control problem under discrete observations: a simple but non‐trivial mathematical problem. We give optimality and comparison results of the corresponding HJB equation having unique non‐smoothness and discontinuity. To numerically compute HJB equations, we propose a new fast‐sweep method resorting to neither pseudo‐time integration nor vanishing discount. The optimal policy and the effective Hamiltonian are then computed simultaneously. Convergence rate of numerical solutions is computationally analyzed. An advanced robust control counterpart where the dynamics involve uncertainties is also numerically considered.

Viscosity solutions to first order path-dependent Hamilton–Jacobi–Bellman equations in Hilbert space

In this article, a notion of viscosity solutions is introduced for first order path-dependent Hamilton–Jacobi–Bellman (PHJB) equations associated with optimal control problems for path-dependent evolution equations in Hilbert space. We identify the value functional of optimal control problems as unique viscosity solution to the associated PHJB equations without the assumption (A.4) on page 231 of Li and Yong (1995). We also show that our notion of viscosity solutions is consistent with the corresponding notion of classical solutions, and satisfies a stability property.

Data-driven optimal tracking control for SMA actuated systems with prescribed performance via reinforcement learning

This article addresses the data-driven performance-prescribed continuous-time optimal tracking control problem for the shape memory alloy (SMA) actuated systems with completely unknown model knowledge. Firstly, the error constraint problem is transformed into the unconstrained error tracking problem by the prescribed performance control (PPC) method. Then, the optimal tracking control problem (OTCP) is pre-processed by establishing an unconstrained augmented system. Furthermore, the Hamilton–Jacobi–Bellman equation (HJBE) of the OTCP is solved iteratively by utilizing reinforcement learning (RL) without the SMA actuator model information requirement. The value function and execution strategy of the RL are approximated by two neural networks, acting as actor and critic, respectively, and the actor–critic based RL is implemented using the least-squares method. In addition, the Lyapunov method ensures the stability of the closed-loop system actuated by SMA, as well as the user-specified error constraints concerning the error convergence rate, overshoot, and tracking accuracy. Finally, the experimental results and comparisons illustrate the validity of the proposed method.

RETRACTED ARTICLE: A Single Solution to the HJB Equation for Optimal Regulation Control of Euler–Lagrange Systems

Applying Lyapunov theory, the asymptotic stability of the closed-loop system is assured for fully-actuated non-linear Euler–Lagrange (EL) systems. The proposed finite-time quadratic criterion is minimized by solving a Hamilton–Jacobi–Bellman (HJB) equation. The proposed analytic solution for the HJB equation exploits the properties of the EL systems, arriving at an optimal control law in a feedback configuration. Gravity, friction, and external disturbances compensations are considered. A single solution is presented that is based on the proposed Lagrange and value functions. The value function consists of the kinetic energy and a quadratic term of positions, and the Lagrange function includes the kinetic energy and a quadratic term of the EL system input. It is assumed that the kinetic energy is a quadratic form, i.e., the positions of the center of mass do not depend explicitly on time. The proposed solution does not require solving a state-dependent Riccati differential equation or solving a state-dependent ordinary differential equation. Instead, a single analytical solution is presented that converges asymptotically to the optimal value as the states approach zero. The optimal regulation control problem is solved and is illustrated by a mechanical example.

Valuation of European Options Under an Uncertain Market Price of Volatility Risk

We propose a model to quantify the effect of parameter uncertainty on the option price in the Heston model. More precisely, we present a Hamilton–Jacobi–Bellman framework which allows us to evaluate best and worst-case scenarios under an uncertain market price of volatility risk. For the numerical approximation, the Hamilton–Jacobi–Bellman equation is reformulated to enable the solution with a finite element method. A case study with butterfly options exhibits how the dependence of Delta on the magnitude of the uncertainty is nonlinear and highly varied across the parameter regime.

Second order local minimal-time mean field games

The paper considers a forward–backward system of parabolic PDEs arising in a Mean Field Game (MFG) model where every agent controls the drift of a trajectory subject to Brownian diffusion, trying to escape a given bounded domain $$\Omega $$ in minimal expected time. Agents are constrained by a bound on the drift depending on the density of other agents at their location. Existence for a finite time horizon T is proven via a fixed point argument, but the natural setting for this problem is in infinite time horizon. Estimates are needed to treat the limit $$T\rightarrow \infty $$ , and the asymptotic behavior of the solution obtained in this way is also studied. This passes through classical parabolic arguments and specific computations for MFGs. Both the Fokker–Planck equation on the density of agents and the Hamilton–Jacobi–Bellman equation on the value function display Dirichlet boundary conditions as a consequence of the fact that agents stop as soon as they reach $$\partial \Omega $$ . The initial datum for the density is given, and the long-time limit of the value function is characterized as the solution of a stationary problem.

Reinforcement learning‐based optimized backstepping control of nonlinear strict feedback system with unknown control gain function

AbstractThe article develops an optimized control via learning from the design idea of optimized backstepping (OB) technique for the nonlinear strict feedback systems containing the unknown dynamic and control gain functions. OB technique requires to deal with the actual and virtual controls of backstepping as the optimized solutions of corresponding subsystems so that the entire backstepping control is optimized. In the work, for achieving the optimization control, reinforcement learning (RL) of critic‐actor structure is constructed in every backstepping step on the basis of the neural network approximation of the Hamilton–Jacobi–Bellman equation's solution. Since the unknown nonlinear control gain function is considered, the complexity of control algorithm is greatly increased. However, the proposed RL is with the simple training laws, it can greatly alleviate the algorithm complexity for the optimized control. Finally, the feasibility of the method is demonstrated by both theory and simulation.

Solution for optimal control of general nonlinear systems using generating functions

AbstractIn this paper, we present an analytical local solution for optimal control problems applicable to a wide class of general nonlinear systems. We first introduce the optimal control problem for a general nonlinear system and formulate the associated Hamilton–Jacobi–Bellman equation. Starting from the necessary conditions for optimality represented by the Hamiltonian system of the optimal control problem, we solve the Hamilton–Jacobi equation for the associated generating functions. The generating functions are obtained in the form of series expansions by using canonical transformation theory and the semi‐tensor product of matrices. The coefficients of the generating functions satisfy simply the algebraic Riccati equation and linear algebraic equations for the second‐degree homogeneous component and higher degree components, respectively, from which an algorithm and data structure can be derived. This enables us to obtain analytical optimal feedback controllers and the corresponding Lyapunov function. Through theoretical analysis, it is established that the proposed methodology guarantees stability of the closed‐loop system. This approach is not affected by the complexity of the performance index and state equations, and most importantly, it does not require any initial admissible control law. A simulation example is presented to illustrate the effectiveness and applicability of the proposed approach.

Continuous-Time Portfolio Optimization for Absolute Return Funds

This paper investigates a continuous-time portfolio optimization problem with the following features: (i) a no-short selling constraint; (ii) a leverage constraint, that is, an upper limit for the sum of portfolio weights; and (iii) a performance criterion based on the lower mean square error between the investor’s wealth and a predetermined target wealth level. Since the target level is defined by a deterministic function independent of market indices, it corresponds to the criterion of absolute return funds. The model is formulated using the stochastic control framework with explicit boundary conditions. The corresponding Hamilton–Jacobi–Bellman equation is solved numerically using the kernel-based collocation method. However, a straightforward implementation does not offer a stable and acceptable investment strategy; thus, some techniques to address this shortcoming are proposed. By applying the proposed methodology, two numerical results are obtained: one uses artificial data, and the other uses empirical data from Japanese organizations. There are two implications from the first result: how to stabilize the numerical solution, and a technique to circumvent the plummeting achievement rate close to the terminal time. The second result implies that leverage is inevitable to achieve the target level in the setting discussed in this paper.

Continuous Differentiability of the Value Function of Semilinear Parabolic Infinite Time Horizon Optimal Control Problems on L2(Ω)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$L^2(\\Omega )$$\\end{document} Under Control Constraints

An abstract framework guaranteeing the local continuous differentiability of the value function associated with optimal stabilization problems subject to abstract semilinear parabolic equations subject to a norm constraint on the controls is established. It guarantees that the value function satisfies the associated Hamilton–Jacobi–Bellman equation in the classical sense. The applicability of the developed framework is demonstrated for specific semilinear parabolic equations.

Incentives, lockdown, and testing: from Thucydides' analysis to the COVID-19 pandemic.

In this work, we provide a general mathematical formalism to study the optimal control of an epidemic, such as the COVID-19 pandemic, via incentives to lockdown and testing. In particular, we model the interplay between the government and the population as a principal–agent problem with moral hazard, à la Cvitanić et al. (Finance Stoch 22(1):1–37, 2018), while an epidemic is spreading according to dynamics given by compartmental stochastic SIS or SIR models, as proposed respectively by Gray et al. (SIAM J Appl Math 71(3):876–902, 2011) and Tornatore et al. (Phys A Stat Mech Appl 354(15):111–126, 2005). More precisely, to limit the spread of a virus, the population can decrease the transmission rate of the disease by reducing interactions between individuals. However, this effort—which cannot be perfectly monitored by the government—comes at social and monetary cost for the population. To mitigate this cost, and thus encourage the lockdown of the population, the government can put in place an incentive policy, in the form of a tax or subsidy. In addition, the government may also implement a testing policy in order to know more precisely the spread of the epidemic within the country, and to isolate infected individuals. In terms of technical results, we demonstrate the optimal form of the tax, indexed on the proportion of infected individuals, as well as the optimal effort of the population, namely the transmission rate chosen in response to this tax. The government’s optimisation problems then boils down to solving an Hamilton–Jacobi–Bellman equation. Numerical results confirm that if a tax policy is implemented, the population is encouraged to significantly reduce its interactions. If the government also adjusts its testing policy, less effort is required on the population side, individuals can interact almost as usual, and the epidemic is largely contained by the targeted isolation of positively-tested individuals.

Optimal renewable resource harvesting model using price and biomass stochastic variations: a utility based approach

In this paper, we provide a general framework for analyzing the optimal harvest of a renewable resource (i.e. fish, shrimp) assuming that the price and biomass evolve stochastically and harvesters have a constant relative risk aversion. In order to take into account the impact of a sudden change in the environment linked to the ecosystem, we assume that the biomass are governed by a stochastic differential equation of the Gilpin–Ayala type, with regime change in the parameters of the drift and variance. Under the above assumptions, we find the optimal effort to be deployed by the collector (fishery for example) in order to maximize the expected utility of its profit function. To do this, we give the proof of the existence and uniqueness of the value function, which is derived from the Hamilton–Jacobi–Bellman equations associated with this problem, by resorting to a definition of the viscosity solution.

Risk-sensitive discounted cost criterion for continuous-time Markov decision processes on a general state space

In this paper, we consider risk-sensitive discounted control problem for continuous-time jump Markov processes taking values in general state space. The transition rates of underlying continuous-time jump Markov processes and the cost rates are allowed to be unbounded. Under certain Lyapunov condition, we establish the existence and uniqueness of the solution to the Hamilton–Jacobi–Bellman equation. Also, we prove the existence of optimal risk-sensitive control in the class of Markov control and completely characterized the optimal control.

Optimal control of multiple Markov-switching stochastic systems with numerical applications

In this article the authors set up an optimal control framework for a hybrid stochastic system with dual or multiple Markov switching diffusion processes, while Markov chains governing these switching diffusions are not identical as assumed in the existing literature. For this purpose, the authors first combine two or more separate Markov chains into one compound Markov chain within continuous time context and give out concrete procedures about how to estimate corresponding generator matrix of the newly combined Markov chain via a modified EM algorithm, then the HJB equations for the optimal control problem with synthetic Markov switching diffusion are stated. As an application and illustration, the authors lay out an adapted numerical solution scheme for the problem about optimal consumption–investment decisions under multiple cyclical or switching markets. Relevant economic analysis and interpretations are provided as results.

Hybrid optimal impulse control

This paper is concerned with an optimal impulse control problem under a hybrid diffusion (or, regime switching) model, where the state of the system consists of a number of diffusions coupled by a continuous-time finite-state Markov chain. The objective is to minimize the expected discounted cost from exerting the impulse control in the infinite horizon. Based on the dynamic programming principle (DPP), the value function of the hybrid optimal impulse control problem is shown to be the unique viscosity solution to the associated Hamilton–Jacobi–Bellman (HJB) equation, which is in the form of a coupled system of variational inequalities. Moreover, a verification theorem as the sufficient condition for optimality of a solution is also established. The optimal impulse control, indicates when and how it is optimal to intervene, is described by the obstacle part of the HJB equation. Finally, the general theoretical results are applied to an optimal cash management problem. The value function in closed-form and an explicit optimal policy are obtained, which exhibit clearly the effect of regime switching on the agent’s decision.

Optimal dividends for regulated insurers with a nonlinear penalty

This study investigates an optimal dividend problem for an insurer with random time regulation and a nonlinear penalty from the payment of dividends. The surplus process of the insurer is a jump–diffusion model and the goal of the insurer is to maximise the expected accumulated discount dividend payment until ruin occurs or regulatory time arrives. The introduction of the external random regulatory time leads that the HJB equation associated with this control problem is a time in-homogeneous parabolic variational inequality. There is no common explicit solution to this variational inequality. We derived explicit solutions for optimal policies when both the regulatory time and the claims follow an exponential distribution. For general cases, this paper provides a Markov chain approximation method for the value function and optimal policies and presents an example to illustrate the algorithm when the random regulatory time follows the Gompertz–Makeham distribution.

An envelope method for solving continuous-time stochastic models with occasionally binding constraints

I introduce a finite-difference solution method based on the envelope condition in continuous-time stochastic dynamic programming problems. The envelope method is easier to code and, in the presence of occasionally binding constraints, faster and more stable than popular methods based on the Hamilton–Jacobi–Bellman equation. As an illustration, I solve a stochastic growth model with irreversible investment.