Hamilton-Jacobi-Bellman Equation Research Articles

Hamilton‐Jacobi‐Bellman equation based on fractional random impulses system

AbstractIn this talk, we consider the optimal control problem for fractional order random impulses differential equations (), and there is no corresponding Hamilton‐Jacobi‐Bellman (HJB) equation proposed for fractional order differential equations in existing literature. The lack of semi group properties in fractional order makes the original dynamic programming methods unable to directly handle fractional order problems. We have dealt with this problem by combining the properties of fractional order integrals. In solving this problem, we found that the order of the original system in HJB equation is at least 1. When the order is less than 1, our approach to fractional order ideas provides a possibility to solve the problem.

Optimal synchronization for multi-agent systems: A performance-dependent switching topology

This paper proposes an optimal output synchronization control method for heterogeneous multi-agent systems (HMASs) under a performance-dependent switching topology and DoS attacks. First, local and global switched performance index functions (SPIF) and SPIF-dependent topology switching law are proposed, respectively, thus, the control performance and topology quality can be quantitatively expressed. Second, an adaptive dynamic programming (ADP) algorithm with mode switching is proposed, aimed at dealing with the difficult Hamilton–Jacobi–Bellman equation, as well as the analytical complexity caused by the switching dynamics. The convergence of the switched ADP algorithm is proven to ensure its correct implementation. Then, for different topologies, multi-mode Actor–Critic neural networks (NNs) are built for each agent to calculate optimized control policies and SPIF, respectively. Furthermore, an NN-based state compensation mechanism is designed to expand the applicability of the designed switched ADP algorithm when the leader’s output transmission is unreliable. Finally, the results of numerical examples confirm that the proposed method is feasible.

Bellman Neural Networks for the Class of Optimal Control Problems With Integral Quadratic Cost

This work introduces Bellman Neural Networks (BeNNs) and employs them to learn the optimal control actions for the class of optimal control problems (OCPs) with integral quadratic cost. BeNNs represent a particular family of Physics-Informed Neural Networks (PINNs) specifically designed and trained to tackle OCPs via applying the Bellman Principle of Optimality (BPO). The BPO provides necessary and sufficient optimality conditions, which result in a nonlinear partial differential equation known as the Hamilton-Jacobi-Bellman (HJB) equation. BeNNs learn the optimal control actions from the unknown solution of the arising HJB equation (i.e., the value function), where the unknown solution is modeled using a Neural Network. Additionally, the paper shows how to estimate the upper bounds on the generalization error of BeNNs while learning the solutions for the OCP class under consideration. The generalization error estimate is provided in terms of the choice and number of the training points as well as the training error. Numerical studies show that BeNNs can be successfully applied to learn the feedback control actions for the class of optimal control problems considered and, after the training is completed, deployed to control the system in a closed-loop fashion. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Impact Statement</i> —The proposed research improves our understanding of how to solve optimal control problems with closed-loop solutions and has potentially a countless number of applications in several different areas. The study is at the intersection between optimal control theory and artificial intelligence connected with mathematical tools for functional interpolation. This advances the ability to implement a higher level of autonomy in decision-making for practical applications with a beneficial impact on our society.

Optimized tracking control using reinforcement learning and backstepping technique for canonical nonlinear unknown dynamic system

AbstractThe work addresses the optimized tracking control problem by combining both reinforcement learning (RL) and backstepping technique for the canonical nonlinear unknown dynamic system. Since such dynamic system contains multiple state variables with differential relation, the backstepping technique is considered by making a virtual control sequence in accordance with Lyapunov functions. In the last backstepping step, the optimized actual control is derived by performing the RL under identifier‐critic‐actor structure, where RL is to overcome the difficulty coming from solving Hamilton‐Jacobi‐Bellman (HJB) equation. Different from the traditional RL optimizing methods that find the RL updating laws from the square of the HJB equation's approximation, this optimized control is to find the RL training laws from the negative gradient of a simple positive definite function, which is equivalent to the HJB equation. The result shows that this optimized control can obviously alleviate the algorithm complexity. Meanwhile, it can remove the requirement of known dynamic as well. Finally, theory and simulation indicate the feasibility of this optimized control.

Nonlinear Optimal Control for Stochastic Dynamical Systems

This paper presents a comprehensive framework addressing optimal nonlinear analysis and feedback control synthesis for nonlinear stochastic dynamical systems. The focus lies on establishing connections between stochastic Lyapunov theory and stochastic Hamilton–Jacobi–Bellman theory within a unified perspective. We demonstrate that the closed-loop nonlinear system’s asymptotic stability in probability is ensured through a Lyapunov function, identified as the solution to the steady-state form of the stochastic Hamilton–Jacobi–Bellman equation. This dual assurance guarantees both stochastic stability and optimality. Additionally, optimal feedback controllers for affine nonlinear systems are developed using an inverse optimality framework tailored to the stochastic stabilization problem. Furthermore, the paper derives stability margins for optimal and inverse optimal stochastic feedback regulators. Gain, sector, and disk margin guarantees are established for nonlinear stochastic dynamical systems controlled by nonlinear optimal and inverse optimal Hamilton–Jacobi–Bellman controllers.

Optimal portfolio strategy of wealth process: a Lévy process model-based method

This paper is concerned with the optimal portfolio problem for a company that can invest in two risky assets, where a novel Lévy-process-driven model is constructed to describe the dynamics of the wealth process by using a constant elasticity of variance model and a jump-diffusion process. A delicately designed value function is proposed under the mean–variance criterion to reflect the optimal portfolio for the stochastic volatility model. By using the verification theorem, the desired optimal portfolio strategy is proposed by the solution to certain Hamilton–Jacobi–Bellman equations. Furthermore, the corresponding expressions are achieved by using the stochastic analysis theory. Finally, a numerical simulation example is provided to verify the effectiveness of the proposed optimal portfolio strategy.

Scheduling in the High-Uncertainty Heavy Traffic Regime

We propose a model uncertainty approach to heavy traffic asymptotics that allows for a high level of uncertainty. That is, the uncertainty classes of underlying distributions accommodate disturbances that are of order 1 at the usual diffusion scale as opposed to asymptotically vanishing disturbances studied previously in relation to heavy traffic. A main advantage of the approach is that the invariance principle underlying diffusion limits makes it possible to define uncertainty classes in terms of the first two moments only. The model we consider is a single-server queue with multiple job types. The problem is formulated as a zero sum stochastic game played between the system controller, who determines scheduling and attempts to minimize an expected linear holding cost, and an adversary, who dynamically controls the service time distributions of arriving jobs and attempts to maximize the cost. The heavy traffic asymptotics of the game are fully solved. It is shown that an asymptotically optimal policy for the system controller is to prioritize according to an index rule, and for the adversary, it is to select distributions based on the system’s current workload. The workload-to-distribution feedback mapping is determined by a Hamilton–Jacobi–Bellman equation, which also characterizes the game’s limit value. Unlike in the vast majority of results in the heavy traffic theory and as a direct consequence of the diffusive size disturbances, the limiting dynamics under asymptotically optimal play are captured by a stochastic differential equation where both the drift and the diffusion coefficients may be discontinuous. Funding: R. Atar is supported by the Israeli Science Foundation [Grant 1035/20].

Optimal mean-variance investment and reinsurance strategies with a general Lévy process risk model

This paper is concerned with the optimal time-consistent investment and reinsurance strategies for mean-variance insurers with a general Lévy Process model. Expressly, the insurers are allowed to purchase proportional reinsurance and invest in a financial market, where the surplus of the insurers is assumed to follow a Cramér–Lundberg model and the financial market consists of one risk-free asset and one risky asset whose price process is driven by a general Lévy process. Through the verification theorem, the closed-form expressions of the optimal strategies under the mean-variance criterion are derived by a complex partial integral differential Hamilton–Jacobi–Bellman equations. Finally, numerical simulations are provided to verify the effectiveness of the proposed optimal strategies and some economic interpretations are drawn.

A Viscous Ergodic Problem with Unbounded and Measurable Ingredients, Part 1: HJB Equation

A Viscous Ergodic Problem with Unbounded and Measurable Ingredients, Part 1: HJB Equation

Optimal Control of Probability on a Target Set for Continuous-Time Markov Chains

In this paper, a stochastic optimal control problem is considered for a continuous-time Markov chain taking values in a denumerable state space over a fixed finite horizon. The optimality criterion is the probability that the process remains in a target set before and at a certain time. The optimal value is a superadditive capacity of target sets. Under some minor assumptions for the controlled Markov process, we establish the dynamic programming principle, based on which we prove that the value function is a classical solution of the Hamilton-Jacobi-Bellman equation on a discrete lattice space. We then prove that there exists an optimal deterministic Markov control under the compactness assumption of control domain. We further prove that the value function is the unique solution of the HJB equation. We also consider the case starting from the outside of target set and give the corresponding results. Finally, we apply our results to two examples.

Hierarchical Sliding-Mode Surface-Based Adaptive Actor-Critic Optimal Control for Switched Nonlinear Systems With Unknown Perturbation.

This article studies the hierarchical sliding-mode surface (HSMS)-based adaptive optimal control problem for a class of switched continuous-time (CT) nonlinear systems with unknown perturbation under an actor-critic (AC) neural networks (NNs) architecture. First, a novel perturbation observer with a nested parameter adaptive law is designed to estimate the unknown perturbation. Then, by constructing an especial cost function related to HSMS, the original control issue is further converted into the problem of finding a series of optimal control policies. The solution to the HJB equation is identified by the HSMS-based AC NNs, where the actor and critic updating laws are developed to implement the reinforcement learning (RL) strategy simultaneously. The critic update law is designed via the gradient descent approach and the principle of standardization, such that the persistence of excitation (PE) condition is no longer needed. Based on the Lyapunov stability theory, all the signals of the closed-loop switched nonlinear systems are strictly proved to be bounded in the sense of uniformly ultimate boundedness (UUB). Finally, the simulation results are presented to verify the validity of the proposed adaptive optimal control scheme.

Numerical approximation based on deep convolutional neural network for high‐dimensional fully nonlinear merged PDEs and 2BSDEs

This paper proposes two efficient approximation methods to solve high‐dimensional fully nonlinear partial differential equations (NPDEs) and second‐order backward stochastic differential equations (2BSDEs), where such high‐dimensional fully NPDEs are extremely difficult to solve because the computational cost of standard approximation methods grows exponentially with the number of dimensions. Therefore, we consider the following methods to overcome this difficulty. For the merged fully NPDEs and 2BSDEs system, combined with the time forward discretization and ReLU function, we use multiscale deep learning fusion and convolutional neural network (CNN) techniques to obtain two numerical approximation schemes, respectively. Finally, three practical high‐dimensional test problems involving Allen–Cahn, Black–Scholes–Barenblatt, and Hamilton–Jacobi–Bellman equations are given so that the first proposed method exhibits higher efficiency and accuracy than the existing method, while the second proposed method can extend the dimensionality of the completely NPDEs–2BSDEs system over 400 dimensions, from which the numerical results highlight the effectiveness of proposed methods.

Large deviations for Markov processes with switching and homogenisation via Hamilton–Jacobi–Bellman equations

We consider the context of molecular motors modelled by a diffusion process driven by the gradient of a weakly periodic potential that depends on an internal degree of freedom. The switch of the internal state, that can freely be interpreted as a molecular switch, is modelled as a Markov jump process that depends on the location of the motor. Rescaling space and time, the limit of the trajectory of the diffusion process homogenises over the periodic potential as well as over the internal degree of freedom. Around the homogenised limit, we prove the large deviation principle of trajectories with a method developed by Feng and Kurtz based on the analysis of an associated Hamilton–Jacobi–Bellman equation with an Hamiltonian that here, as an innovative fact, depends on both position and momenta.

UAV Trajectory Optimization for Maximum Soaring in Windy Environment

Optimal trajectory planning in a windy environment is a complex problem when airplane performance characteristics are considered. This paper introduces a novel form of Legendre pseudospectral optimization to solve boundary value problems in UAV trajectory planning. The proposed architecture applies Legendre–Gauss–Lobatto and Hamilton–Jacobi–Bellman equations to generate candidate pieces of trajectories with respect to the UAV dynamic constraints. Analytical performance-based solutions are also developed for sample cases to achieve an optimal criterion in trajectory planning. Moreover, the notion of wind soaring is exploited to use the beneficial effects of tailwind velocities in trajectory planning. Integral cost functions are handled by Gauss-type quadrature rules. Simulations demonstrate the efficiency of the pseudospectral method compared with other solvers, in eliminating the difficulties of boundary value problems by employing the boundary points in the interpolation equation. The effectiveness of the proposed approach is demonstrated through dynamic simulations.

On the fragility of the basis on the Hamilton–Jacobi–Bellman equation in economic dynamics

In this paper, we provide an example of the optimal growth model in which there exist infinitely many solutions to the Hamilton–Jacobi–Bellman equation but the value function does not satisfy this equation. We consider the cause of this phenomenon, and find that the lack of a solution to the original problem is crucial. We show that under several conditions, there exists a solution to the original problem if and only if the value function solves the Hamilton–Jacobi–Bellman equation. Moreover, in this case, the value function is the unique nondecreasing concave solution to the Hamilton–Jacobi–Bellman equation. We also show that without our conditions, this uniqueness result does not hold.

Decentralized Event-Triggered Tracking Control for Unmatched Interconnected Systems via Particle Swarm Optimization-Based Adaptive Dynamic Programming.

The problem of the large-scale interconnected system (LSIS) control is prevalent in practical engineering and is becoming increasingly complex. In this article, we propose a novel decentralized event-triggered tracking control (ETTC) strategy for a class of continuous-time nonlinear LSIS with unmatched interconnected terms and asymmetric input constraints. First, auxiliary subsystems are established to address the unmatched cross-linking terms. Next, the dynamics states of the tracking error and the exosystem are combined to construct a nominal augmented subsystem. By employing a nonquadratic performance function, the input-constrained decentralized tracking control problem is transformed into an optimal control problem for the nominal augmented subsystem. A group of independent parameters and event-triggered conditions are designed to save communication bandwidth and computational resources. Subsequently, the critic-only adaptive dynamic programming (ADP) method is used to solve the Hamilton-Jacobi-Bellman equation (HJBE) associated with the optimal control problem. To improve training success rate, the weights of the critic neural network (NN) are updated by introducing a particle swarm optimization algorithm (PSOA). The tracking error and the NN weights are proved to be uniformly ultimately bounded (UUB) under the proposed ETTC by using the Lyapunov extension theorem. Finally, the simulation example of an unmatched interconnected system is provided to verify the validity of the proposed decentralized method.

The defined contribution pension plan after retirement under the criterion of a revised loss considering the economic situation

&lt;abstract&gt; &lt;p&gt;Considering the economic situation, we investigate the optimal asset allocation of defined contribution pension funds with random payouts after retirement under a modified criterion of quadratic loss. The HJB equation is derived adhering to the dynamic programming principle, and the time-consistent optimal investment strategy is designed based on the calculus theory. Finally, under two different risk attitudes, namely surplus preference and risk aversion, the impact of key parameters on the optimal investment strategy and the function of minimum loss at the initial moment is compared and analyzed, the economic significance is demonstrated, and the rationality of the model is verified.&lt;/p&gt; &lt;/abstract&gt;

Optimal robust reinsurance contracts with investment strategy under variance premium principle

This paper investigates the optimal robust proportional reinsurance contracts with investment in a liquid financial market under variance premium principle in a principal-agent framework. The surplus process of the insurer (agent) is assumed to follow an approximating compound Poisson risk process. Both the insurer and reinsurer (principal) are allowed to invest in a risk-free asset and a risky asset whose price process is governed by the constant elasticity of variance model. The insurer and reinsurer aim to maximize the expected exponential utility of terminal wealth. The reinsurer is ambiguity-averse and has deterministic ambiguity aversion preferences against the diffusion risk caused by the financial market and the approximated diffusion risk which comes from the claim process. The reinsurance price is described by the reinsurer's safety loading which can be decided by the optimal strategies of the insurer and the reinsurer. By utilizing stochastic optimal control principle and HJB (or HJBI) equations, the optimal (robust) proportional reinsurance-investment strategies and the corresponding value functions are obtained explicitly. Numerical examples are provided.

Value functional and optimal feedback control in linear-quadratic optimal control problem for fractional-order system

In this paper, a finite-horizon optimal control problem involving a dynamical system described by a linear Caputo fractional differential equation and a quadratic cost functional is considered. An explicit formula for the value functional is given, which includes a solution of a certain Fredholm integral equation. A step-by-step feedback control procedure for constructing $ \varepsilon $-optimal controls with any accuracy $ \varepsilon &gt; 0 $ is proposed. The basis for obtaining these results is the study of a solution of the associated Hamilton–Jacobi–Bellman equation with so-called fractional coinvariant derivatives.

Mathematical modelling and optimal control analysis of pandemic dynamics as a hybrid system

The study of epidemics using mathematical modelling is critical in understanding its dynamics and proposing potential control measures. We propose a generalised epidemiological model corresponding to a pandemic wherein its dynamics is represented as a novel hybrid system obtained by coupling a deterministic model with a stochastic model. The hybrid system dynamics is established in individualistic (macroscopic) and intra-individualistic (microscopic) scales. The established hybrid system is then considered the basis for an optimal control problem, with the rate of vaccination and velocity of spatial dynamics taken as the control parameters affecting the system’s trajectory. We define the cost functional constituted by the continuous cost corresponding to the deterministic model and discrete costs corresponding to the transitions in the microscopic scale. The objective of the control problem is to find an optimal control pair of vaccination rate and spatial velocity, which minimises the cost functional. We use the Dynamic Programming Principle (DPP) as the optimisation technique, followed by verification of the value function obtained by DPP as a viscosity solution of the appropriate Hamilton–Jacobi–Bellman equation to analyse the existence of an optimal control pair to the hybrid system. We prove the existence of optimal controls to the multi-scale dynamics for pandemic modelling, along with an abstract method to synthesise it.