Hamilton-Jacobi-Bellman Research Articles

Complex stochastic optimal control foundation of quantum mechanics

Abstract Recent studies have extended the use of the stochastic Hamilton-Jacobi-Bellman (HJB) equation to include complex variables for deriving quantum mechanical equations. However, these studies often assume that it is valid to apply the HJB equation directly to complex numbers, an approach that overlooks the fundamental problem of comparing complex numbers when finding optimal controls. This paper explores the application of the HJB equation in the context of complex variables. It provides an in-depth investigation of the stochastic movement of quantum particles within the framework of stochastic optimal control theory. We obtain the complex diffusion coefficient in the stochastic equation of motion using the Cauchy-Riemann theorem, considering that the particle’s stochastic movement is described by two perfectly correlated real and imaginary stochastic processes. During the development of the covariant form of the HJB equation, we demonstrate that if the temporal stochastic increments of the two processes are perfectly correlated, then the spatial stochastic increments must be perfectly anti-correlated, and vice versa. The diffusion coefficient we derive has a form that enables the linearization of the HJB equation. The method for linearizing the HJB equation, along with the subsequent derivation of the Dirac equation, was developed in our previous work [V. Yordanov, Scientific Reports 14, 6507 (2024)]. These insights deepen our understanding of quantum dynamics and enhance the application of stochastic optimal control theory to quantum mechanics.

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

AbstractIn this article, we study a stochastic optimal control problem in the pathwise sense, as initially proposed by Lions and Souganidis in [C. R. Acad. Sci. Paris Ser. I Math., 327 (1998), pp. 735-741]. The corresponding Hamilton-Jacobi-Bellman (HJB) equation, which turns out to be a non-adapted stochastic partial differential equation, is analyzed. Making use of the viscosity solution framework, we show that the value function of the optimal control problem is the unique solution of the HJB equation. When the optimal drift is defined, we provide its characterization. Finally, we describe the associated conserved quantities, namely the space-time transformations leaving our pathwise action invariant.

Portfolio optimization with feedback strategies based on artificial neural networks

Dynamic portfolio optimization has significantly benefited from a wider adoption of deep learning (DL). While existing research has focused on how DL can applied to solving the Hamilton–Jacobi–Bellman (HJB) equation, some very recent developments propose to forego the derivation of HJB in favor of empirical utility maximization over dynamic allocation strategies expressed through artificial neural networks. In addition to simplicity and transparency, this approach is universally applicable, as it is essentially agnostic about market dynamics. We apply it to optimal portfolio allocation between cash account and risky asset following Heston model. The results appear on par with theoretical ones.

Optimized distributed formation control using identifier-critic-actor reinforcement learning for a class of stochastic nonlinear multi-agent systems

This article is to propose an adaptive reinforcement learning (RL)-based optimized distributed formation control for the unknown stochastic nonlinear single-integrator dynamic multi-agent system (MAS). For solving the issue of unknown dynamic, an adaptive identifier neural network (NN) is developed to determine the stochastic MAS under expectation sense. And then, for deriving the optimized formation control, the RL is putted into effect via constructing a pair of critic and actor NNs. With regard of the traditional RL optimal controls, their algorithm exists the inherent complexity, because their adaptive RL algorithm are derived from negative gradient of the square of Hamilton–Jacobi-Bellman (HJB) equation. As a result, these methods are difficultly extended to stochastic dynamical systems. However, since this adaptive RL laws are derived from a simple positive function rather than the square of HJB equation, it can make optimal control with simple algorithm. Therefore, this optimized formation scheme can be smoothly performed to the stochastic MAS. Finally, according to theorem proof and computer simulation, the optimized method can realize the required control objective.

Optimized Backstepping-Based Containment Control for Multiagent Systems With Deferred Constraints Using a Universal Nonlinear Transformation.

This article investigates an optimized containment control problem for multiagent systems (MASs), where all followers are subject to deferred full-state constraints. A universal nonlinear transformation is proposed for simultaneously handling the cases with and without constraints. Particularly, for the constrained case, initial values of states are flexibly managed to the midpoint between upper and lower boundaries by utilizing a state-shifting function, thus eliminating the initial restriction conditions. By deferred constraints, the state is forced to fall back into the restrictive boundaries within a preassigned time. A neural network (NN)-based reinforcement learning (RL) algorithm is executed under the identifier-critic-actor architecture, where the Hamilton-Jacobi-Bellman (HJB) equation is built in every subsystem to optimize control performance. For actor and critic NNs, updating laws are simplified, since the gradient descent method is performed based on a simple positive function rather than square of Bellman residual error. In view of the Lyapunov stability theorem and graph theory, it is proved that all signals are bounded and the outputs of followers can eventually enter into the convex hull constituted by leaders. Finally, simulations confirm the validity of the proposed approach.

Environmental management and restoration under unified risk and uncertainty using robustified dynamic Orlicz risk

Environmental management and restoration should be designed such that the risk and uncertainty owing to nonlinear stochastic systems can be successfully addressed. We apply the robustified dynamic Orlicz risk to the modeling and analysis of environmental management and restoration to consider both the risk and uncertainty within a unified theory. We focus on the control of a jump-driven hybrid stochastic system that represents macrophyte dynamics. The dynamic programming equation based on the Orlicz risk is first obtained heuristically, from which the associated Hamilton–Jacobi–Bellman (HJB) equation is derived. In the proposed Orlicz risk, the risk aversion of the decision-maker is represented by a power coefficient that resembles a certainty equivalence, whereas the uncertainty aversion is represented by the Kullback–Leibler divergence, in which the risk and uncertainty are handled consistently and separately. The HJB equation includes a new state-dependent discount factor that arises from the uncertainty aversion, which leads to a unique, nonlinear, and nonlocal term. The link between the proposed and classical stochastic control problems is discussed with a focus on control-dependent discount rates. We propose a finite difference method for computing the HJB equation. Finally, the proposed model is applied to an optimal harvesting problem for macrophytes in a brackish lake that contains both growing and drifting populations.

Optimized dead-zone inverse control using reinforcement learning and sliding-mode mechanism for a class of high-order nonlinear systems

An optimized control method is developed for a class of high-order nonlinear dynamic systems having controller dead-zone phenomenon. Dead-zone refers to the controller with zero behavior within a certain range, so it will inevitably affect system performance. In order to make the optimized control eliminate the effect of dead zone, the adaptive dead-zone inverse and reinforcement learning (RL) techniques are combined. The main idea is to find the desired optimized control using RL as the input of dead-zone inverse function and then to design the adaptive algorithm to estimate the unknown parameters of dead-zone inverse function, so that the competent system control can be yielded from the dead-zone function. However, most existing RL algorithms are difficult to apply in the dead zone inverse methods because of the algorithm complexity. The proposed RL greatly simplifies the algorithm because it derives the training rules from the negative gradient of a simple positive function yielded by the partial derivative of Hamilton–Jacobi-Bellman (HJB) equation. Meanwhile, the proposed dead-zone inverse method also requires fewer adaptive parameters. Finally, the proposed control is attested through theoretical proofs and simulation examples.

Minimizing the penalized goal-reaching probability with multiple dependent risks

We consider a robust optimal investment and reinsurance problem with multiple dependent risks for an Ambiguity-Averse Insurer (AAI), who wishes to minimize the probability that the value of the wealth process reaches a low barrier before a high goal. We assume that the insurer can purchase per-loss reinsurance for every class of insurance business and invest its surplus in a risk-free asset and a risky asset. Using the technique of stochastic control theory and solving the associated Hamilton-Jacobi-Bellman (HJB) equation, we derive the robust optimal investment-reinsurance strategy and the associated value function. We conclude that the robust optimal investment-reinsurance strategy coincides with the one without model ambiguity, but the value function differs. We also illustrate our results by numerical examples.

Time-consistent asset allocation for risk measures in a Lévy market

Focusing on gains & losses relative to a risk-free benchmark instead of terminal wealth, we consider an asset allocation problem to maximize time-consistently a mean-risk reward function with a general risk measure which is (i) law-invariant, (ii) cash- or shift-invariant, and (iii) positively homogeneous, and possibly plugged into a general function. Examples include (relative) Value at Risk, coherent risk measures, variance, and generalized deviation risk measures. We model the market via a generalized version of the multi-dimensional Black–Scholes model using α-stable Lévy processes and give supplementary results for the classical Black–Scholes model. The optimal solution to this problem is a Nash subgame equilibrium given by the solution of an extended Hamilton–Jacobi–Bellman equation. Moreover, we show that the optimal solution is deterministic under appropriate assumptions.

ЗАДАЧА ОПТИМАЛЬНОГО СТОХАСТИЧЕСКОГО УПРАВЛЕНИЯ И ОЦЕНКИ СТОИМОСТИ КОРПОРАТИВНЫХ ОБЛИГАЦИЙ, ЗАВИСЯЩИХ ОТ ВРЕМЕНИ И СЛУЧАЙНЫХ СОСТОЯНИЙ ПАРАМЕТРОВ

In the optimal stochastic control problem, one estimates, using the utility indifference method, the bond price and the premium of a default swap contract (Credit default swap) when the model parameters (market interest rate, drift coef-ficient, and volatility of risky underlying assets) are random functions of time and state. Namely, the interest rate of the risk-free asset depends on time, and the prices of risky assets are described by linear homogeneous stochastic dif-ferential equations (SDEs) with multiplicative noise. To do this, the authors consider a portfolio with a risk-free asset and a risky asset with no default risk. For each such portfolio, the authors determine the amount of risky assets that maximizes the expected utility of its final wealth. This quantity allows one to solve the parabolic partial differential equations arising from the Hamilton-Jacobi-Bellman (HJB) equation by transforming them into ordinary differential equations using the method of variable separation to obtain the instantaneous value function of each portfolio. The authors derive the bond price and the default swap-contract premium (CDS) rate, which are the amounts that provide the same level of the expected utility by investing all of one’s wealth in a portfolio that does not contain these credit instruments, or by investing these amounts in credit instruments and the remainder of one’s wealth in the portfolio.

Reinforcement learning-based robust formation control for Multi-UAV systems with switching communication topologies

This paper introduces a novel optimal robust formation method for quadcopter multiple unmanned aerial vehicle (multi-UAV) systems. Firstly, a reinforcement learning (RL) algorithm based on a unique gradient descent training approach is proposed to solve the Hamilton–Jacobi–Bellman (HJB) equation, which can effectively eliminate the requirement of the Persistent Excitation (PE) condition. Secondly, the robustness of the controlled system is emphasized, and an Uncertainty and Disturbance Estimator (UDE) observer is developed to suppress model uncertainty and external disturbances through filtering techniques. Furthermore, a switched sliding mode control technique according to the average dwell time (ADT) is employed to convert switching communication topology between UAVs dynamically, and the stability analysis of the corresponding closed-loop control systems is then performed by the use of Lyapunov analysis. Finally, the simulation examples are provided to verify the effectiveness of the designed control strategy.

Portfolio problem for the α−hypergeometric stochastic volatility model with consumption

This paper solves the finite horizon optimal consumption-investment problem for an investor that trades in the α−hypergeometric stochastic volatility model, with preferences based on a power utility function. To find the optimal strategy, we follow the dynamic programming approach. First, we perform a suitable change of variables in order to fully transform the non-linear Hamilton-Jacobi-Bellman (HJB) equation into a semilinear one. Next, we apply the Girsanov theorem to deduce an implicit Feynman-Kac formula depending upon the volatility process. The operator defined by the Feynman-Kac representation is shown to be a contraction operator on a designed function space. Finally, the Banach fixed point theorem yields the existence of a solution to the HJB equation in the designed space. Moreover, we apply a verification theorem to guarantee that the solution of the HJB equation coincides with the value function.

Deep neural networks-prescribed performance optimal control for stochastic nonlinear strict-feedback systems

This article explores the application of deep neural networks (DNNs) for optimized backstepping control design in a category of stochastic nonlinear strict-feedback systems with unknown dynamics, focusing on prescribed performance. DNNs, distinguished by their ability to handle high-complexity functions and enhance function approximation, are employed to estimate unknown nonlinear functions. The weight updating strategy for each layer of DNNs is determined through a Lyapunov function analysis. Tomitigate potential issues such as the tracking error exploding at uncertain moments, prescribed performance control (PPC) is introduced to constrain the tracking error within a predetermined range. Subsequently, a novel optimal tracking control scheme, integrating the backstepping method and Hamilton–Jacobi–Bellman (HJB) equation, is presented. The results indicate that all signals in the closed-loop system are bounded in probability, and the tracking error is maintained within a set of predefined arbitrarily small residuals. Simulation outcomes affirm the efficacy of the proposed control scheme.

Concurrent learning for adaptive pontryagin's maximum principle of nonlinear systems with inequality constraints

AbstractIn this article, a finite‐horizon adaptive Pontryagin's maximum principle is presented for nonlinear systems with state inequality constraints. Concurrent learning (CL) technique is adopted to identify the unknown parameters of the dynamic systems. Based on the identification model, a novel adaptive iterative algorithm under the Pontryagin's framework is introduced to learn the finite‐horizon optimal control solution. Convergence analysis of the algorithm is provided by showing that the cost function sequence is monotonically decreasing. Furthermore, we extend the adaptive iterative algorithm to time‐varying nonlinear systems. The new algorithm overcomes the technical obstacles of the existing adaptive/approximate dynamic programming (ADP) approaches to deal with the time‐varying characteristic of Hamilton–Jacobi–Bellman (HJB) partial differential equation (PDE), especially when state constraints exist. Simulation examples are carried out to validate the effectiveness of the theoretical results.

Optimal avoidance strategy based on nonlinear approximate analytic solution of non-cooperative differential game

Abstract This study examines the pursuit-evasion game involving unmanned aerial vehicles (UAVs), with a specific focus on the scenario of N-pursuers-one-escapee. The primary objective is to develop an optimal strategy for the escapee when the pursuers possess superior capabilities. To obtain this objective, we conduct the following study. Firstly, to enhance realism, a non-cooperative differential game model is formulated, incorporating multiple motion characteristics, including aerodynamics, overloading, and imposed constraints. Secondly, the end-value performance index is subsequently converted to an integral one, simplifying the solution process of the Hamilton-Jacobi-Bellman (HJB) equation. An iterative method is utilised to determine the covariates using the Cauchy initial value problem, and its convergence and uniqueness are established. The optimal avoidance strategy is subsequently derived from the covariates. Finally, the superiority of the proposed strategy is validated through simulation experiments and compared to three advanced optimal avoidance strategies. A total of 1,000 anti-jamming simulation experiments are conducted to verify the robustness of the proposed strategy.

Adaptive neural network dynamic surface optimal saturation control for single‐phase grid‐connected photovoltaic systems

AbstractAn adaptive neural network (NN) based optimal saturation control scheme is investigated for single‐phase grid‐connected photovoltaic (PV) systems by incorporating dynamic surface control (DSC) and adaptive dynamic programming (ADP) based on the backstepping control design framework. For each backstepping step, a critic‐actor architecture is constructed via reinforcement learning (RL), and the PV system is optimized according to the cost function in the architecture. Due to the nonlinearity, it is difficult to solve the Hamilton–Jacobi–Bellman (HJB) equation. The neural networks (NNs) are employed to approximate the solution of the HJB equation such that the optimal virtual control and the actual controller are obtained. By considering control input symmetric saturation nonlinearity link, constraints on pulse width modulation (PWM) are ensured. On this basis, the combination of backstepping control design and dynamic surface technique is used to overcome the shortcomings of “differential explosion” and simplify calculations. Based on the Lyapunov method, the stability analysis proves that all signals of the closed‐loop PV systems are semiglobally uniformly ultimately bounded (SGUUB). Simulation experiments and comparative results are given to verify the efficacy of the studied control strategy.

Energy Maximization Absorption of Wave Energy Converter Based on Fourier Pseudo-Spectral Method and Adaptive Dynamic Programming

In this paper, a novel noncausal control framework to address the energy maximization problem of wave energy converters (WECs) with physical constraints is proposed. The energy maximization problem of WECs is a constrained optimal control problem. The proposed control framework converts this problem into a reference trajectory-tracking problem through the Fourier pseudo-spectral method (FPSM) and utilizes the online tracking adaptive dynamic programming (OTADP) algorithm to realize real-time trajectory tracking for practical use in the ocean environment. Using the wave prediction technique, the optimal trajectory is generated online through a receding horizon (RH) implementation. A critic neural network (NN) is applied to approximate the optimal cost value function and calculate the error-tracking control by solving the associated Hamilton-Jacobi-Bellman (HJB) equation. The proposed WEC control framework improves computational efficiency and makes it feasible for the online control of the WEC in practice. Simulation results show the effects of the receding-horizon implementation of FPSM with different window lengths and window functions, while verifying the performances of tracking control and energy absorption of WECs in two different sea conditions.

ADP-based online compensation hierarchical sliding-mode control for partially unknown switched nonlinear systems with actuator failures

This article investigates an adaptive dynamic programming-based online compensation hierarchical sliding-mode control problem for a class of partially unknown switched nonlinear systems with actuator failures and uncertain perturbations under an identifier-critic neural networks architecture. Firstly, by introducing a cost function related to hierarchical sliding-mode surfaces for the nominal system, the original control problem is equivalently converted into an optimal control problem. To obtain this optimal control policy, the Hamilton–Jacobi–Bellman equation is solved through an adaptive dynamic programming method. Compared with conventional adaptive dynamic programming methods, the identifier-critic network architecture not only overcomes the limitation on the unknown internal dynamic but also eliminates the approximation error arising from the actor network. The weights in the critic network are tuned via the gradient descent approach and the experience replay technology, such that the persistence of excitation condition can be relaxed. Then, a compensation term containing hierarchical sliding-mode surfaces is used to offset uncertain actuator failures without the fault detection and isolation unit. Based on the Lyapunov stability theory, all states of the closed-loop nonlinear system are stable in the sense of uniformly ultimately boundedness. Finally, numerical and practical examples are given to demonstrate the effectiveness of our presented online compensation control strategy.

Large deviations for slow–fast processes on connected complete Riemannian manifolds

We consider a class of slow–fast processes on a connected complete Riemannian manifold M. The limiting dynamics as the scale separation goes to ∞ is governed by the averaging principle. Around this limit, we prove large deviation principles with an action-integral rate function for the slow process by nonlinear semigroup methods together with Hamilton–Jacobi–Bellman (HJB) equation techniques. Our main innovation is solving the comparison principle for viscosity solutions for the HJB equation on M and the construction of a variational viscosity solution for the non-smooth Hamiltonian, which lies at the heart of deriving the action integral representation for the rate function.

Adaptive dynamic programming-based multi-fault tolerant control of reconfigurable manipulator with input constraint

In this paper, a multi-fault tolerant controller considering actuator saturation is proposed. Based on the adaptive dynamic programming(ADP) algorithm, the fault tolerant control of the reconfigurable manipulator with sensor and actuator faults are carried out. Firstly, combined with the state space expression, the nonlinear transformation of sensor fault is performed by adopting the differential homeomorphism principle. An improved cost function is constructed based on the fault estimation function obtained by the fault observer, and combined with hyperbolic tangent function to deal with input constraint problem. Then, an evaluation neural network (NN) is established and the Hamilton–Jacobi–Bellman (HJB) equation is solved by online strategy iterative algorithm. Furthermore, based on Lyapunov theorem, the stability of reconfigurable manipulator systems with multi-fault are proved. Lastly, the simulation studies are used to certify the effectiveness of the presented fault tolerant control (FTC) scheme.