Hamilton-Jacobi-Bellman Equation Research Articles

Online adaptive optimal control algorithm based on synchronous integral reinforcement learning with explorations

In this study, we present a novel algorithm, based on synchronous policy iteration, to solve the continuous-time infinite-horizon optimal control problem of input affine system dynamics. The integral reinforcement is measured as an excitation signal to estimate the solution to the Hamilton–Jacobi–Bellman equation. In addition, the proposed method is completely model-free, that is, no a priori knowledge of the system is required. Using the adaptive tuning law, the actor and critic neural networks can simultaneously approximate the optimal value function and policy. The persistence of excitation condition is required to guarantee the convergence of the two networks. Unlike in traditional policy iteration algorithms, the restriction of the initial admissible policy was eliminated using this method. The effectiveness of the proposed algorithm is verified through numerical simulations.

On the existence and uniqueness properties of the Hoogendoorn–Bovy pedestrian flow model

This study addresses the existence and uniqueness of solutions to the Hoogendoorn–Bovy (HB) pedestrian flow model, which describes the dynamic user-optimal pedestrian flow assignment problem in continuous space and time. The HB model consists of a forward conservation law (CL) equation that governs density and a backward Hamilton–Jacobi–Bellman (HJB) equation that contains a maximum admissible speed constraint (MASC), in which the flow direction is determined by the path-choice strategy. The existence and uniqueness of solutions are significantly more difficult to determine when the HJB equation contains an MASC; however, we prove that the HB model can be formulated as a forward CL equation and backward Hamilton–Jacobi (HJ) equation in which the MASC is non-binding if suitable model parameters are selected. This model is formulated as a fixed-point problem upon the simultaneous satisfaction of both equations. To verify the existence and uniqueness results, we first demonstrate the existence and uniqueness of the solutions to the CL and HJ equations, and then show that the coupled HB model is well-posed and has a unique solution. A numerical example is presented to illustrate the properties of the HB model.

Event-triggered-based online integral reinforcement learning for optimal control of unknown constrained nonlinear systems

For unknown nonlinear systems with actuator saturation, an online policy iteration-based algorithm is employed to solve the optimal event-triggered control problem. To learn the system dynamics, a novel identifier is proposed to make the estimation error converge quickly and the experience replay technique is employed to release the persistence of the excitation condition. To approximate the cost function and the event-triggered control law, we present event-triggered-based critic and actor networks, whose weights are updated only at triggered instants. During the policy iteration process, an event-triggered-based integral reinforcement learning method is proposed to solve the Hamilton–Jacobi–Bellman equation. By utilising the integral reinforcement learning, the network resource is saved and learning efficiency is improved. Based on the Lyapunov method, stability for the closed-loop system and estimation errors for the three networks are analysed. At last, simulation results of two numerical examples are used to show the effectiveness of the proposed method.

Event-triggered adaptive dynamic programming for decentralized tracking control of input constrained unknown nonlinear interconnected systems

This paper addresses decentralized tracking control (DTC) problems for input constrained unknown nonlinear interconnected systems via event-triggered adaptive dynamic programming. To reconstruct the system dynamics, a neural-network-based local observer is established by using local input–output data and the desired trajectories of all other subsystems. By employing a nonquadratic value function, the DTC problem of the input constrained nonlinear interconnected system is transformed into an optimal control problem. By using the observer-critic architecture, the DTC policy is obtained by solving the local Hamilton–Jacobi–Bellman equation through the local critic neural network, whose weights are tuned by the experience replay technique to relax the persistence of excitation condition. Under the event-triggering mechanism, the DTC policy is updated at the event-triggering instants only. Then, the computational resource and the communication bandwidth are saved. The stability of the closed-loop system is guaranteed by implementing event-triggered DTC policy via Lyapunov’s direct method. Finally, simulation examples are provided to demonstrate the effectiveness of the proposed scheme.

Optimal Controls of Stochastic Differential Equations with Jumps and Random Coefficients: Stochastic Hamilton–Jacobi–Bellman Equations with Jumps

We study the stochastic Hamilton–Jacobi–Bellman (HJB) equation with jump, which arises from a non-Markovian optimal control problem with a recursive utility cost functional. The solution to the equation is a predictable triplet of random fields. We show that the value function of the control problem, under some regularity assumptions, is the solution to the stochastic HJB equation; and a classical solution to this equation is the value function and characterizes the optimal control. With some additional assumptions on the coefficients, an existence and uniqueness result in the sense of Sobolev space is shown by recasting the stochastic HJB equation as a backward stochastic evolution equation in Hilbert spaces with the Brownian motion and Poisson jump.

Dynamic event-triggered-based single-network ADP optimal tracking control for the unknown nonlinear system with constrained input

This paper presents an event-triggered-based optimal tracking control scheme for the unknown nonlinear system with unmeasurable state, constrained input, and limited communication. Firstly, an event-triggered Hamilton–Jacobi-Bellman (HJB) equation is derived based on an augmented system and a discounted cost function. Next, a novel online event-triggered-based neural networks (NNs) observer (NNsO) is constructed to obtain the unmeasurable state and the unknown system dynamics. Then, a single critic neural network (NN) based on the adaptive dynamic programming (ADP) framework is established to solve the event-triggered HJB equation, and the designed critic NN weight updating law can relax the restrictions of initial admissible control and persistence of excitation condition. Furthermore, a static triggering rule with an exponential term is designed to reduce communication and computing burden, and based on this, a dynamic triggering rule is established by introducing an auxiliary dynamic variable to obtain efficient communication and computing. It is proved that the tracking error and all the NN weight estimation errors are uniformly ultimately bounded (UUB), and the Zeno behavior is excluded. Finally, two simulation examples are presented to verify the effectiveness and superiority of the proposed control scheme.

Optimal control of compartmental models: The exact solution

We formulate a control problem for positive compartmental systems formed by nodes (buffers) and arcs (flows). Our main result is that, on a finite horizon, we can solve the Pontryagin equations in one shot without resorting to trial and error via shooting. As expected, the solution is bang–bang and the switching times can be easily determined. We are also able to find a cost-to-go-function, in an analytic form, by solving a simple nonlinear differential equation. On an infinite horizon, we consider the Hamilton–Jacobi–Bellman theory and we show that the HJB equation can be solved exactly. Moreover, we show that the optimal solution is constant and the cost-to-go function is linear and copositive. This function is the solution of a nonlinear equation. We propose an iterative scheme for solving this equation, which converges in finite time. We also show that an exact solution can be found if there is a positive external disturbance affecting the process and the problem is formulated in a minsup framework. We finally provide illustrative examples related to flood control and epidemiology.

A Multi-Population Mean-Field Game Approach for Large-Scale Agents Cooperative Attack-Defense Evolution in High-Dimensional Environments

As one of the important issues of multi-agent collaboration, the large-scale agents’ cooperative attack–defense evolution requires a large number of agents to make stress-effective strategies to achieve their goals in complex environments. Multi-agent attack and defense in high-dimensional environments (3D obstacle scenarios) present the challenge of being able to accurately control high-dimensional state quantities. Moreover, the large scale makes the dynamic interactions in the attack and defense problems increase dramatically, which, using traditional optimal control techniques, can cause a dimensional explosion. How to model and solve the cooperative attack–defense evolution problem of large-scale agents in high-dimensional environments have become a challenge. We jointly considered energy consumption, inter-group attack and defense, intra-group collision avoidance, and obstacle avoidance in their cost functions. Meanwhile, the high-dimensional state dynamics were used to describe the motion of agents under environmental interference. Then, we formulated the cooperative attack–defense evolution of large-scale agents in high-dimensional environments as a multi-population high-dimensional stochastic mean-field game (MPHD-MFG), which significantly reduced the communication frequency and computational complexity. We tractably solved the MPHD-MFG with a generative-adversarial-network (GAN)-based method using the MFGs’ underlying variational primal–dual structure. Based on our approach, we carried out an integrative experiment in which we analytically showed the fast convergence of our cooperative attack–defense evolution algorithm by the convergence of the Hamilton–Jacobi–Bellman equation’s residual errors. The experiment also showed that a large number of drones can avoid obstacles and smoothly evolve their attack and defense behaviors while minimizing their energy consumption. In addition, the comparison with the baseline methods showed that our approach is advanced.

Decentralized Neurocontroller Design With Critic Learning for Nonlinear-Interconnected Systems.

We consider the decentralized control problem of a class of continuous-time nonlinear systems with mismatched interconnections. Initially, with the discounted cost functions being introduced to auxiliary subsystems, we have the decentralized control problem converted into a set of optimal control problems. To derive solutions to these optimal control problems, we first present the related Hamilton-Jacobi-Bellman equations (HJBEs). Then, we develop a novel critic learning method to solve these HJBEs. To implement the newly developed critic learning approach, we only use critic neural networks (NNs) and tune their weight vectors via the combination of a modified gradient descent method and concurrent learning. By using the present critic learning method, we not only remove the restriction of initial admissible control but also relax the persistence-of-excitation condition. After that, we employ Lyapunov's direct method to demonstrate that the critic NNs' weight estimation error and the states of closed-loop auxiliary systems are stable in the sense of uniform ultimate boundedness. Finally, we separately provide a nonlinear-interconnected plant and an unstable interconnected power system to validate the present critic learning approach.

A Reconfiguration-Based Fault-Tolerant Control Method for Nonlinear Uncertain Systems

This article addresses reconfiguration-based fault-tolerant control (FTC) with robust guaranteed performance for nonlinear uncertain systems under actuator faults. A PAssive/ACTive combined FTC method is proposed based on reliable control, which synthesizes the advantages of the active and passive FTC and strikes a balance between the performance and complexity of the method. Moreover, the reliability overcost of this method is reduced to balance the system nominal performance and reconfigurability. In this procedure, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\theta$</tex-math></inline-formula> -D method is used to solve the Hamilton–Jacobi–Bellman (HJB) and generalized HJB equations. Effectiveness of the proposed method is verified by a numerical example.

Risk-Averse Optimal Control in Continuous Time by Nesting Risk Measures

This paper extends dynamic control problems from a risk-neutral to a risk-averse setting. We establish a limit for consecutive risk-averse decision making by consistently and adequately nesting coherent risk measures. This approach provides a new perspective on multistage optimal control problems in continuous time. For the limiting case, we elaborate a new dynamic programming principle, which is risk averse, and give risk-averse Hamilton–Jacobi–Bellman equations by generalizing the infinitesimal generator. In doing so, we provide a constructive explanation of the driver g in g-expectation, a dynamic risk measure based on backward stochastic differential equations. Funding: This work was supported by Deutsche Forschungsgemeinschaft [Project-ID 416228727 – SFB 1410].

Research on the Emission Reduction Decision of Cost-Sharing Logistics Service Supply Chain in the O2O Model

As an effective way to realize energy savings and environmental protection, cost sharing is gradually becoming an important measure to reduce emissions in the logistics service supply chain under O2O mode in recent years. How to conduct contract selection and design optimization under the cost-sharing situation, and then improve the operational efficiency of the logistics service supply chain is an important issue that needs to be addressed. Firstly, based on the initial market demand for logistics, this paper involves the influence of both online logistics service integrators and onsite functional logistics service providers on logistics market demand in terms of emission reduction and platform brand image and develops a model based on the logistics service demand function in the O2O mode. Secondly, for the role of online and onsite emission reduction services in multi-cycle continuous cooperation to enhance the platform integrator’s brand image, a cost-sharing differential game model between online and onsite services is developed to facilitate providers’ adoption of high-quality emission reduction services. Finally, the HJB equation is used to compare the non-cooperative Nash game, the cost-sharing Stackelberg game, and the cooperative game to make the optimal abatement decision, the optimal benefit, and the cost-sharing ratio of the abatement service supply chain in the non-cooperative Nash game, the cost-sharing Stackelberg game, and the cooperative game. By comparing the results of the three games, we find that the optimal onsite and online abatement service decision is related to the cost, marginal revenue, and the impact of the service on demand; the abatement cost-sharing contract and cooperation are both better than the non-cooperative independent decision state, which can effectively guide the provision of high-quality onsite abatement service and improve the revenue of both parties involved in the logistics service supply chain and the total system’s revenue in the O2O mode. Compared with the cooperative game model, the cost-sharing contract can more effectively facilitate close cooperation between the actors, and the relationship between onsite and online marginal revenue affects the improvement of both parties’ revenue. The findings of the study can provide useful managerial insights for the selection and design optimization of abatement contracts for logistics service supply chains considering cost-sharing via the O2O model.

An algorithm based on an iterative optimal stopping method for Feller processes with applications to impulse control, perturbation, and possibly zero random discount problems

In this paper we present an iterative optimal stopping method for general optimal stopping problems for Feller processes. We show using an approximating scheme that the value function of an optimal stopping problem for some general operator is the unique viscosity solution to an Hamilton–Jacobi–Bellman equation (see for example Theorems 2.3 and 2.4). We apply our results to study impulse control problems for Feller–Markov processes and derive explicit solutions in the case of one dimensional regular Feller diffusion. We also use our result to study optimal stopping problems for both regime switching and semi-Markov processes and characterize their value functions as the limit of iterative optimal stopping problems (see Corollary 4.2 and Proposition 4.3 ). Finally, we examine optimal stopping problems for random (possibly zero) discount.

Wind Markov models and reliability control in wind energy conversion systems

The integration of renewable energy systems affects the entire power system by increasing uncertainty. Power system operators always have to manage uncertainty in one way or another; the fundamental principle in this regard is to guarantee reliability, i.e. balancing load variations with generation resources. This paper addresses the problem of uncertainty management or equivalently reliability control in power systems with wind generation. Our main approach is a reformulation as stochastic optimal control for a hybrid stochastic differential system, namely, by means of modelling the wind speed as a diffusion process in steady state. Accordingly, in theory, the dynamic programming principle provides the only closed-form exact solution to the problem of reliability control. But in practice, its complexity resides in the resolution of the Hamilton–Jacobi–Bellman equation. This paper explores a systematic and feasible alternative: successive approximations of the exact solution that could be referred to as sequential linear-dynamic programming (SLDP in abridged notation). The convergence result of the approximating solution brings together two key points: first, the convergence in the sense of the L2-norm; second, the decay rate of the cost over the course of the iterations until convergence. The effectiveness of the SLDP approach has been demonstrated by simulation experiments.

Optimized Backstepping Control Using Reinforcement Learning of Observer-Critic-Actor Architecture Based on Fuzzy System for a Class of Nonlinear Strict-Feedback Systems

In this article, a fuzzy logic system (FLS)-based adaptive optimized backstepping control is developed by employing reinforcement learning (RL) strategy for a class of nonlinear strict feedback systems with unmeasured states. For making the virtual and actual controls are optimized solution of the corresponding subsystem, RL of observer-critic-actor architecture based on FLS approximations is constructed in every backstepping step, where the observer aims to estimate the unmeasurable states, and the critic and actor aim to evaluate control performance and perform control behavior, respectively. In the proposed optimized control, on the one hand, the state observer method can avoid to require the design constants making its characteristic polynomial Hurwitz, which is universally demanded in the existing observer methods; on the other hand, the RL is significantly simple in algorithm, because the critic and actor training laws are derived from the negative gradient of a simple positive function, which is produced from the partial derivative of Hamilton–Jacobi–Bellman (HJB) equation, instead of the square of approximated HJB equation. Therefore this optimized scheme can be more easily applied and widely extended. Finally, from two aspects of theory and simulation, it is demonstrated that the desired objective can be fulfilled.

Mean–variance portfolio selection with dynamic attention behavior in a hidden Markov model

In this paper, we study closed-loop equilibrium strategies for mean–variance portfolio selection problems in a hidden Markov model with dynamic attention behavior. In addition to the investment strategy, the investor’s attention to news is introduced as a control of the accuracy of the news signal process. The objective is to find equilibrium strategies by numerically solving an extended HJB equation by using Markov chain approximation method. An iterative algorithm is constructed and its convergence is established. Numerical examples are provided to illustrate the results.

Robust optimal asset–liability management with delay and ambiguity aversion in a jump-diffusion market

In this paper, we consider a robust optimal asset–liability management problem with delay for an ambiguity-averse investor (AAI), who does not hold a firm belief towards not only the drift parameters but also the jump intensity in the wealth process. The objective of AAI is to maximise the robust value involving the expected utility of a weighted sum of terminal wealth and a penalisation of model uncertainty. In addition, the wealth process is modelled by a stochastic delay equation with jump via introducing the performance-related capital inflow/outflow. Applying the technique of robust optimal theory and Hamilton–Jacobi–Bellman equation, we derive the optimal results and the proof of verification theorem for the general utility functions and liability processes. In the case of the exponential utility function and a positive liability process, we prove the existence and uniqueness of the robust optimal investment strategy and find it naturally satisfies the constraint of no-short selling. Meanwhile, we adopt a special method to solve the corresponding Riccati equation and obtain the value function in explicit form. Furthermore, the results under the complete memory are discussed in detail. Finally, some numerical examples are provided to illustrate the influence of model parameters on the optimal strategy as well as the economic interpretation behind it.

Robust tracking control with reinforcement learning for nonlinear‐constrained systems

AbstractThis article considers the robust tracking control problem of uncertain nonlinear systems with asymmetric input constraints. Initially, the tracking error dynamics and the desired trajectory dynamics are constructed as an augmented system. Then, with a discounted value function being introduced for the nominal augmented system, the original tracking control problem is transformed into a constrained optimal control problem. To solve the constrained optimal control problem, its related Hamilton–Jacobi–Bellman equation (HJBE) is developed. After that, a critic approximator is constructed to solve the HJBE in the reinforcement learning framework. To tune the parameters used in the critic approximator, a novel concurrent learning technique is introduced, which could remove the persistence of excitation condition. Moreover, the uniform ultimate boundedness of the tracking error and the parameters' estimation error of the critic approximator is assured via the Lyapunov's approach. Finally, a spring‐mass‐damper mechanical system and a robot manipulator system are given to validate the theoretical claims.

Time-energy suboptimal control of nonlinear input-affine systems

In this paper, we derive the necessary conditions for time-energy optimal control of input-affine nonlinear systems using Hamilton–Jacobi–Bellman equation. The objective function is chosen as a weighted combination of control effort and time-horizon minimisation. The necessary conditions of optimality are transformed such that the resultant conditions are structurally similar to a differential matrix Riccati equation. A closed-form approximate solution to these necessary conditions is derived such that the resulting closed-loop dynamics is locally exponentially stable around the origin. Furthermore, it is also shown that the states converge within a ball of an arbitrarily small radius around the origin within a finite time. Finally, the performance of the proposed controller is tested using numerical simulations for Vander Pol's oscillator as well as spacecraft detumbling problem. Furthermore, the performance of the proposed controller is also tested in comparison to the infinite-horizon state-dependent Riccati equation controller.

Uncertain Stochastic Optimal Control with Jump and Its Application in a Portfolio Game

This article describes a class of jump-uncertain stochastic control systems, and derives an Itô–Liu formula with jump. We characterize an optimal control law, that satisfies the Hamilton–Jacobi–Bellman equation with jump. Then, this paper deduces the optimal portfolio game under uncertain stochastic financial markets with jump. The information of players is symmetrical. The financial market is constituted of a risk-free asset and a risky asset whose price process is subjected to the jump-uncertain stochastic Black–Scholes model. The game is formulated by two utility maximization problems, each investor tries to maximize his relative utility, which is the weighted average of terminal wealth difference between his terminal wealth and that of his competitor. Finally, the explicit expressions of equilibrium investment strategies and value functions for the constant absolute risk-averse and constant relative risk-averse utility function are derived by using the dynamic programming principle.