Hamilton-Jacobi-Bellman Equation Research Articles

Finite Differences on Sparse Grids for Continuous-Time Heterogeneous Agent Models

We present a finite difference method working on sparse grids to solve higher dimensional heterogeneous agent models. If one wants to solve the arising Hamilton–Jacobi–Bellman equation on a standard full grid, one faces the problem that the number of grid points grows exponentially with the number of dimensions. Discretizations on sparse grids only involve O(N(logN)d−1) degrees of freedom in comparison to the O(Nd) degrees of freedom of conventional methods, where N denotes the number of grid points in one coordinate direction and d is the dimension of the problem. While one can show convergence for the used finite difference method on full grids by using the theory introduced by Barles and Souganidis, we explain why one cannot simply use their results for sparse grids. Our numerical studies show that our method converges to the full grid solution for a two-dimensional model. We analyze the convergence behavior for higher dimensional models and experiment with different sparse grid adaptivity types.

Two stackelberg games in life insurance: Mean-variance criterion

Abstract We study two continuous-time Stackelberg games between a life insurance buyer and seller over a random time horizon. The buyer invests in a risky asset and purchases life insurance, and she maximizes a mean-variance criterion applied to her wealth at death. The seller chooses the insurance premium rate to maximize its expected wealth at the buyer’s random time of death. We consider two life insurance games: one with term life insurance and the other with whole life insurance—the latter with pre-commitment of the constant investment strategy. In the term life insurance game, the buyer chooses her life insurance death benefit and investment strategy continuously from a time-consistent perspective. We find the buyer’s equilibrium control strategy explicitly, along with her value function, for the term life insurance game by solving the extended Hamilton–Jacobi–Bellman equations. By contrast, in the whole life insurance game, the buyer pre-commits to a constant life insurance death benefit and a constant amount to invest in the risky asset. To solve the whole life insurance problem, we first obtain the buyer’s objective function and then we maximize that objective function over constant controls. Under both models, the seller maximizes its expected wealth at the buyer’s time of death, and we use the resulting optimal life insurance premia to find the Stackelberg equilibria of the two life insurance games. We also analyze the effects of the parameters on the Stackelberg equilibria, and we present some numerical examples to illustrate our results.

Robust Optimal Consensus Control for Nonlinear Multi‐agent Systems: Two Hybrid Dynamic Event‐Triggered‐Based Approach

ABSTRACTIn this paper, the dynamic event‐triggered robust optimal consensus control problem is investigated for nonlinear multi‐agent systems (MASs) with matched and mismatched disturbances via integral sliding mode (ISM) control and the adaptive dynamic programming (ADP) algorithm. First, a dynamic event‐triggered fixed‐time ISM controller is designed to guarantee that the system state converges to the predefined ISM manifold, thereby eliminating the matched disturbances. Next, the consensus control problem is transformed into an optimal control problem by constructing neighborhood error dynamics and a new modified cost function; thus, an event‐triggered‐based coupled Hamilton‐Jacobi‐Bellman equation (HJBE) is established. Then, an ADP‐based single‐critic neural network (NN) is constructed to solve coupled HJBE to obtain the event‐triggered optimal controller, in which the NN weight is updated only at the triggering instants. Through the implementation of these two dynamic event‐triggered mechanisms, resources and controller execution time can be saved. It is proved that the whole closed‐loop system signals are uniformly ultimately bounded by the Lyapunov technique. Finally, two illustrative examples verify the effectiveness and superiority of the proposed control scheme.

Phase transition in a kinetic mean-field game model of inertial self-propelled agents

The framework of mean-field games (MFGs) is used for modeling the collective dynamics of large populations of non-cooperative decision-making agents. We formulate and analyze a kinetic MFG model for an interacting system of non-cooperative motile agents with inertial dynamics and finite-range interactions, where each agent is minimizing a biologically inspired cost function. By analyzing the associated coupled forward–backward in a time system of nonlinear Fokker–Planck and Hamilton–Jacobi–Bellman equations, we obtain conditions for closed-loop linear stability of the spatially homogeneous MFG equilibrium that corresponds to an ordered state with non-zero mean speed. Using a combination of analysis and numerical simulations, we show that when energetic cost of control is reduced below a critical value, this equilibrium loses stability, and the system transitions to a traveling wave solution. Our work provides a game-theoretic perspective to the problem of collective motion in non-equilibrium biological and bio-inspired systems.

Optimal Portfolio and Retirement Decisions with Costly Job Switching Options

In this paper, we consider the utility maximization problem of an agent regarding optimal consumption-investment, job-switching strategy, and the optimal early retirement date. The agent can switch between two jobs or job categories at any time before retirement, but incurs a cost when switching to a position offering higher labor income. The agent's utility maximization involves a combination of stochastic control for consumption and investment, switching control for job-switching, and optimal stopping for early retirement decisions, making it a non-trivial and highly challenging problem. By utilizing the dynamic programming principle, we can derive the nonlinear Hamilton-Jacobi-Bellman (HJB) equation in the form of a system of variational inequalities with obstacle constraints, which arises from the agent's optimization problem. We employ guess and verify methods based on economic intuition to derive the closed-form solution of this HJB equation and demonstrate, through a verification theorem, that this solution aligns with the solution to the agent's utility maximization problem.

Decentralized Robust Tracking Control of Interconnected Nonlinear‐Constrained Systems: A Dynamic Event‐Sampled Method

ABSTRACTThis paper presents a decentralized robust dynamic event‐sampled tracking (EST) control law for interconnected nonlinear‐constrained systems. The core of developing such a control law is to convert the original EST control problem into the event‐sampled decentralized stabilization problem of augmented interconnected systems. To address the transformed decentralized stabilization problem, an indirect approach relying on the optimal control methodology is proposed. Initially, a group of cost functions are constructed for the nominal subsystems related to the augmented interconnected systems. Then, the dynamic event‐sampling mechanisms are introduced for lessening the computational burden. Meanwhile, the event‐sampled Hamilton–Jacobi–Bellman equations (ES‐HJBEs) are proposed for the augmented interconnected systems. To approximately solve the ES‐HJBEs, the critic approximators are used with their parameters tuned under the reinforcement learning framework. After that, the uniform ultimate boundedness of the tracking errors and the approximators' parameter estimation errors are assured based on the Lyapunov theorem. Finally, a nonlinear plant is provided to validate the decentralized robust dynamic EST control law.

Data-driven distributed output consensus control for multi-agent systems with unknown internal state

This paper discusses the topic of optimal output consensus control of linear time-invariant discrete-time multi-agent systems (MASs). The attainment of optimal output consensus control for MASs hinges upon the resolution of the interconnected Hamilton–Jacobi-Bellman equation, a task typically precluded by the inherent intractability of analytical solutions. Furthermore, most real-world systems are too complex to obtain the internal states of systems. To address these issues, a modified deep Q-learning network is constructed using current and historical system data rather than a precise model of the system. First, reconstructing the internal state of each agent using an adaptive distributed observer based on output feedback prevents the system instability brought on by the augmented systems. Then the local error system of the agent can be redefined. Based on the redefined error system, a data-driven adaptive dynamic programming (ADP) method is introduced, realized by using the actor–critic neural network structure. In addition, an experience replay strategy is proposed to reduce the propagation of estimation bias and improve the learning speed. Finally, the comparative numerical simulations substantiate the efficacy of the proposed algorithm in a quantifiable manner.

Relaxed optimal control problem for a finite horizon G-SDE with delay and its application in economics

This paper investigates the existence of a G-relaxed optimal control of a controlled stochastic differential delay equation driven by G-Brownian motion (G-SDDE in short). First, we show that optimal control of G-SDDE exists for the finite horizon case. We present as an application of our result an economic model, which is represented by a G-SDDE, where we studied the optimisation of this model. We connected the corresponding Hamilton–Jacobi–Bellman equation of our controlled system to a decoupled G-forward backward stochastic differential delay equation (G-FBSDDE in short). Finally, we simulate this G-FBSDDE to get the optimal strategy and cost.

Event-triggered approximately optimized formation control of multi-agent systems with unknown disturbances via simplified reinforcement learning

An event-triggered formation control strategy is proposed for a multi-agent system (MAS) suffered from unknown disturbances. In identifier-critic-actor neural networks (NNs), the strategy only needs to calculate the negative gradient of an approximated Hamilton-Jacobi-Bellman (HJB) equation, instead of the gradient descent method associated with Bellman residual errors. This simplified method removes the requirement for a complicated gradient calculation process of residual square of HJB equation. The weights in critic-actor NNs only update as the triggered condition is violated, and the computational burdens caused by frequent updates are thus reduced. Without dynamics information in prior, a disturbance observer is also constructed to approximate disturbances in an MAS. From stability analysis, it is proven that all signals are bounded. Two numerical examples are illustrated to verify that the proposed control strategy is effective.

Optimal Consumption and Investment with Income Adjustment and Borrowing Constraints

In this paper, we address the utility maximization problem of an infinitely lived agent who has the option to increase their income. The agent can increase their income at any time, but doing so incurs a wealth cost proportional to the amount of the increase. To prevent the agent from infinitely increasing their income and borrowing against future income, we additionally consider a non-negative wealth constraint that prohibits borrowing based on future income. This utility maximization problem is a mixture of stochastic control, where the agent chooses consumption and investment, and singular control, where the agent chooses a non-decreasing income process. To solve this non-trivial and challenging problem, we derive the Hamilton–Jacobi–Bellman (HJB) equation with a gradient constraint using the dynamic programming principle (DPP). Then, using the guess-and-verify method and a linearization technique, we obtain a closed-form solution to the HJB equation and, based on this, find the optimal strategy.

On penalized goal-reaching probability minimization under borrowing and short-selling constraints

Abstract We consider a robust optimal investment–reinsurance problem to minimize the goal-reaching probability that the value of the wealth process reaches a low barrier before a high goal for an ambiguity-averse insurer. The insurer invests its surplus in a constrained financial market, where the proportion of borrowed amount to the current wealth level is no more than a given constant, and short-selling is prohibited. We assume that the insurer purchases per-claim reinsurance to transfer its risk exposure to a reinsurer whose premium is computed according to the mean–variance premium principle. Using the stochastic control approach based on the Hamilton–Jacobi–Bellman equation, we derive robust optimal investment–reinsurance strategies and the corresponding value functions. We conclude that the behavior of borrowing typically occurs with a lower wealth level. Finally, numerical examples are given to illustrate our results.

Decentralized Event-Triggered Tracking Control for Unmatched Interconnected Systems via Particle Swarm Optimization-Based Adaptive Dynamic Programming.

The problem of the large-scale interconnected system (LSIS) control is prevalent in practical engineering and is becoming increasingly complex. In this article, we propose a novel decentralized event-triggered tracking control (ETTC) strategy for a class of continuous-time nonlinear LSIS with unmatched interconnected terms and asymmetric input constraints. First, auxiliary subsystems are established to address the unmatched cross-linking terms. Next, the dynamics states of the tracking error and the exosystem are combined to construct a nominal augmented subsystem. By employing a nonquadratic performance function, the input-constrained decentralized tracking control problem is transformed into an optimal control problem for the nominal augmented subsystem. A group of independent parameters and event-triggered conditions are designed to save communication bandwidth and computational resources. Subsequently, the critic-only adaptive dynamic programming (ADP) method is used to solve the Hamilton-Jacobi-Bellman equation (HJBE) associated with the optimal control problem. To improve training success rate, the weights of the critic neural network (NN) are updated by introducing a particle swarm optimization algorithm (PSOA). The tracking error and the NN weights are proved to be uniformly ultimately bounded (UUB) under the proposed ETTC by using the Lyapunov extension theorem. Finally, the simulation example of an unmatched interconnected system is provided to verify the validity of the proposed decentralized method.

Evolutionary Dynamics-Driven Learning of Optimal Decisions for Nonlinear System: Extend-Policy Iterative Algorithm.

An extend-policy iterative algorithm is proposed for solving the ecological evolving-lung cancer cells growth inhibition optimal drug delivery scheme. With the analysis of the cell proliferation-apoptosis process of lung cancer cells with primitive immune system and external drug interventions, such as chemotherapeutic drugs and immunological agents, a model of ecological containment of lung cancer cells mimicking injection labeling is constructed. The HJB equation for biological tissue damage has also been established by considering the concentration of lung cancer cells in the blood and the amount of drug administered. The final simulation experiment proved the effectiveness of the drug delivery scheme.

Complex stochastic optimal control foundation of quantum mechanics

Abstract Recent studies have extended the use of the stochastic Hamilton-Jacobi-Bellman (HJB) equation to include complex variables for deriving quantum mechanical equations. However, these studies often assume that it is valid to apply the HJB equation directly to complex numbers, an approach that overlooks the fundamental problem of comparing complex numbers when finding optimal controls. This paper explores the application of the HJB equation in the context of complex variables. It provides an in-depth investigation of the stochastic movement of quantum particles within the framework of stochastic optimal control theory. We obtain the complex diffusion coefficient in the stochastic equation of motion using the Cauchy-Riemann theorem, considering that the particle’s stochastic movement is described by two perfectly correlated real and imaginary stochastic processes. During the development of the covariant form of the HJB equation, we demonstrate that if the temporal stochastic increments of the two processes are perfectly correlated, then the spatial stochastic increments must be perfectly anti-correlated, and vice versa. The diffusion coefficient we derive has a form that enables the linearization of the HJB equation. The method for linearizing the HJB equation, along with the subsequent derivation of the Dirac equation, was developed in our previous work [V. Yordanov, Scientific Reports 14, 6507 (2024)]. These insights deepen our understanding of quantum dynamics and enhance the application of stochastic optimal control theory to quantum mechanics.

Discretization of Fractional Fully Nonlinear Equations by Powers of Discrete Laplacians

AbstractWe study discretizations of fractional fully nonlinear equations by powers of discrete Laplacians. Our problems are parabolic and of order $$\sigma \in (0,2)$$ σ ∈ ( 0 , 2 ) since they involve fractional Laplace operators $$(-\Delta )^{\sigma /2}$$ ( - Δ ) σ / 2 . They arise e.g. in control and game theory as dynamic programming equations – HJB and Isaacs equation – and solutions are non-smooth in general and should be interpreted as viscosity solutions. Our approximations are realized as finite-difference quadrature approximations and are 2nd order accurate for all values of $$\sigma $$ σ . The accuracy of previous approximations of fractional fully nonlinear equations depend on $$\sigma $$ σ and are worse when $$\sigma $$ σ is close to 2. We show that the schemes are monotone, consistent, $$L^\infty $$ L ∞ -stable, and convergent using a priori estimates, viscosity solutions theory, and the method of half-relaxed limits. We also prove a second order error bound for smooth solutions and present many numerical examples.

Environmental management and restoration under unified risk and uncertainty using robustified dynamic Orlicz risk

Environmental management and restoration should be designed such that the risk and uncertainty owing to nonlinear stochastic systems can be successfully addressed. We apply the robustified dynamic Orlicz risk to the modeling and analysis of environmental management and restoration to consider both the risk and uncertainty within a unified theory. We focus on the control of a jump-driven hybrid stochastic system that represents macrophyte dynamics. The dynamic programming equation based on the Orlicz risk is first obtained heuristically, from which the associated Hamilton–Jacobi–Bellman (HJB) equation is derived. In the proposed Orlicz risk, the risk aversion of the decision-maker is represented by a power coefficient that resembles a certainty equivalence, whereas the uncertainty aversion is represented by the Kullback–Leibler divergence, in which the risk and uncertainty are handled consistently and separately. The HJB equation includes a new state-dependent discount factor that arises from the uncertainty aversion, which leads to a unique, nonlinear, and nonlocal term. The link between the proposed and classical stochastic control problems is discussed with a focus on control-dependent discount rates. We propose a finite difference method for computing the HJB equation. Finally, the proposed model is applied to an optimal harvesting problem for macrophytes in a brackish lake that contains both growing and drifting populations.

Dynamic Programming-Based Approach to Model Antigen-Driven Immune Repertoire Synthesis

This paper presents a novel approach to modeling the repertoire of the immune system and its adaptation in response to the evolutionary dynamics of pathogens associated with their genetic variability. It is based on application of a dynamic programming-based framework to model the antigen-driven immune repertoire synthesis. The processes of formation of new receptor specificity of lymphocytes (the growth of their affinity during maturation) are described by an ordinary differential equation (ODE) with a piecewise-constant right-hand side. Optimal control synthesis is based on the solution of the Hamilton–Jacobi–Bellman equation implementing the dynamic programming approach for controlling Gaussian random processes generated by a stochastic differential equation (SDE) with the noise in the form of the Wiener process. The proposed description of the clonal repertoire of the immune system allows us to introduce an integral characteristic of the immune repertoire completeness or the integrative fitness of the whole immune system. The quantitative index for characterizing the immune system fitness is analytically derived using the Feynman–Kac–Kolmogorov equation.

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

In this article, we study a stochastic optimal control problem in the pathwise sense, as initially proposed by Lions and Souganidis in [C. R. Acad. Sci. Paris Ser. I Math., 327 (1998), pp. 735-741]. The corresponding Hamilton-Jacobi-Bellman (HJB) equation, which turns out to be a non-adapted stochastic partial differential equation, is analyzed. Making use of the viscosity solution framework, we show that the value function of the optimal control problem is the unique solution of the HJB equation. When the optimal drift is defined, we provide its characterization. Finally, we describe the associated conserved quantities, namely the space-time transformations leaving our pathwise action invariant.

Optimized distributed formation control using identifier–critic–actor reinforcement learning for a class of stochastic nonlinear multi-agent systems

This article is to propose an adaptive reinforcement learning (RL)-based optimized distributed formation control for the unknown stochastic nonlinear single-integrator dynamic multi-agent system (MAS). For solving the issue of unknown dynamic, an adaptive identifier neural network (NN) is developed to determine the stochastic MAS under expectation sense. And then, for deriving the optimized formation control, the RL is putted into effect via constructing a pair of critic and actor NNs. With regard of the traditional RL optimal controls, their algorithm exists the inherent complexity, because their adaptive RL algorithm are derived from negative gradient of the square of Hamilton–Jacobi–Bellman (HJB) equation. As a result, these methods are difficultly extended to stochastic dynamical systems. However, since this adaptive RL laws are derived from a simple positive function rather than the square of HJB equation, it can make optimal control with simple algorithm. Therefore, this optimized formation scheme can be smoothly performed to the stochastic MAS. Finally, according to theorem proof and computer simulation, the optimized method can realize the required control objective.

Linear State Optimal Control Problem with a Stochastic Switching Time

In this paper, we analyse an optimal control problem over a finite horizon with a stochastic switching time, assuming that the two optimal control problems present in its two stages have a particularly simple form called linear state. It is well known that linear state optimal control problems can be solved easily using the HJB equation approach and assuming that the value function is linear in the state. Unfortunately, this simplicity of solution does not extend to the problem with stochastic switching time. We prove that a necessary and sufficient condition for the problem to maintain a linear state structure is to assume that the hazard rate of the switching time depends only on the temporal variable. Finally, assuming that the hazard rate is constant, we completely characterise the solution of the obtained linear state optimal control problem.