Hamilton-Jacobi-Bellman Equation Research Articles

Phase transition in a kinetic mean-field game model of inertial self-propelled agents

The framework of mean-field games (MFGs) is used for modeling the collective dynamics of large populations of non-cooperative decision-making agents. We formulate and analyze a kinetic MFG model for an interacting system of non-cooperative motile agents with inertial dynamics and finite-range interactions, where each agent is minimizing a biologically inspired cost function. By analyzing the associated coupled forward–backward in a time system of nonlinear Fokker–Planck and Hamilton–Jacobi–Bellman equations, we obtain conditions for closed-loop linear stability of the spatially homogeneous MFG equilibrium that corresponds to an ordered state with non-zero mean speed. Using a combination of analysis and numerical simulations, we show that when energetic cost of control is reduced below a critical value, this equilibrium loses stability, and the system transitions to a traveling wave solution. Our work provides a game-theoretic perspective to the problem of collective motion in non-equilibrium biological and bio-inspired systems.

Optimal Portfolio and Retirement Decisions with Costly Job Switching Options

In this paper, we consider the utility maximization problem of an agent regarding optimal consumption-investment, job-switching strategy, and the optimal early retirement date. The agent can switch between two jobs or job categories at any time before retirement, but incurs a cost when switching to a position offering higher labor income. The agent's utility maximization involves a combination of stochastic control for consumption and investment, switching control for job-switching, and optimal stopping for early retirement decisions, making it a non-trivial and highly challenging problem. By utilizing the dynamic programming principle, we can derive the nonlinear Hamilton-Jacobi-Bellman (HJB) equation in the form of a system of variational inequalities with obstacle constraints, which arises from the agent's optimization problem. We employ guess and verify methods based on economic intuition to derive the closed-form solution of this HJB equation and demonstrate, through a verification theorem, that this solution aligns with the solution to the agent's utility maximization problem.

Decentralized Robust Tracking Control of Interconnected Nonlinear‐Constrained Systems: A Dynamic Event‐Sampled Method

ABSTRACTThis paper presents a decentralized robust dynamic event‐sampled tracking (EST) control law for interconnected nonlinear‐constrained systems. The core of developing such a control law is to convert the original EST control problem into the event‐sampled decentralized stabilization problem of augmented interconnected systems. To address the transformed decentralized stabilization problem, an indirect approach relying on the optimal control methodology is proposed. Initially, a group of cost functions are constructed for the nominal subsystems related to the augmented interconnected systems. Then, the dynamic event‐sampling mechanisms are introduced for lessening the computational burden. Meanwhile, the event‐sampled Hamilton–Jacobi–Bellman equations (ES‐HJBEs) are proposed for the augmented interconnected systems. To approximately solve the ES‐HJBEs, the critic approximators are used with their parameters tuned under the reinforcement learning framework. After that, the uniform ultimate boundedness of the tracking errors and the approximators' parameter estimation errors are assured based on the Lyapunov theorem. Finally, a nonlinear plant is provided to validate the decentralized robust dynamic EST control law.

Data-driven distributed output consensus control for multi-agent systems with unknown internal state

This paper discusses the topic of optimal output consensus control of linear time-invariant discrete-time multi-agent systems (MASs). The attainment of optimal output consensus control for MASs hinges upon the resolution of the interconnected Hamilton–Jacobi-Bellman equation, a task typically precluded by the inherent intractability of analytical solutions. Furthermore, most real-world systems are too complex to obtain the internal states of systems. To address these issues, a modified deep Q-learning network is constructed using current and historical system data rather than a precise model of the system. First, reconstructing the internal state of each agent using an adaptive distributed observer based on output feedback prevents the system instability brought on by the augmented systems. Then the local error system of the agent can be redefined. Based on the redefined error system, a data-driven adaptive dynamic programming (ADP) method is introduced, realized by using the actor–critic neural network structure. In addition, an experience replay strategy is proposed to reduce the propagation of estimation bias and improve the learning speed. Finally, the comparative numerical simulations substantiate the efficacy of the proposed algorithm in a quantifiable manner.

Relaxed optimal control problem for a finite horizon G-SDE with delay and its application in economics

This paper investigates the existence of a G-relaxed optimal control of a controlled stochastic differential delay equation driven by G-Brownian motion (G-SDDE in short). First, we show that optimal control of G-SDDE exists for the finite horizon case. We present as an application of our result an economic model, which is represented by a G-SDDE, where we studied the optimisation of this model. We connected the corresponding Hamilton–Jacobi–Bellman equation of our controlled system to a decoupled G-forward backward stochastic differential delay equation (G-FBSDDE in short). Finally, we simulate this G-FBSDDE to get the optimal strategy and cost.

Event-triggered approximately optimized formation control of multi-agent systems with unknown disturbances via simplified reinforcement learning

An event-triggered formation control strategy is proposed for a multi-agent system (MAS) suffered from unknown disturbances. In identifier-critic-actor neural networks (NNs), the strategy only needs to calculate the negative gradient of an approximated Hamilton-Jacobi-Bellman (HJB) equation, instead of the gradient descent method associated with Bellman residual errors. This simplified method removes the requirement for a complicated gradient calculation process of residual square of HJB equation. The weights in critic-actor NNs only update as the triggered condition is violated, and the computational burdens caused by frequent updates are thus reduced. Without dynamics information in prior, a disturbance observer is also constructed to approximate disturbances in an MAS. From stability analysis, it is proven that all signals are bounded. Two numerical examples are illustrated to verify that the proposed control strategy is effective.

Optimal Consumption and Investment with Income Adjustment and Borrowing Constraints

In this paper, we address the utility maximization problem of an infinitely lived agent who has the option to increase their income. The agent can increase their income at any time, but doing so incurs a wealth cost proportional to the amount of the increase. To prevent the agent from infinitely increasing their income and borrowing against future income, we additionally consider a non-negative wealth constraint that prohibits borrowing based on future income. This utility maximization problem is a mixture of stochastic control, where the agent chooses consumption and investment, and singular control, where the agent chooses a non-decreasing income process. To solve this non-trivial and challenging problem, we derive the Hamilton–Jacobi–Bellman (HJB) equation with a gradient constraint using the dynamic programming principle (DPP). Then, using the guess-and-verify method and a linearization technique, we obtain a closed-form solution to the HJB equation and, based on this, find the optimal strategy.

ANALYSIS OF OPTIMAL PORTFOLIO ON FINITE AND SMALL-TIME HORIZONS FOR A STOCHASTIC VOLATILITY MODEL WITH MULTIPLE CORRELATED ASSETS

In this paper, we consider the portfolio optimization problem in a financial market where the underlying stochastic volatility model is driven by [Formula: see text]-dimensional Brownian motions. At first, we derive a Hamilton–Jacobi–Bellman equation including the correlations among the standard Brownian motions. We use an approximation method for the optimization of portfolios. With such approximation, the value function is analyzed using the first-order terms of expansion of the utility function in the powers of time to the horizon. The error of this approximation is controlled using the second-order terms of expansion of the utility function. It is also shown that the one-dimensional version of this analysis corresponds to a known result in the literature. We also generate a close-to-optimal portfolio near the time to horizon using the first-order approximation of the utility function. It is shown that the error is controlled by the square of the time to the horizon. Finally, we provide an approximation scheme to the value function for all times and generate a close-to-optimal portfolio.

Viscosity solutions of centralized control problems in measure spaces

This work focuses on a control problem in the Wasserstein space of probability measures over $\mathbb{R}^d$. Our aim is to link this control problem to a suitable Hamilton-Jacobi-Bellman (HJB) equation. We explore a notion of viscosity solution using test functions that are locally Lipschitz and locally semiconvex or semiconcave functions. This regularity allows to define a notion of viscosity and a Hamiltonian function relying on directional derivatives. Using a generalization of Ekeland's principle, we show that the corresponding HJB equation admits a comparison principle, and deduce that the value function is the unique solution in this viscosity sense. The PDE tools are developed in the general framework of Measure Differential Equations.

On penalized goal-reaching probability minimization under borrowing and short-selling constraints

Abstract We consider a robust optimal investment–reinsurance problem to minimize the goal-reaching probability that the value of the wealth process reaches a low barrier before a high goal for an ambiguity-averse insurer. The insurer invests its surplus in a constrained financial market, where the proportion of borrowed amount to the current wealth level is no more than a given constant, and short-selling is prohibited. We assume that the insurer purchases per-claim reinsurance to transfer its risk exposure to a reinsurer whose premium is computed according to the mean–variance premium principle. Using the stochastic control approach based on the Hamilton–Jacobi–Bellman equation, we derive robust optimal investment–reinsurance strategies and the corresponding value functions. We conclude that the behavior of borrowing typically occurs with a lower wealth level. Finally, numerical examples are given to illustrate our results.

Decentralized Event-Triggered Tracking Control for Unmatched Interconnected Systems via Particle Swarm Optimization-Based Adaptive Dynamic Programming.

The problem of the large-scale interconnected system (LSIS) control is prevalent in practical engineering and is becoming increasingly complex. In this article, we propose a novel decentralized event-triggered tracking control (ETTC) strategy for a class of continuous-time nonlinear LSIS with unmatched interconnected terms and asymmetric input constraints. First, auxiliary subsystems are established to address the unmatched cross-linking terms. Next, the dynamics states of the tracking error and the exosystem are combined to construct a nominal augmented subsystem. By employing a nonquadratic performance function, the input-constrained decentralized tracking control problem is transformed into an optimal control problem for the nominal augmented subsystem. A group of independent parameters and event-triggered conditions are designed to save communication bandwidth and computational resources. Subsequently, the critic-only adaptive dynamic programming (ADP) method is used to solve the Hamilton-Jacobi-Bellman equation (HJBE) associated with the optimal control problem. To improve training success rate, the weights of the critic neural network (NN) are updated by introducing a particle swarm optimization algorithm (PSOA). The tracking error and the NN weights are proved to be uniformly ultimately bounded (UUB) under the proposed ETTC by using the Lyapunov extension theorem. Finally, the simulation example of an unmatched interconnected system is provided to verify the validity of the proposed decentralized method.

Evolutionary Dynamics-Driven Learning of Optimal Decisions for Nonlinear System: Extend-Policy Iterative Algorithm.

An extend-policy iterative algorithm is proposed for solving the ecological evolving-lung cancer cells growth inhibition optimal drug delivery scheme. With the analysis of the cell proliferation-apoptosis process of lung cancer cells with primitive immune system and external drug interventions, such as chemotherapeutic drugs and immunological agents, a model of ecological containment of lung cancer cells mimicking injection labeling is constructed. The HJB equation for biological tissue damage has also been established by considering the concentration of lung cancer cells in the blood and the amount of drug administered. The final simulation experiment proved the effectiveness of the drug delivery scheme.

Complex stochastic optimal control foundation of quantum mechanics

Abstract Recent studies have extended the use of the stochastic Hamilton-Jacobi-Bellman (HJB) equation to include complex variables for deriving quantum mechanical equations. However, these studies often assume that it is valid to apply the HJB equation directly to complex numbers, an approach that overlooks the fundamental problem of comparing complex numbers when finding optimal controls. This paper explores the application of the HJB equation in the context of complex variables. It provides an in-depth investigation of the stochastic movement of quantum particles within the framework of stochastic optimal control theory. We obtain the complex diffusion coefficient in the stochastic equation of motion using the Cauchy-Riemann theorem, considering that the particle’s stochastic movement is described by two perfectly correlated real and imaginary stochastic processes. During the development of the covariant form of the HJB equation, we demonstrate that if the temporal stochastic increments of the two processes are perfectly correlated, then the spatial stochastic increments must be perfectly anti-correlated, and vice versa. The diffusion coefficient we derive has a form that enables the linearization of the HJB equation. The method for linearizing the HJB equation, along with the subsequent derivation of the Dirac equation, was developed in our previous work [V. Yordanov, Scientific Reports 14, 6507 (2024)]. These insights deepen our understanding of quantum dynamics and enhance the application of stochastic optimal control theory to quantum mechanics.

Discretization of Fractional Fully Nonlinear Equations by Powers of Discrete Laplacians

AbstractWe study discretizations of fractional fully nonlinear equations by powers of discrete Laplacians. Our problems are parabolic and of order $$\sigma \in (0,2)$$ σ ∈ ( 0 , 2 ) since they involve fractional Laplace operators $$(-\Delta )^{\sigma /2}$$ ( - Δ ) σ / 2 . They arise e.g. in control and game theory as dynamic programming equations – HJB and Isaacs equation – and solutions are non-smooth in general and should be interpreted as viscosity solutions. Our approximations are realized as finite-difference quadrature approximations and are 2nd order accurate for all values of $$\sigma $$ σ . The accuracy of previous approximations of fractional fully nonlinear equations depend on $$\sigma $$ σ and are worse when $$\sigma $$ σ is close to 2. We show that the schemes are monotone, consistent, $$L^\infty $$ L ∞ -stable, and convergent using a priori estimates, viscosity solutions theory, and the method of half-relaxed limits. We also prove a second order error bound for smooth solutions and present many numerical examples.

Dynamic Programming-Based Approach to Model Antigen-Driven Immune Repertoire Synthesis

This paper presents a novel approach to modeling the repertoire of the immune system and its adaptation in response to the evolutionary dynamics of pathogens associated with their genetic variability. It is based on application of a dynamic programming-based framework to model the antigen-driven immune repertoire synthesis. The processes of formation of new receptor specificity of lymphocytes (the growth of their affinity during maturation) are described by an ordinary differential equation (ODE) with a piecewise-constant right-hand side. Optimal control synthesis is based on the solution of the Hamilton–Jacobi–Bellman equation implementing the dynamic programming approach for controlling Gaussian random processes generated by a stochastic differential equation (SDE) with the noise in the form of the Wiener process. The proposed description of the clonal repertoire of the immune system allows us to introduce an integral characteristic of the immune repertoire completeness or the integrative fitness of the whole immune system. The quantitative index for characterizing the immune system fitness is analytically derived using the Feynman–Kac–Kolmogorov equation.

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

In this article, we study a stochastic optimal control problem in the pathwise sense, as initially proposed by Lions and Souganidis in [C. R. Acad. Sci. Paris Ser. I Math., 327 (1998), pp. 735-741]. The corresponding Hamilton-Jacobi-Bellman (HJB) equation, which turns out to be a non-adapted stochastic partial differential equation, is analyzed. Making use of the viscosity solution framework, we show that the value function of the optimal control problem is the unique solution of the HJB equation. When the optimal drift is defined, we provide its characterization. Finally, we describe the associated conserved quantities, namely the space-time transformations leaving our pathwise action invariant.

Linear State Optimal Control Problem with a Stochastic Switching Time

In this paper, we analyse an optimal control problem over a finite horizon with a stochastic switching time, assuming that the two optimal control problems present in its two stages have a particularly simple form called linear state. It is well known that linear state optimal control problems can be solved easily using the HJB equation approach and assuming that the value function is linear in the state. Unfortunately, this simplicity of solution does not extend to the problem with stochastic switching time. We prove that a necessary and sufficient condition for the problem to maintain a linear state structure is to assume that the hazard rate of the switching time depends only on the temporal variable. Finally, assuming that the hazard rate is constant, we completely characterise the solution of the obtained linear state optimal control problem.

Multiplayer hierarchical decision‐making for discrete‐time nonlinear networks of service via value iteration adaptive dynamic programming

AbstractIn this article, the multiplayer hierarchical decision‐making problem for discrete‐time nonlinear networks of service is studied from the perspective of Nash–Stackelberg–Nash games. A novel two‐level value iteration adaptive dynamic programming algorithm is developed to solve the coupled nonlinear Hamilton–Jacobi–Bellman equations associated with the game problem, which neither requires the system drift dynamics to be known as a priori nor requests the initial strategies to be admissible. Moreover, both the value function sequences and the strategy sequences generated by the algorithm converge to their theoretical optimality. An implementation framework for the algorithm is constructed by leveraging the regularized least‐squares method, and a convergence criterion that guarantees the admissibility of the approximated strategies is also proposed. The effectiveness of the proposed algorithm and implementation framework are finally demonstrated through two numerical simulation examples.

Time-consistent asset allocation for risk measures in a Lévy market

Focusing on gains & losses relative to a risk-free benchmark instead of terminal wealth, we consider an asset allocation problem to maximize time-consistently a mean-risk reward function with a general risk measure which is (i) law-invariant, (ii) cash- or shift-invariant, and (iii) positively homogeneous, and possibly plugged into a general function. Examples include (relative) Value at Risk, coherent risk measures, variance, and generalized deviation risk measures. We model the market via a generalized version of the multi-dimensional Black–Scholes model using α-stable Lévy processes and give supplementary results for the classical Black–Scholes model. The optimal solution to this problem is a Nash subgame equilibrium given by the solution of an extended Hamilton–Jacobi–Bellman equation. Moreover, we show that the optimal solution is deterministic under appropriate assumptions.

Deep Neural Network Approximations for the Stable Manifolds of the Hamilton–Jacobi–Bellman Equations

Deep Neural Network Approximations for the Stable Manifolds of the Hamilton–Jacobi–Bellman Equations