Hamilton-Jacobi-Bellman Equation Research Articles

Relaxed optimal control problem for a finite horizon G-SDE with delay and its application in economics

This paper investigates the existence of a G-relaxed optimal control of a controlled stochastic differential delay equation driven by G-Brownian motion (G-SDDE in short). First, we show that optimal control of G-SDDE exists for the finite horizon case. We present as an application of our result an economic model, which is represented by a G-SDDE, where we studied the optimisation of this model. We connected the corresponding Hamilton–Jacobi–Bellman equation of our controlled system to a decoupled G-forward backward stochastic differential delay equation (G-FBSDDE in short). Finally, we simulate this G-FBSDDE to get the optimal strategy and cost.

Optimal Consumption and Investment with Income Adjustment and Borrowing Constraints

In this paper, we address the utility maximization problem of an infinitely lived agent who has the option to increase their income. The agent can increase their income at any time, but doing so incurs a wealth cost proportional to the amount of the increase. To prevent the agent from infinitely increasing their income and borrowing against future income, we additionally consider a non-negative wealth constraint that prohibits borrowing based on future income. This utility maximization problem is a mixture of stochastic control, where the agent chooses consumption and investment, and singular control, where the agent chooses a non-decreasing income process. To solve this non-trivial and challenging problem, we derive the Hamilton–Jacobi–Bellman (HJB) equation with a gradient constraint using the dynamic programming principle (DPP). Then, using the guess-and-verify method and a linearization technique, we obtain a closed-form solution to the HJB equation and, based on this, find the optimal strategy.

ANALYSIS OF OPTIMAL PORTFOLIO ON FINITE AND SMALL-TIME HORIZONS FOR A STOCHASTIC VOLATILITY MODEL WITH MULTIPLE CORRELATED ASSETS

In this paper, we consider the portfolio optimization problem in a financial market where the underlying stochastic volatility model is driven by [Formula: see text]-dimensional Brownian motions. At first, we derive a Hamilton–Jacobi–Bellman equation including the correlations among the standard Brownian motions. We use an approximation method for the optimization of portfolios. With such approximation, the value function is analyzed using the first-order terms of expansion of the utility function in the powers of time to the horizon. The error of this approximation is controlled using the second-order terms of expansion of the utility function. It is also shown that the one-dimensional version of this analysis corresponds to a known result in the literature. We also generate a close-to-optimal portfolio near the time to horizon using the first-order approximation of the utility function. It is shown that the error is controlled by the square of the time to the horizon. Finally, we provide an approximation scheme to the value function for all times and generate a close-to-optimal portfolio.

Viscosity solutions of centralized control problems in measure spaces

This work focuses on a control problem in the Wasserstein space of probability measures over $\mathbb{R}^d$. Our aim is to link this control problem to a suitable Hamilton-Jacobi-Bellman (HJB) equation. We explore a notion of viscosity solution using test functions that are locally Lipschitz and locally semiconvex or semiconcave functions. This regularity allows to define a notion of viscosity and a Hamiltonian function relying on directional derivatives. Using a generalization of Ekeland's principle, we show that the corresponding HJB equation admits a comparison principle, and deduce that the value function is the unique solution in this viscosity sense. The PDE tools are developed in the general framework of Measure Differential Equations.

On penalized goal-reaching probability minimization under borrowing and short-selling constraints

Abstract We consider a robust optimal investment–reinsurance problem to minimize the goal-reaching probability that the value of the wealth process reaches a low barrier before a high goal for an ambiguity-averse insurer. The insurer invests its surplus in a constrained financial market, where the proportion of borrowed amount to the current wealth level is no more than a given constant, and short-selling is prohibited. We assume that the insurer purchases per-claim reinsurance to transfer its risk exposure to a reinsurer whose premium is computed according to the mean–variance premium principle. Using the stochastic control approach based on the Hamilton–Jacobi–Bellman equation, we derive robust optimal investment–reinsurance strategies and the corresponding value functions. We conclude that the behavior of borrowing typically occurs with a lower wealth level. Finally, numerical examples are given to illustrate our results.

Complex stochastic optimal control foundation of quantum mechanics

Abstract Recent studies have extended the use of the stochastic Hamilton-Jacobi-Bellman (HJB) equation to include complex variables for deriving quantum mechanical equations. However, these studies often assume that it is valid to apply the HJB equation directly to complex numbers, an approach that overlooks the fundamental problem of comparing complex numbers when finding optimal controls. This paper explores the application of the HJB equation in the context of complex variables. It provides an in-depth investigation of the stochastic movement of quantum particles within the framework of stochastic optimal control theory. We obtain the complex diffusion coefficient in the stochastic equation of motion using the Cauchy-Riemann theorem, considering that the particle’s stochastic movement is described by two perfectly correlated real and imaginary stochastic processes. During the development of the covariant form of the HJB equation, we demonstrate that if the temporal stochastic increments of the two processes are perfectly correlated, then the spatial stochastic increments must be perfectly anti-correlated, and vice versa. The diffusion coefficient we derive has a form that enables the linearization of the HJB equation. The method for linearizing the HJB equation, along with the subsequent derivation of the Dirac equation, was developed in our previous work [V. Yordanov, Scientific Reports 14, 6507 (2024)]. These insights deepen our understanding of quantum dynamics and enhance the application of stochastic optimal control theory to quantum mechanics.

Discretization of Fractional Fully Nonlinear Equations by Powers of Discrete Laplacians

AbstractWe study discretizations of fractional fully nonlinear equations by powers of discrete Laplacians. Our problems are parabolic and of order $$\sigma \in (0,2)$$ σ ∈ ( 0 , 2 ) since they involve fractional Laplace operators $$(-\Delta )^{\sigma /2}$$ ( - Δ ) σ / 2 . They arise e.g. in control and game theory as dynamic programming equations – HJB and Isaacs equation – and solutions are non-smooth in general and should be interpreted as viscosity solutions. Our approximations are realized as finite-difference quadrature approximations and are 2nd order accurate for all values of $$\sigma $$ σ . The accuracy of previous approximations of fractional fully nonlinear equations depend on $$\sigma $$ σ and are worse when $$\sigma $$ σ is close to 2. We show that the schemes are monotone, consistent, $$L^\infty $$ L ∞ -stable, and convergent using a priori estimates, viscosity solutions theory, and the method of half-relaxed limits. We also prove a second order error bound for smooth solutions and present many numerical examples.

Dynamic Programming-Based Approach to Model Antigen-Driven Immune Repertoire Synthesis

This paper presents a novel approach to modeling the repertoire of the immune system and its adaptation in response to the evolutionary dynamics of pathogens associated with their genetic variability. It is based on application of a dynamic programming-based framework to model the antigen-driven immune repertoire synthesis. The processes of formation of new receptor specificity of lymphocytes (the growth of their affinity during maturation) are described by an ordinary differential equation (ODE) with a piecewise-constant right-hand side. Optimal control synthesis is based on the solution of the Hamilton–Jacobi–Bellman equation implementing the dynamic programming approach for controlling Gaussian random processes generated by a stochastic differential equation (SDE) with the noise in the form of the Wiener process. The proposed description of the clonal repertoire of the immune system allows us to introduce an integral characteristic of the immune repertoire completeness or the integrative fitness of the whole immune system. The quantitative index for characterizing the immune system fitness is analytically derived using the Feynman–Kac–Kolmogorov equation.

Pathwise Stochastic Control and a Class of Stochastic Partial Differential Equations

AbstractIn this article, we study a stochastic optimal control problem in the pathwise sense, as initially proposed by Lions and Souganidis in [C. R. Acad. Sci. Paris Ser. I Math., 327 (1998), pp. 735-741]. The corresponding Hamilton-Jacobi-Bellman (HJB) equation, which turns out to be a non-adapted stochastic partial differential equation, is analyzed. Making use of the viscosity solution framework, we show that the value function of the optimal control problem is the unique solution of the HJB equation. When the optimal drift is defined, we provide its characterization. Finally, we describe the associated conserved quantities, namely the space-time transformations leaving our pathwise action invariant.

Linear State Optimal Control Problem with a Stochastic Switching Time

In this paper, we analyse an optimal control problem over a finite horizon with a stochastic switching time, assuming that the two optimal control problems present in its two stages have a particularly simple form called linear state. It is well known that linear state optimal control problems can be solved easily using the HJB equation approach and assuming that the value function is linear in the state. Unfortunately, this simplicity of solution does not extend to the problem with stochastic switching time. We prove that a necessary and sufficient condition for the problem to maintain a linear state structure is to assume that the hazard rate of the switching time depends only on the temporal variable. Finally, assuming that the hazard rate is constant, we completely characterise the solution of the obtained linear state optimal control problem.

Multiplayer hierarchical decision‐making for discrete‐time nonlinear networks of service via value iteration adaptive dynamic programming

AbstractIn this article, the multiplayer hierarchical decision‐making problem for discrete‐time nonlinear networks of service is studied from the perspective of Nash–Stackelberg–Nash games. A novel two‐level value iteration adaptive dynamic programming algorithm is developed to solve the coupled nonlinear Hamilton–Jacobi–Bellman equations associated with the game problem, which neither requires the system drift dynamics to be known as a priori nor requests the initial strategies to be admissible. Moreover, both the value function sequences and the strategy sequences generated by the algorithm converge to their theoretical optimality. An implementation framework for the algorithm is constructed by leveraging the regularized least‐squares method, and a convergence criterion that guarantees the admissibility of the approximated strategies is also proposed. The effectiveness of the proposed algorithm and implementation framework are finally demonstrated through two numerical simulation examples.

Environmental management and restoration under unified risk and uncertainty using robustified dynamic Orlicz risk

Environmental management and restoration should be designed such that the risk and uncertainty owing to nonlinear stochastic systems can be successfully addressed. We apply the robustified dynamic Orlicz risk to the modeling and analysis of environmental management and restoration to consider both the risk and uncertainty within a unified theory. We focus on the control of a jump-driven hybrid stochastic system that represents macrophyte dynamics. The dynamic programming equation based on the Orlicz risk is first obtained heuristically, from which the associated Hamilton–Jacobi–Bellman (HJB) equation is derived. In the proposed Orlicz risk, the risk aversion of the decision-maker is represented by a power coefficient that resembles a certainty equivalence, whereas the uncertainty aversion is represented by the Kullback–Leibler divergence, in which the risk and uncertainty are handled consistently and separately. The HJB equation includes a new state-dependent discount factor that arises from the uncertainty aversion, which leads to a unique, nonlinear, and nonlocal term. The link between the proposed and classical stochastic control problems is discussed with a focus on control-dependent discount rates. We propose a finite difference method for computing the HJB equation. Finally, the proposed model is applied to an optimal harvesting problem for macrophytes in a brackish lake that contains both growing and drifting populations.

Deep Neural Network Approximations for the Stable Manifolds of the Hamilton–Jacobi–Bellman Equations

Deep Neural Network Approximations for the Stable Manifolds of the Hamilton–Jacobi–Bellman Equations

Time-consistent asset allocation for risk measures in a Lévy market

Focusing on gains & losses relative to a risk-free benchmark instead of terminal wealth, we consider an asset allocation problem to maximize time-consistently a mean-risk reward function with a general risk measure which is (i) law-invariant, (ii) cash- or shift-invariant, and (iii) positively homogeneous, and possibly plugged into a general function. Examples include (relative) Value at Risk, coherent risk measures, variance, and generalized deviation risk measures. We model the market via a generalized version of the multi-dimensional Black–Scholes model using α-stable Lévy processes and give supplementary results for the classical Black–Scholes model. The optimal solution to this problem is a Nash subgame equilibrium given by the solution of an extended Hamilton–Jacobi–Bellman equation. Moreover, we show that the optimal solution is deterministic under appropriate assumptions.

Viscosity solutions to second order elliptic Hamilton-Jacobi–Bellman equations with infinite delay

Viscosity solutions to second order elliptic Hamilton-Jacobi–Bellman equations with infinite delay

Optimized distributed formation control using identifier–critic–actor reinforcement learning for a class of stochastic nonlinear multi-agent systems

This article is to propose an adaptive reinforcement learning (RL)-based optimized distributed formation control for the unknown stochastic nonlinear single-integrator dynamic multi-agent system (MAS). For solving the issue of unknown dynamic, an adaptive identifier neural network (NN) is developed to determine the stochastic MAS under expectation sense. And then, for deriving the optimized formation control, the RL is putted into effect via constructing a pair of critic and actor NNs. With regard of the traditional RL optimal controls, their algorithm exists the inherent complexity, because their adaptive RL algorithm are derived from negative gradient of the square of Hamilton–Jacobi–Bellman (HJB) equation. As a result, these methods are difficultly extended to stochastic dynamical systems. However, since this adaptive RL laws are derived from a simple positive function rather than the square of HJB equation, it can make optimal control with simple algorithm. Therefore, this optimized formation scheme can be smoothly performed to the stochastic MAS. Finally, according to theorem proof and computer simulation, the optimized method can realize the required control objective.

Unsupervised random quantum networks for PDEs

Classical Physics-informed neural networks (PINNs) approximate solutions to PDEs with the help of deep neural networks trained to satisfy the differential operator and the relevant boundary conditions. We revisit this idea in the quantum computing realm, using parameterised random quantum circuits as trial solutions. We further adapt recent PINN-based techniques to our quantum setting, in particular Gaussian smoothing. Our analysis concentrates on the Poisson, the Heat and the Hamilton–Jacobi–Bellman equations, which are ubiquitous in most areas of science. On the theoretical side, we develop a complexity analysis of this approach, and show numerically that random quantum networks can outperform more traditional quantum networks as well as random classical networks.

A risk based approach to the principal–agent problem

PurposeWe aim to generalize the continuous-time principal–agent problem to incorporate time-inconsistent utility functions, such as those of mean-variance type, which are prevalent in risk management and finance.Design/methodology/approachWe use recent advancements of the Pontryagin maximum principle for forward-backward stochastic differential equations (FBSDEs) to develop a method for characterizing optimal contracts in such models. This approach addresses the challenges posed by the non-applicability of the classical Hamilton–Jacobi–Bellman equation due to time inconsistency.FindingsWe provide a framework for deriving optimal contracts in the principal–agent problem under hidden action, specifically tailored for time-inconsistent utilities. This is illustrated through a fully solved example in the linear-quadratic setting, demonstrating the practical applicability of the method.Originality/valueThe work contributes to the existing literature by presenting a novel mathematical approach to a class of continuous time principal–agent problems, particularly under hidden action with time-inconsistent utilities, a scenario not previously addressed. The results offer potential insights for both theoretical development and practical applications in finance and economics.

Portfolio problem for the α−hypergeometric stochastic volatility model with consumption

This paper solves the finite horizon optimal consumption-investment problem for an investor that trades in the α−hypergeometric stochastic volatility model, with preferences based on a power utility function. To find the optimal strategy, we follow the dynamic programming approach. First, we perform a suitable change of variables in order to fully transform the non-linear Hamilton-Jacobi-Bellman (HJB) equation into a semilinear one. Next, we apply the Girsanov theorem to deduce an implicit Feynman-Kac formula depending upon the volatility process. The operator defined by the Feynman-Kac representation is shown to be a contraction operator on a designed function space. Finally, the Banach fixed point theorem yields the existence of a solution to the HJB equation in the designed space. Moreover, we apply a verification theorem to guarantee that the solution of the HJB equation coincides with the value function.

Adaptive neural inverse optimal control for uncertain switched nonlinear systems: an improved predefined-time method

For the first time, this paper introduces an improved predefined-time inverse optimal method to address the adaptive tracking control problem of a class of uncertain switched nonlinear systems. The proposed approach enables the system to achieve optimal performance without resorting to the HJB equation. Firstly, we design two-time constants to improve the traditional predefined-time lemma. The improved predefined-time lemma exhibits faster convergence. Subsequently, to realise the design of a predefined-time inverse optimal controller, a specific form of the auxiliary controller is designed using Sontag functions and nonsingular functions. Furthermore, the stability of the system and the optimality of the inverse are proven through the use of a common Lyapunov function method, ensuring that tracking errors converge to the neighbourhood of zero within the predefined time. Finally, the rationality of the proposed control structure is demonstrated through the example results.