Hamilton-Jacobi-Bellman Equation Research Articles

Optimal investment for asset–liability management with delay and partial information under Ornstein–Uhlenbeck process

In this paper, we investigate the optimal investment strategy of asset liability management (ALM) with bounded memory and partial information. Suppose that investors invest their assets in a financial market consisting of a risk-free bond and a risk-free stock, while also taking on liabilities, in which the value of liabilities and the price of risky assets satisfy the Ornstein–Uhlenbeck (O–U) processes whose drift terms are unobserved. By constructing a dynamic portfolio of risk-free bonds, risky stocks and liabilities, a stochastic delay differential equation is obtained to depict the surplus process of investor. The ALM problem is formulated as finding the best strategy to maximize the terminal utility of the sum of terminal surplus and some historical wealth under partial information, and the corresponding full information case is also studied as a supplement. For both cases of partial information and full information, we apply the dynamic programming method to derive HJB equations, verification theorems, and closed-form solutions of optimal strategies and value functions. Moreover the relationship between optimal strategy and value function under full information and partial information is also given. Finally, numerical examples are carried out to illustrate the influence of some important parameters on the obtained results.

A Preconditioned Policy–Krylov Subspace Method for Fractional Partial Integro-Differential HJB Equations in Finance

To better simulate the prices of underlying assets and improve the accuracy of pricing financial derivatives, an increasing number of new models are being proposed. Among them, the Lévy process with jumps has received increasing attention because of its capacity to model sudden movements in asset prices. This paper explores the Hamilton–Jacobi–Bellman (HJB) equation with a fractional derivative and an integro-differential operator, which arise in the valuation of American options and stock loans based on the Lévy-α-stable process with jumps model. We design a fast solution strategy that includes the policy iteration method, Krylov subspace method, and banded preconditioner, aiming to solve this equation rapidly. To solve the resulting HJB equation, a finite difference method including an upwind scheme, shifted Grünwald approximation, and trapezoidal method is developed with stability and convergence analysis. Then, an algorithmic framework involving the policy iteration method and the Krylov subspace method is employed. To improve the performance of the above solver, a banded preconditioner is proposed with condition number analysis. Finally, two examples, sugar option pricing and stock loan valuation, are provided to illustrate the effectiveness of the considered model and the efficiency of the proposed preconditioned policy–Krylov subspace method.

Model-Free Optimal Vibration Control of a Nonlinear System Based on Deep Reinforcement Learning

Optimal control of nonlinear vibration requires precise knowledge of the system and the solution to Hamilton–Jacobi–Bellman (HJB) equation. However, in practical engineering applications, acquiring precise system parameters poses challenges, and the analytical solutions for the HJB equation are difficult to obtain. In this paper, two reinforcement learning algorithms, Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm and Soft Actor-Critic (SAC) algorithm, are employed to train neural network-based optimal controllers for the van der Pol vibration system in the presence of unknown system parameters. To validate their performance, the controllers undergo testing in a series of experiments, including assessments of free vibration, frequency sweep excitation, and Gaussian noise excitation. The results indicate that both the TD3-trained and SAC-trained neural network-based controller are capable of proficiently suppress the vibration of the van der Pol oscillator. Additionally, these two model-free controllers can approximate the optimal control law which solved based on the dynamic model of the nonlinear system.

Relaxation and asymptotic expansion of controlled stiff differential equations

The control of relaxation-type systems of ordinary differential equations is investigated using the Hamilton–Jacobi–Bellman equation. Firstly, we recast the model as a singularly perturbed dynamics which we embed in a family of controlled systems. Then we study this dynamics together with the value function of the associated optimal control problem. We provide an asymptotic expansion in the relaxation parameter of the value function. We also show that its solution converges toward the solution of a Hamilton–Jacobi–Bellman equation for a reduced control problem. Such systems are motivated by semi-discretisation of kinetic and hyperbolic partial differential equations. Several examples are presented including Jin–Xin relaxation.

Optimal control of stochastic differential equations with random impulses and the Hamilton–Jacobi–Bellman equation

AbstractIn this article, we study the optimal control of stochastic differential equations with random impulses. We optimize the performance index and add the influence of random impulses to the performance index with a random compensation function. Using the idea of stochastic analysis and dynamic programming principle, a new Hamilton–Jacobi–Bellman (HJB) equation is obtained, and the existence and uniqueness of its viscosity solution are proved.

Reliable optimal controls for SEIR models in epidemiology

We present and compare two different optimal control approaches applied to SEIR models in epidemiology, which allow us to obtain some policies for controlling the spread of an epidemic. The first approach uses Dynamic Programming to characterise the value function of the problem as the solution of a partial differential equation, the Hamilton–Jacobi–Bellman equation, and derive the optimal policy in feedback form. The second is based on Pontryagin’s maximum principle and directly gives open-loop controls, via the solution of an optimality system of ordinary differential equations. This method, however, may not converge to the optimal solution. We propose a combination of the two methods in order to obtain high-quality and reliable solutions. Several simulations are presented and discussed, also checking first and second order necessary optimality conditions for the corresponding numerical solutions.

Neural network-based adaptive critic control for saturated nonlinear systems with full state constraints via a novel event-triggered mechanism

This paper investigates the issue of event-triggered optimal control for saturated nonlinear systems with full state constraints. A smooth function is defined to map the constrained states, then the considered system can be transformed into a state unconstrained system with unknown approximate errors. In addition, a novel event-triggered mechanism is proposed via an adaptive dynamic programming algorithm, which can obtain the optimal control law without constructing the so-called event-triggered HJB equation. Moreover, in order to conquer the difficulty caused by the persistence of excitation conditions, the experience replay technique is utilized to design the critic update law. Meanwhile, it is demonstrated that all signals are uniformly ultimately bounded. Finally, two simulation examples are given to demonstrate the effectiveness of the control strategy.

Scalable computation of energy functions for nonlinear balanced truncation

Nonlinear balanced truncation is a model order reduction technique that reduces the dimension of nonlinear systems in a manner that accounts for either open- or closed-loop observability and controllability aspects of the system. A computational challenges that has so far prevented its deployment on large-scale systems is that the energy functions required for characterization of controllability and observability are solutions of various high-dimensional Hamilton–Jacobi–(Bellman) equations, which are computationally intractable in high dimensions. This work proposes a unifying and scalable approach to this challenge by considering a Taylor-series-based approximation to solve a class of parametrized Hamilton–Jacobi–Bellman equations that are at the core of nonlinear balancing. The value of a formulation parameter provides either open-loop balancing or a variety of closed-loop balancing options. To solve for the coefficients of Taylor-series approximations to the energy functions, the presented method derives a linear tensor system and heavily utilizes it to numerically solve structured linear systems with billions of unknowns. The strength and scalability of the algorithm is demonstrated on two semi-discretized partial differential equations, namely the Burgers and the Kuramoto–Sivashinsky equations.

The Time-Consistent Optimal Reinsurance Strategy of Insurance Group under the CEV Model

The article introduces proportional reinsurance contracts under the mean-variance criterion, studying the time-consistence investment portfolio problem considering the interests of both insurance companies and reinsurance companies. The insurance claims process follows a jump-diffusion model, assuming that the risk asset prices of insurance companies and reinsurance companies follow CEV models different from each other. In the framework of game theory, the time-consistent equilibrium reinsurance strategy is obtained by solving the extended HJB equation analytically. Finally, numerical examples are used to illustrate the impact of model parameters on equilibrium strategies and provide economic explanations. The results indicate that the decision weights of insurance companies and reinsurance companies do have a significant impact on both the reinsurance ratio and the equilibrium reinsurance strategy.

An analytical adaptive optimal control approach without solving HJB equation for nonlinear systems with input constraints

AbstractThis paper presents an analytical method to solve the optimal control problem for affine nonlinear systems with unknown drift dynamics. A new non‐quadratic cost function over an infinite horizon is presented that considers input constraints and includes the cost of the feed‐forward component of the control law. The mean value theorem for vector‐valued functions has been used to derive an integral form of this theorem. Based on this theorem, a rigorous proof is provided demonstrating that the cost function can be converted into another form. In the presence of input constraints, this converted form enables extracting the optimal control solution without solving the HJB equation. Additionally, unknown nonlinearity effects in drift dynamics are compensated in the control input. This is accomplished by estimating the unknown drift dynamics via an adaptive neural network (NN) approach. It is proven that the states and weights of NN are uniformly ultimately bounded based on a Lyapunov technique. The necessary and sufficient conditions are provided that ensure the optimality of the infinite horizon optimal control problem with a discount factor. As a result, it is demonstrated that the proposed approach satisfies the optimality criteria. To evaluate the effectiveness of the proposed approach, simulation examples are provided.

A comparison principle for semilinear Hamilton–Jacobi–Bellman equations in the Wasserstein space

The goal of this paper is to prove a comparison principle for viscosity solutions of semilinear Hamilton–Jacobi equations in the space of probability measures. The method involves leveraging differentiability properties of the 2-Wasserstein distance in the doubling of variables argument, which is done by introducing a further entropy penalization that ensures that the relevant optima are achieved at positive, Lipschitz continuous densities with finite Fischer information. This allows to prove uniqueness and stability of viscosity solutions in the class of bounded Lipschitz continuous (with respect to the 1-Wasserstein distance) functions. The result does not appeal to a mean field control formulation of the equation, and, as such, applies to equations with nonconvex Hamiltonians and measure-dependent volatility. For convex Hamiltonians that derive from a potential, we prove that the value function associated with a suitable mean-field optimal control problem with nondegenerate idiosyncratic noise is indeed the unique viscosity solution.

Optimization for stochastic model arisen from investment problem associated with default risk

In this paper, we study the optimal strategy arisen from the excess-of-loss reinsurance and asset allocation in defaultable markets under a general stochastic model. By developing the corresponding dynamic programming approach, we establish the optimal investment approach through two sub-problems: a pre-default case and a post-default case, respectively, characterized by the obtained Hamilton–Jacobi–Bellman (HJB) equations. We show the existence of a classical solution to the pre-default case via super-sub solution techniques and give an explicit characterization of the optimal reinsurance and investment policy that maximizes the common used utility associated with the terminal wealth. Verification theorem is further established to show the uniqueness of the corresponding solution of HJB equation that is critical to the desired optimal solution.

Minimum loss optimization of flywheel energy storage systems via distributed adaptive dynamic programming

AbstractIn this article, a distributed controller based on adaptive dynamic programming is proposed to solve the minimum loss problem of flywheel energy storage systems (FESS). We first formulate a performance function aiming to reduce total losses of FESS in power distribution applications. Then we use the Hamilton–Jacobi–Bellman (HJB) equation to solve this optimal control problem. The solution of the HJB equation is approximated by neural networks. To achieve distributed control, we estimate the global variables in the HJB equation by using the dynamic average consensus algorithm. A barrier Lyapunov function and a saturation function are introduced to handle the issue of state and input constraints, respectively. Then the stability of the system is proved through the Lyapunov stability analysis. Finally the effectiveness of the proposed strategy is verified by simulations. Simulation results show that FESS can track the power command while minimizing total power losses by interacting with neighbors. The proposed algorithm leads to a loss reduction of compared to the equal power distribution strategy.

Viscosity solutions approach to finite-horizon continuous-time Markov decision process

This paper investigates the optimal control problems for the finite-horizon continuous-time Markov decision processes with delay-dependent control policies. We develop compactification methods in decision processes and show that the existence of optimal policies. Subsequently, through the dynamic programming principle of the delay-dependent control policies, the differential-difference Hamilton-Jacobi-Bellman (HJB) equation in the setting of discrete space is established. Under certain conditions, we give the comparison principle and further prove that the value function is the unique viscosity solution to this HJB equation. Based on this, we show that among the class of delay-dependent control policies, there is an optimal one which is Markovian.

Cobra aerobatic flight for quadrotors using θ-D nonlinear control

The control design for the Cobra maneuver of quadrotors is presented based on quaternion dynamics. Cobra maneuver commands the quadrotor in ± π / 2 (rad) pitch angle such that the conventional hovering assumption for quadrotor flight is not applicable. The finite-time θ-D method is proposed to control the Cobra maneuver. Based on this technique, the intractable Hamilton–Jacobi–Bellman equation can be approximated by a differential Riccati equation and a series of state-dependent Lyapunov equations, which can be solved recursively and lead to the closed-form state feedback control law . This control law can achieve semi-global asymptomatic stability of the closed-loop system. Numerical simulations verify that the proposed technique is compared favourably with the finite-time state-dependent Riccati equation (SDRE) technique in terms of computation efficiency and control effort, and also shows faster recovery of the quadrotor after the Cobra maneuver compared with conventional dynamics without the derivative of the rotation matrix.

A numerical method for ergodic optimal control of switching diffusions with reflection

The ergodic or long run average cost control problem for diffusions is one of the classical problems of stochastic control that still eludes a completely satisfactory treatment. This is certainly true for the setting in which the systems to be controlled is modeled by the solution of a switching stochastic differential equation with reflection (SSDER). In this paper, we advance in this rather difficult problem by setting forth preliminary results for the unidimensional case. Besides carving out a numerical method, our treatment of the ergodic control in this scenario straddles issues of existence and uniqueness of solution of the SSDER and a verification theorem for the associated HJB equation. We conclude by illustrating the effectiveness of the method considering the control of energy consumption in a large parallel processing computer system composed of one queue and several processing stations.

Data-Driven Stochastic Optimal Control With Safety Constraints Using Linear Transfer Operators

We provide a data-driven framework for optimal control of a continuous-time control-affine stochastic dynamical system. The proposed framework relies on the linear operator theory involving Perron-Frobenius (P-F) and Koopman operators. Our first result involving the P-F operator provides a convex formulation to the optimal control problem in the dual space of densities. This convex formulation of the stochastic optimal control problem leads to an infinite-dimensional convex program. The finite-dimensional approximation of the convex program is obtained using a data-driven approximation of the linear operators. We provide rigorous error bounds and convergence rate for the data-driven approximation of the linear operators and comment on the convergence rate of the optimal control w.r.t. data size. Our second results demonstrate using the Koopman operator, dual to the P-F operator, for the stochastic optimal control design. We show that the Hamilton Jacobi Bellman (HJB) equation can be expressed using the Koopman operator. We provide an iterative procedure along the lines of a popular policy iteration algorithm based on the data-driven approximation of the Koopman operator for solving the HJB equation. The two formulations, namely the convex formulation involving the P-F operator and the Koopman-based formulation using the HJB equation, can be viewed as dual to each other where the duality follows due to the dual nature of the P-F and Koopman operators. Finally, we present examples to demonstrate the efficacy of the developed framework and verify the convergence rates for the operator and optimal control numerically.

Adaptive Virotherapy Strategy for Organism With Constrained Input Using Medicine Dosage Regulation Mechanism.

In this article, the constrained adaptive control strategy based on virotherapy is investigated for organism using the medicine dosage regulation mechanism (MDRM). First, the tumor-virus-immune interaction dynamics is established to model the relations among the tumor cells (TCs), virus particles, and the immune response. The adaptive dynamic programming (ADP) method is extended to approximately obtain the optimal strategy for the interaction system to reduce the populations of TCs. Due to the consideration of asymmetric control constraints, the nonquadratic functions are proposed to formulate the value function such that the corresponding Hamilton-Jacobi-Bellman equation (HJBE) is derived which can be deemed as the cornerstone of ADP algorithms. Then, the ADP method of a single-critic network architecture which integrates MDRM is proposed to obtain the approximate solutions of HJBE and eventually derive the optimal strategy. The design of MDRM makes it possible for the dosage of the agentia containing oncolytic virus particles to be regulated timely and necessarily. Furthermore, the uniform ultimate boundedness of the system states and critic weight estimation errors is validated by Lyapunov stability analysis. Finally, simulation results are given to show the effectiveness of the derived therapeutic strategy.

Markov Chain Approximation for Hamilton–Jacobi–Bellman Equation with Absorbing Boundary

Markov Chain Approximation for Hamilton–Jacobi–Bellman Equation with Absorbing Boundary

Optimized leader–follower consensus control for high-order nonlinear multi-agent system modeled in canonical dynamic form

This paper addresses the optimized leader–follower consensus control problem of nonlinear canonical dynamic multi-agent system (MAS). Since every agent is modeled in high-order dynamic, they contain the various state variables having derivative relations. Consequently, to attain the optimized consensus control, backstepping technique and reinforcement learning (RL) are combined together. In the first backstepping step, the virtual control is formulated to have the consensus error term consisting of neighbor agents’ output states. In the last backstepping step, since the nonlinearity dynamic is arisen, the optimized actual control is derived by performing the critic-actor RL. It is worth mentioning that, in the traditional RL optimal control, the critic and actor updating laws are obtained by performing the gradient descent algorithm to square of the approximated Hamilton–Jacobi-Bellman (HJB) equation, which includes multiple nonlinearity items, as a result, the algorithm is very intricate. However, in this optimized consensus control, since the RL training laws are yielded in the light of negative gradient of a simple positive function that is relevant to the HJB equation, its algorithm is very simple. At the same time, it can also avoid the persistence excitation conditions. Finally, theory and simulation confirm the validity of this optimized consensus method.