Hamilton-Jacobi-Bellman Equation Research Articles

Optimal consumption with reference to past spending maximum

This paper studies the infinite-horizon optimal consumption problem with a path-dependent reference under exponential utility. The performance is measured by the difference between the nonnegative consumption rate and a fraction of the historical consumption maximum. The consumption running maximum process is chosen as an auxiliary state process, and hence the value function depends on two state variables. The Hamilton–Jacobi–Bellman (HJB) equation can be heuristically expressed in a piecewise manner across different regions to take into account all constraints. By employing the dual transform and smooth-fit principle, some thresholds of the wealth variable are derived such that a classical solution to the HJB equation and feedback optimal investment and consumption strategies can be obtained in closed form in each region. A complete proof of the verification theorem is provided, and numerical examples are presented to illustrate some financial implications.

On Regularized Optimal Execution Problems and Their Singular Limits

We investigate the portfolio execution problem under a framework in which volatility and liquidity are both uncertain. In our model, we assume that a multidimensional Markovian stochastic factor drives both of them. Moreover, we model indirect liquidity costs as temporary price impact, stipulating a power law to relate it to the agent's turnover rate. We first analyse the regularized setting, in which the admissible strategies do not ensure complete execution of the initial inventory. We prove the existence and uniqueness of a continuous and bounded viscosity solution of the Hamilton–Jacobi–Bellman equation, whence we obtain a characterization of the optimal trading rate. As a byproduct of our proof, we obtain a numerical algorithm. Then, we analyse the constrained problem, in which admissible strategies must guarantee complete execution to the trader. We solve it through a monotonicity argument, obtaining the optimal strategy as a singular limit of the regularized counterparts.

A Stochastic Model of Economic Growth in Time-Space

We deal with an infinite horizon, infinite dimensional stochastic optimal control problem arising in the study of economic growth in time-space. Such a problem has been the object of various papers in deterministic cases when the possible presence of stochastic disturbances is ignored (see, e.g., [P. Brito, The Dynamics of Growth and Distribution in a Spatially Heterogeneous World, working paper 2004/14, ISEG-Lisbon School of Economics and Management, University of Lisbon, 2004], [R. Boucekkine, C. Camacho, and G. Fabbri, J. Econom. Theory, 148 (2013), pp. 2719--2736], [G. Fabbri, J. Econom. Theory, 162 (2016), pp. 114--136], and [R. Boucekkine, G. Fabbri, S. Federico, and F. Gozzi, J. Econom. Geography, 19 (2019), pp. 1287--1318]). Here we propose and solve a stochastic generalization of such models where the stochastic term, in line with the standard stochastic economic growth models (see, e.g., the books [A. G. Malliaris and W. A. Brock, Stochastic Methods in Economics and Finance, Advanced Textbooks in Economics 17, North Holland, 1982, Chapter 3] and [H. Morimoto, Stochastic Control and Mathematical Modeling: Applications in Economics, Cambridge Books, 2010, Chapter 9]), is a multiplicative one, driven by a cylindrical Wiener process. The problem is studied using the dynamic programming approach. We find an explicit solution of the associated HJB equation, use a verification type result to prove that such a solution is the value function, and find the optimal feedback strategies. Finally, we use this result to study the asymptotic behavior of the optimal trajectories.

Optimal Synchronization Control of Heterogeneous Asymmetric Input-Constrained Unknown Nonlinear MASs via Reinforcement Learning

The asymmetric input-constrained optimal synchronization problem of heterogeneous unknown nonlinear multiagent systems (MASs) is considered in the paper. Intuitively, a state-space transformation is performed such that satisfaction of symmetric input constraints for the transformed system guarantees satisfaction of asymmetric input constraints for the original system. Then, considering that the leader's information is not available to every follower, a novel distributed observer is designed to estimate the leader's state using only exchange of information among neighboring followers. After that, a network of augmented systems is constructed by combining observers and followers dynamics. A nonquadratic cost function is then leveraged for each augmented system (agent) for which its optimization satisfies input constraints and its corresponding constrained Hamilton-Jacobi-Bellman (HJB) equation is solved in a data-based fashion. More specifically, a data-based off-policy reinforcement learning (RL) algorithm is presented to learn the solution to the constrained HJB equation without requiring the complete knowledge of the agents' dynamics. Convergence of the improved RL algorithm to the solution to the constrained HJB equation is also demonstrated. Finally, the correctness and validity of the theoretical results are demonstrated by a simulation example.

Adaptive output‐feedback control for a class of nonlinear systems based on optimized backstepping technique

SummaryIn this article, an adaptive output‐feedback control for a class of strict‐feedback nonlinear systems is developed based on optimized backstepping technique. Neural networks are utilized to approximate unknown functions, while a state observer is designed to estimate the unmeasurable system state signals. Since the presented optimized control scheme requires training the adaptive parameters for reinforcement learning (RL), it will be more challenging for designing control algorithms and deriving the adaptive update rates. In general, optimization control is designed based on the solution of Hamilton–Jacobi–Bellman equation, but solving the equation is very difficult due to the inherent nonlinearity and intractability. So, RL strategy of actor‐critic architecture is used. According to the Lyapunov stability theory, it is proved that all signals of the closed‐loop systems are semi‐global uniformly ultimately bounded. Finally, the results of the simulation cases are provided to show the effectiveness of the designed controller scheme.

Crandall–Lions viscosity solutions for path-dependent PDEs: The case of heat equation

We address our interest to the development of a theory of viscosity solutions à la Crandall–Lions for path-dependent partial differential equations (PDEs), namely PDEs in the space of continuous paths C([0,T];Rd). Path-dependent PDEs can play a central role in the study of certain classes of optimal control problems, as for instance optimal control problems with delay. Typically, they do not admit a smooth solution satisfying the corresponding HJB equation in a classical sense, it is therefore natural to search for a weaker notion of solution. While other notions of generalized solution have been proposed in the literature, the extension of the Crandall–Lions framework to the path-dependent setting is still an open problem. The question of uniqueness of the solutions, which is the most delicate issue, will be based on early ideas from the theory of viscosity solutions and a suitable variant of Ekeland’s variational principle. This latter is based on the construction of a smooth gauge-type function, where smooth is meant in the horizontal/vertical (rather than Fréchet) sense. In order to make the presentation more readable, we address the path-dependent heat equation, which in particular simplifies the smoothing of its natural “candidate” solution. Finally, concerning the existence part, we provide a functional Itô formula under general assumptions, extending earlier results in the literature.

Singular Limit of Two-Scale Stochastic Optimal Control Problems in Infinite Dimensions by Vanishing Noise Regularization

In this paper we study the limit of the value function for a two-scale, infinite-dimensional, stochastic controlled system with cylindrical noise and possibly degenerate diffusion. The limit is represented as the value function of a new reduced control problem (on a reduced state space). The presence of a cylindrical noise prevents representation of the limit by viscosity solutions of HJB equations, while degeneracy of diffusion coefficients prevents representation as a classical BSDE. We use a vanishing noise regularization technique.

Model‐free incremental adaptive dynamic programming based approximate robust optimal regulation

AbstractThis article presents a new formulation for model‐free robust optimal regulation of continuous‐time nonlinear systems. The proposed reinforcement learning based approach, referred to as incremental adaptive dynamic programming (IADP), utilizes measured input‐state data to allow the design of the approximate optimal incremental control strategy, stabilizing the controlled system incrementally under model uncertainties, environmental disturbances, and input saturation. By leveraging the time delay estimation (TDE) technique, we first use sensor data to reduce the requirement of a complete dynamics, where input‐state data is adopted to construct an incremental dynamics which reflects the system evolution in an incremental form. Then, the resulting incremental dynamics serves to design the approximate optimal incremental control strategy based on adaptive dynamic programming, which is implemented as a simplified single critic structure to get the approximate solution to the value function of the Hamilton–Jacobi–Bellman equation. Furthermore, for the critic neural network, experience data are used to design an off‐policy weight update law with guaranteed weight convergence. Rather importantly, we incorporate a TDE error bound related term into the cost function, whereby the unintentionally introduced TDE error is attenuated during the optimization process. The proofs of system stability and weight convergence are provided. Numerical simulations are conducted to validate the effectiveness and superiority of our proposed IADP, especially regarding the reduced control energy expenditure and the enhanced robustness.

A data-based neural policy learning strategy towards robust tracking control design for uncertain dynamic systems

In this paper, a data-based neural policy learning method is established to solve the robust tracking control problem of a class of continuous-time systems which have two kinds of uncertainties at the same time. First, the robust trajectory tracking is achieved by controlling the tracking error to zero. The specific implementation strategy is to construct an augmented system including the tracking error and then transform the robust tracking control problem into an optimal control problem by selecting a suitable cost function. Then, a neural network identifier is built to reconstruct the unknown dynamics and a policy iteration algorithm is adopted by using a critic neural network. In this way, the Hamilton–Jacobi–Bellman equation can be solved. Through this learning algorithm, the approximate optimal control policy is obtained and the solution of the robust tracking control problem can be derived. Finally, two simulation examples are proposed to verify the effectiveness of the developed method.

Asymptotic Behavior of an Optimal Investment‐Reinsurance Problem with General Utility Functions

It is usually extremely difficult to derive an analytical solution to the Hamilton‐Jacobi‐Bellman (HJB) equation for some optimal control problems under general utility functions, while this paper provides a dual control method to solve the HJB equation for an optimal investment‐reinsurance problem with general utility functions. We first change the HJB equation into its dual one. By solving the solution to the dual HJB equation, we derive a classical solution to the primal HJB equation and obtain the expression of the optimal investment‐reinsurance strategy, then some examples, which would be difficult to solve with the standard method, are given to demonstrate the usefulness of our dual control method. Moreover, applying the dual method along with the partial differential equation method, we verify the asymptotic behavior of the optimal control strategy, which means that the optimal investment‐reinsurance strategy with the general utility functions converges to that with the power utility function at any level of the initial wealth.

Adaptive Event-Triggered Near-Optimal Tracking Control for Unknown Continuous-Time Nonlinear Systems

This paper studies the event-triggered optimal tracking control (ETOTC) problem of continuous-time (CT) unknown nonlinear systems. In order to solve the ETOTC problem, an augmented system composed of the error system dynamics and the reference dynamics is used to introduce a new discounted performance index function (DPIF). A novel event-triggered (ET) adaptive dynamic programming (ADP) method is developed to solve the ET Hamilton-Jacobi-Bellman equation (HJBE). The presented method is implemented via an identifier-critic architecture, which consists of two neural networks (NNs): an identifier NN is applied to estimate the unknown system dynamics, and a critic NN is constructed to obtain the approximate solution of the ET HJBE. The augmented closed-loop system and the critic estimation error are proved to be ultimately uniformly bounded (UUB) by the Lyapunov direct method. Finally, two simulations illustrate the effectiveness of the developed method.

Random information horizon for a class of differential games with continuous updating

In the paper we consider a class of the differential games with continuous updating with random information horizon. It is assumed that at each time instant, players have information about the game (motion equations and payoff functions) for a time interval with the length theta and as the time evolves information about the game updates. We first considered this type of games in 2019. Here we additionally assume that theta is a random variable. The subject of the current paper is definition of Nash equilibrium based solution concept and solution technique based on Hamilton—Jacobi—Bellman equations.

Reducing the Possibility of Ruin by Maximizing the Survival Function for the Insurance Company’s Portfolio

In this paper, the intention was to reduce the possibility of ruin in the insurance company by maximizing its survival function. This paper uses a perturbed classical risk process as the basic model. The basic model was later compounded by refinancing and return on investment. The Hamilton–Jacobi–Bellman equation and integro‐differential equation of Volterra type were obtained. The Volterra integro‐differential equation for the survival function of the insurance company was converted to a third‐order ordinary differential equation which was later converted into a system of first‐order ordinary differential equations. This system was then solved numerically using the fourth‐order Runge‐Kutta method. The results show that the survival function increases with the increase in the intensity of the counting process but decreases with an increase in the instantaneous rate of stock return and return volatility. This is due to the fact that the insurance company faces more risk. Thus, this paper suggests that in this situation, more investments should be made in risk‐free assets.

Optimal investment-reinsurance strategy with derivatives trading under the joint interests of an insurer and a reinsurer

<abstract> <p>Considering the common interests of an insurer and a reinsurer, the optimal investment-reinsurance problem with derivatives trading is studied. Suppose that both parties would invest a stock and a risk-free asset for capital appreciation, the insurer could purchase reinsurance and trade derivatives, the optimization problem is formulated by maximizing the expected exponential utility of two parties' wealth processes. The corresponding HJB equations are built for optimal strategy through the dynamic programming principle. In addition, derivatives trading is evaluated based on the certainty-equivalence principle. A numerical study directly illustrates how model parameters influence optimal strategies.</p> </abstract>

An approach to solving optimal control problems of nonlinear systems by introducing detail-reward mechanism in deep reinforcement learning

In recent years, dynamic programming and reinforcement learning theory have been widely used to solve the nonlinear control system (NCS). Among them, many achievements have been made in the construction of network model and system stability analysis, but there is little research on establishing control strategy based on the detailed requirements of control process. Spurred by this trend, this paper proposes a detail-reward mechanism (DRM) by constructing the reward function composed of the individual detail evaluation functions in order to replace the utility function in the Hamilton-Jacobi-Bellman (HJB) equation. And this method is introduced into a wider range of deep reinforcement learning algorithms to solve optimization problems in NCS. After the mathematical description of the relevant characteristics of NCS, the stability of iterative control law is proved by Lyapunov function. With the inverted pendulum system as the experiment object, the dynamic environment is designed and the reward function is established by using the DRM. Finally, three deep reinforcement learning algorithm models are designed in the dynamic environment, which are based on Deep Q-Networks, policy gradient and actor-critic. The effects of different reward functions on the experimental accuracy are compared. The experimental results show that in NCS, using the DRM to replace the utility function in the HJB equation is more in line with the detailed requirements of the designer for the whole control process. By observing the characteristics of the system, designing the reward function and selecting the appropriate deep reinforcement learning algorithm model, the optimization problem of NCS can be solved.

On Durable-strategies Dynamic Game Theory

An empirically meaningful theory of dynamic games has to incorporate real-life phenomena. Durable strategies, which effects last for a period of time, are prevalent in real-life situations. Revenue generating investments, toxic waste disposal and purchase of durable goods are common examples of durable strategies. This paper first provides a review on durable-strategies dynamic game theory. A practically relevant advancement - random horizon durable-strategies dynamic games - yielding novel results in durable-strategies dynamic games theory is then presented. Dynamic optimization theorem, game formulations and HJB equations are derived. An illustrative example is provided. The theory and solution mechanism of durable-strategies cooperative dynamic games are also discussed.

Optimal Investment‐Consumption Strategy of Household Based on CEV Model

To maintain and increase household wealth, this study studies the optimal allocation ratio of household investment and consumption. When considering venture capital, it is assumed that the theoretical price of risky asset obeys the CEV model. Our goal was to maximize the expectation of household cumulative consumption and the discounted utility of terminal wealth and to solve the optimal consumption and investment ratio using the dynamic programming principle and HJB equation. Using logarithmic utility and isoelastic power utility function with residual utility, we get the analytical solution of the household investment‐consumption ratio by means of guessing and variable transformation. Finally, the influence of general parameters on the optimal ratio in the market is analyzed by numerical simulation and diagram, which is consistent with the description of actual situation. This study not only enriches portfolio theory but also provides investors with investment strategies.

Data-Driven Adaptive Dynamic Programming for Optimal Control of Continuous-Time Multicontroller Systems With Unknown Dynamics

This paper investigates the optimal control of continuous-time multi-controller systems with completely unknown dynamics using data-driven adaptive dynamic programming (DD-ADP). In this investigation, all controllers take actions together as a team, and they have precisely the same cost function, which is actually a fully cooperative game. According to optimal control theory, the HJB equation corresponding to the fully cooperative game is derived. To obtain the solution to HJB equation, a model-based policy iteration (PI) algorithm is first presented. On the basis of the PI algorithm, a DD-ADP algorithm without requiring the system dynamics is developed, and the neural networks (NNs) implementation scheme of the developed DD-ADP algorithm is given. Stability and convergence analysis are derived by Lyapunov theory. Finally, numerical simulation examples on linear and nonlinear multi-controller systems demonstrate the effectiveness of the designed scheme.

Robust Trajectory Tracking of Uncertain Systems via Adaptive Critic Learning

This study develops an adaptive dynamic programming (ADP) scheme for uncertain systems to achieve the robust trajectory tracking. In this framework, the augmented state is first established via combining the tracking error and reference trajectory, where the robust tracking control problem can be resolved using the regulation control strategy. Then, the robust control problem of uncertain system can be represented as an optimal control problem of nominal system, which provides a new pathway to address the robust control problem. To realize the optimal control, the derived Hamilton–Jacobi–Bellman equation (HJBE) is solved by training a critic neural network (CNN). Finally, two innovative critic learning techniques are suggested to calculate the unknown NN weights, where the convergence of NN weights can be guaranteed. Simulations are carried out to demonstrate the effectiveness of the proposed method.

Dynamic programming principle and Hamilton-Jacobi-Bellman equation under nonlinear expectation

In this paper, we study a stochastic recursive optimal control problem in which the value functional is defined by the solution of a backward stochastic differential equation (BSDE) under G-expectation. Under standard assumptions, we establish the comparison theorem for this kind of BSDE and give a novel and simple method to obtain the dynamic programming principle. Finally, we prove that the value function is the unique viscosity solution to a type of fully nonlinear HJB equation.