Solution Of Hamilton-Jacobi-Bellman Equation Research Articles

Optimization for stochastic model arisen from investment problem associated with default risk

In this paper, we study the optimal strategy arisen from the excess-of-loss reinsurance and asset allocation in defaultable markets under a general stochastic model. By developing the corresponding dynamic programming approach, we establish the optimal investment approach through two sub-problems: a pre-default case and a post-default case, respectively, characterized by the obtained Hamilton–Jacobi–Bellman (HJB) equations. We show the existence of a classical solution to the pre-default case via super-sub solution techniques and give an explicit characterization of the optimal reinsurance and investment policy that maximizes the common used utility associated with the terminal wealth. Verification theorem is further established to show the uniqueness of the corresponding solution of HJB equation that is critical to the desired optimal solution.

Event-triggered optimal control for nonlinear stochastic systems via adaptive dynamic programming

For nonlinear Itô-type stochastic systems, the problem of event-triggered optimal control (ETOC) is studied in this paper, and the adaptive dynamic programming (ADP) approach is explored to implement it. The value function of the Hamilton–Jacobi–Bellman(HJB) equation is approximated by applying critical neural network (CNN). Moreover, a new event-triggering scheme is proposed, which can be used to design ETOC directly via the solution of HJB equation. By utilizing the Lyapunov direct method, it can be proved that the ETOC based on ADP approach can ensure that the CNN weight errors and states of system are semi-globally uniformly ultimately bounded in probability. Furthermore, an upper bound is given on predetermined cost function. Specifically, there has been no published literature on the ETOC for nonlinear Itô-type stochastic systems via the ADP method. This work is the first attempt to fill the gap in this subject. Finally, the effectiveness of the proposed method is illustrated through two numerical examples.

Optimal Control of Investment-Reinsurance Problem for an Insurer with Jump-Diffusion Risk Process: Independence of Brownian Motions

This paper investigates the excess-of-loss reinsurance and investment problem for a compound Poisson jump-diffusion risk process, with the risk asset price modeled by a constant elasticity of variance (CEV) model. It aims at obtaining the explicit optimal control strategy and the optimal value function. Applying stochastic control technique of jump diffusion, a Hamilton-Jacobi-Bellman (HJB) equation is established. Moreover, we show that a closed-form solution for the HJB equation can be found by maximizing the insurer’s exponential utility of terminal wealth with the independence of two Brownian motionsW(t)andW1(t). A verification theorem is also proved to verify that the solution of HJB equation is indeed a solution of this optimal control problem. Then, we quantitatively analyze the effect of different parameter impacts on optimal control strategy and the optimal value function, which show that optimal control strategy is decreasing with the initial wealthxand decreasing with the volatility rate of risk asset price. However, the optimal value functionV(t;x;s)is increasing with the appreciation rateμof risk asset.

Mean-Variance Portfolio Selection with Margin Requirements

We study the continuous-time mean-variance portfolio selection problem in the situation when investors must pay margin for short selling. The problem is essentially a nonlinear stochastic optimal control problem because the coefficients of positive and negative parts of control variables are different. We can not apply the results of stochastic linearquadratic (LQ) problem. Also the solution of corresponding Hamilton-Jacobi-Bellman (HJB) equation is not smooth. Li et al. (2002) studied the case when short selling is prohibited; therefore they only need to consider the positive part of control variables, whereas we need to handle both the positive part and the negative part of control variables. The main difficulty is that the positive part and the negative part are not independent. The previous results are not directly applicable. By decomposing the problem into several subproblems we figure out the solutions of HJB equation in two disjoint regions and then prove it is the viscosity solution of HJB equation. Finally we formulate solution of optimal portfolio and the efficient frontier. We also present two examples showing how different margin rates affect the optimal solutions and the efficient frontier.

On the Hamilton--Jacobi--Bellman Equation for an Optimal Consumption Problem: II. Verification Theorem

We consider an optimal consumption problem where an investor tries to maximize the infinite horizon expected discounted hyperbolic absolute risk aversion utility of consumption. We treat a stochastic factor model such that the mean returns of risky assets depend on underlying economic factors formulated as the solution of a stochastic differential equation. Using a dynamic programming principle, we derive the Hamilton--Jacobi--Bellman (HJB) equation and study its solutions. In part I, we prove the existence of a classical solution for HJB equation under suitable conditions. In part II, we consider the verification theorem stating a candidate of optimal strategy derived from a solution of HJB equation is indeed optimal. We prove the result under two sets of conditions. One is a linear model, and one is a nonlinear model. In both cases, the HJB equations do not have analytical solutions.

Static Solutions of the Hamilton-Jacobi-BellmanEquation for Circular Orbit Transfers

A LYAPUNOV control law was proposed by Chang et al. [1] for transfer between Keplerian orbits. For circular orbits, a Lyapunov function was chosen based on the energy E and the angularmomentumL, and a control lawwas derived in order tomake the derivative of the Lyapunov function negative-definite. However, the optimality of the control law was not analyzed. In this Note, the optimality of these transfers is shown by deriving a cost function for which these transfers are optimal. In particular, the link between the Lyapunov approach and the static solution of the Hamilton–Jacobi– Bellman equation (HJBE) for the two-body orbit transfer is shown. Moreover, it is shown that for long times of flight, constantpropulsive-acceleration transfers can be designed in feedback form with almost the same fuel cost in comparison with fuel-optimal transfers. Bonnard et al. [2] considered the geometric structure of timeoptimal control for spacecraft transfer in elliptic Keplerian orbits. A time-optimal control problem was proposed for constant specific impulse engines and the controllability of the system was analyzed. The methodology to compute the optimal control was based on the traditional indirect method, which was solved numerically by means of a shooting method. Freeman and Kokotovic [3] introduced the concept of a robust control Lyapunov function (rclf) and showed that every rclf satisfies the steady-state Hamilton–Jacobi–Isaacs equation associated with a meaningful game. Indeed, having a control Lyapunov function, it can be used for determination a stabilizing feedback control that is optimal in some sense. This technique is known as inverse optimality method. As an example, Bharadwaj et al. [4] used this method to design globally stabilizing attitude control laws. Primbs et al. [5] investigated the relation between optimal control problems and control Lyapunov functions from a receding-horizon perspective. In that work, it was shown that a broad class of Lyapunov controllers admit natural extensions to recedinghorizon formulations, which are locally optimal. However, these ideas were of a general character and applied only to a simple nonlinear system. In this Note, these ideas are applied specifically for mediumto low-thrust spacecraft orbital transfer in a two-body setting. In particular, a closed-form solution of an infinite-horizon HJBE for two-body, circular-orbit transfers is derived and is compared with fuel-optimal finite time transfers. The solution of the HJBE provides the optimal control at every point in phase space in feedback form. However, the nonlinearity of this partial differential equation (PDE)makes it extremely difficult to solve analytically. For linear systems subject to quadratic cost functions, it is possible to obtain an analytic form for the optimal cost [6], but for general nonlinear systems that are of practical interest, this is not the case in general. The numerical approach to solve this PDE is still challenging, as the solutions are in general not smooth and differentiable at every point. However, the great drawback of dynamic programming is, as Bellman himself calls it, “the curse of dimensionality.” Even recording the solution to a moderately sized problem involves an enormous amount of storage. If only an optimal path from a known initial point is desired, it is wasteful and tedious to find a whole field of extremals [7]. After a review of the main result obtained by Chang et al. [1], the static form of the HJBE is described and the optimality of the Lyapunov control law for a regulized fuel-optimal cost function is shown. The application of this control law is illustrated with a threedimensional transfer from a low Earth orbit (LEO) to the geostationary Earth orbit (GEO). The results are shown to be applicable to both variable specific impulse (VSI) and engines with constantpropulsive-acceleration capability. The examples illustrate the relation between the (infinite-horizon) transfers and actual finite time, fuel-optimal solutions. Finally, the application of thismethod to provide good initial guesses of the costates (relevant to the indirect formulation and solution of the optimal control problem) is explored.

Robust control of nonlinear systems using neural network based HJB solution

In this paper, a Hamilton-Jacobi-Bellman (HJB) equation based optimal control algorithm for robust controller design is proposed for a nonlinear system. Utilising the Lyapunov direct method, the controller is shown to be optimal with respect to a cost functional, which includes penalty on the control effort, the maximum bound on system uncertainty and cross-coupling between system state and control. The controllers are continuous and require the knowledge of the upper bound of system uncertainty. In the present algorithm, neural network is used to approximate value function to find approximate solution of HJB equation using least squares method. Proposed algorithm has been applied on a nonlinear system with matched uncertainties. It is also applied to the system having uncertainties in input matrix. Results are validated through simulation studies.