Generalized Hamilton-Jacobi-Bellman Equation Research Articles

Sparse Approximation–Based Collocation Scheme for Nonlinear Optimal Feedback Control Design

This paper discusses the development of a computationally efficient approach to generate optimal feedback control laws for infinite time problems by solving the corresponding Hamilton–Jacobi–Bellman (HJB) equation. The solution process consists of iteratively solving the linear generalized HJB (GHJB) equation starting with an admissible stable controller. The collocation methods are exploited to solve the GHJB equation in the specified domain of interest. Recently developed nonproduct quadrature method known as Conjugate Unscented Transformation is used to manage the curse of dimensionality associated with the growth of collocation points with increase in the state dimension. Furthermore, recent advances in sparse approximation are leveraged to automatically generate the appropriate polynomial basis function set for the collocation-based approximation from an overcomplete dictionary of basis functions. It is demonstrated that the solution process uses the basis function selection process to automatically identify a form for the feedback control law, which is frequently unknown. Several numerical examples demonstrate the efficacy of the proposed approach in accurately generating feedback control policies for nonlinear systems.

Optimal control of markovian switching systems with applications to portfolio decisions under inflation

This article is concerned with a class of control systems with Markovian switching, in which an Itô formula for Markov-modulated processes is derived. Moreover, an optimal control law satisfying the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching is characterized. Then, through the generalized HJB equation, we study an optimal consumption and portfolio problem with the financial markets of Markovian switching and inflation. Thus, we deduce the optimal policies and show that a modified Mutual Fund Theorem consisting of three funds holds. Finally, for the CRRA utility function, we explicitly give the optimal consumption and portfolio policies. Numerical examples are included to illustrate the obtained results.

Stochastic recursive optimal control problem with time delay and applications

This paper is concerned with a stochastic recursive optimal control problem with time delay, where the controlled system is described by a stochastic differential delayed equation (SDDE) and the cost functional is formulated as the solution to a backward SDDE (BSDDE). When there are only the pointwise and distributed time delays in the state variable, a generalized Hamilton-Jacobi-Bellman (HJB) equation for the value function in finite dimensional space is obtained, applying dynamic programming principle. This generalized HJB equation admits a smooth solution when the coefficients satisfy a particular system of first order partial differential equations (PDEs). A sufficient maximum principle is derived, where the adjoint equation is a forward-backward SDDE (FBSDDE). Under some differentiability assumptions, the relationship between the value function, the adjoint processes and the generalized Hamiltonian function is obtained. A consumption and portfolio optimization problem with recursive utility in the financial market, is discussed to show the applications of our result. Explicit solutions in a finite dimensional space derived by the two different approaches, coincide.

Optimal Control of Uncertain Stochastic Systems with Markovian Switching and Its Applications to Portfolio Decisions

This article first describes a class of uncertain stochastic control systems with Markovian switching, and derives an Itô-Liu formula for Markov-modulated processes. We characterize an optimal control law, that satisfies the generalized Hamilton-Jacobi-Bellman (HJB) equation with Markovian switching. Then, by using the generalized HJB equation, we deduce the optimal consumption and portfolio policies under uncertain stochastic financial markets with Markovian switching. Finally, for constant relative risk-aversion (CRRA) felicity functions, we explicitly obtain the optimal consumption and portfolio policies. Moreover, we make an economic analysis through numerical examples.

Optimal control of Markovian jump processes with partial information and applications to a parallel queueing model

We consider a stochastic control problem over an infinite horizon where the state process is influenced by an unobservable environment process. In particular, the Hidden-Markov-model and the Bayesian model are included. This model under partial information is transformed into an equivalent one with complete information by using the well-known filter technique. In particular, the optimal controls and the value functions of the original and the transformed problem are the same. An explicit representation of the filter process which is a piecewise-deterministic process, is also given. Then we propose two solution techniques for the transformed model. First, a generalized verification technique (with a generalized Hamilton–Jacobi–Bellman equation) is formulated where the strict differentiability of the value function is weaken to local Lipschitz continuity. Second, we present a discrete-time Markovian decision model by which we are able to compute an optimal control of our given problem. In this context we are also able to state a general existence result for optimal controls. The power of both solution techniques is finally demonstrated for a parallel queueing model with unknown service rates. In particular, the filter process is discussed in detail, the value function is explicitly computed and the optimal control is completely characterized in the symmetric case.

Generalized Hamilton–Jacobi–Bellman Formulation -Based Neural Network Control of Affine Nonlinear Discrete-Time Systems

In this paper, we consider the use of nonlinear networks towards obtaining nearly optimal solutions to the control of nonlinear discrete-time (DT) systems. The method is based on least squares successive approximation solution of the generalized Hamilton-Jacobi-Bellman (GHJB) equation which appears in optimization problems. Successive approximation using the GHJB has not been applied for nonlinear DT systems. The proposed recursive method solves the GHJB equation in DT on a well-defined region of attraction. The definition of GHJB, pre-Hamiltonian function, HJB equation, and method of updating the control function for the affine nonlinear DT systems under small perturbation assumption are proposed. A neural network (NN) is used to approximate the GHJB solution. It is shown that the result is a closed-loop control based on an NN that has been tuned a priori in offline mode. Numerical examples show that, for the linear DT system, the updated control laws will converge to the optimal control, and for nonlinear DT systems, the updated control laws will converge to the suboptimal control.

Galerkin approximations of the generalized Hamilton-Jacobi-Bellman equation

In this paper we study the convergence of the Galerkin approximation method applied to the generalized Hamilton-Jacobi-Bellman (GHJB) equation over a compact set containing the origin. The GHJB equation gives the cost of an arbitrary control law and can be used to improve the performance of this control. The GHJB equation can also be used to successively approximate the Hamilton-Jacobi-Bellman equation. We state sufficient conditions that guarantee that the Galerkin approximation converges to the solution of the GHJB equation and that the resulting approximate control is stabilizing on the same region as the initial control. The method is demonstrated on a simple nonlinear system and is compared to a result obtained by using exact feedback linearization in conjunction with the LQR design method.

Galerkin Approximation of the Generalized Hamilton-Jacobi Equation

If u is a stabilizing control for a nonlinear system that is affine in the control variable, then the solution to the Generalized Hamilton-Jacobi-Bellman (GHJB) equation associated with u is a Lyapunov function for the system and equals the cost associated with u. If an explicit solution to the GHJB equation can be found then it can be used to construct a feedback control law that improves the performance of u. Repeating this process leads to a successive approximation algorithm that uniformly approximates the Hamilton-Jacobi-Bellman equation. The difficulty is that it is very difficult to construct solutions to the GHJB equation such that the control derived from its solution is in feedback form. This paper shows that Galerkin's approximation method can be used to construct arbitrarily close approximations to the GHJB equation while generating stable feedback control laws. We state sufficient conditions for the convergence of Galerkin approximations to the GHJB equation. The sufficient conditions derived in this paper include standard completeness assumptions and the asymptotic stability of the associated vector field. The method is demonstrated on a simple nonlinear system and is compared to a result obtained by using exact feedback linearization in conjunction with the LQR design method.