Hamilton Jacobi Bellman Partial Differential Equation Research Articles

Dynamic cointegrated pairs trading: Mean–variance time-consistent strategies

Cointegration is a useful econometric tool for identifying assets which share a common equilibrium. Cointegrated pairs trading is a trading strategy which attempts to take a profit when cointegrated assets depart from their equilibrium. This paper investigates the optimal dynamic trading of cointegrated assets using the classical mean–variance portfolio selection criterion. To ensure rational economic decisions, the optimal strategy is obtained over the set of time-consistent policies from which the optimization problem is enforced to obey the dynamic programming principle. We solve the optimal dynamic trading strategy in a closed-form explicit solution from a nonlinear Hamilton–Jacobi–Bellman partial differential equation. This analytical tractability enables us to prove rigorously that cointegration ensures the existence of statistical arbitrage using a dynamic time-consistent mean–variance strategy via asymptotic analysis. This provides the theoretical grounds for the market belief in cointegrated pairs trading. Comparison between time-consistent and precommitment trading strategies for cointegrated assets shows the former to be a persistent approach, whereas the latter makes it possible to generate infinite leverage once a cointegrating factor of the assets has a high mean reversion rate.

American option valuation in a stochastic volatility model with transaction costs

In the present paper we analyse the American option valuation problem in a stochastic volatility model when transaction costs are taken into account. We shall show that it can be formulated as a singular stochastic optimal control problem, proving the existence and uniqueness of the viscosity solution for the associated Hamilton–Jacobi–Bellman partial differential equation. Moreover, after performing a dimensionality reduction through a suitable choice of the utility function, we shall provide a numerical example illustrating how American options prices can be computed in the present modelling framework.

Closed-form solution to finite-horizon suboptimal control of nonlinear systems

Finite-horizon optimal control of input-affine nonlinear systems with fixed final time is considered in this study. It is first shown that the associated Hamilton–Jacobi–Bellman partial differential equation to the problem is reducible to a state-dependent differential Riccati equation after some approximations. With a truncation in the control equation, a near optimal solution to the problem is obtained, and the global onvergence properties of the closed-loop system are analyzed. Afterwards, an approximate method, called Finite-horizon State-Dependent Riccati Equation (Finite-SDRE), is suggested for solving the differential Riccati equation, which renders the origin a locally exponentially stable point. The proposed method provides online feedback solution for controlling different initial conditions. Finally, through some examples, the performance of the resulting controller in finite-horizon control is analyzed. Copyright © 2014 John Wiley & Sons, Ltd.

Approximate finite-horizon optimal control for input-affine nonlinear systems with input constraints

The finite-horizon optimal control problem with input constraints consists in controlling the state of a dynamical system over a finite time interval (possibly unknown) minimising a cost functional, while satisfying hard constraints on the input. In this framework, the minimum-time optimal control problem and some related problems are of interest for both theory and applications. For linear systems, the solution of the problem often relies upon the use of bang-bang control signals. For nonlinear systems, the “shape” of the optimal input is in general not known. The control input can be found solving a Hamilton–Jacobi–Bellman (HJB) partial differential equation (PDE): it typically consists of a combination of bang-bang controls and singular arcs. In this paper, a methodology to approximate the solution of the HJB PDE is proposed. This approximation yields a dynamic state feedback law. The theory is illustrated by means of two examples: the minimum-time optimal control problem for an industrial wastewater treatment plant and the Goddard problem, i.e. a maximum-range optimal control problem.

Statistical Control for Performance Shaping Using Cost Cumulants

The cost cumulant statistical control method optimizes the system performance by shaping the probability density function of the random cost function. For a stochastic system, a typical optimal control method minimizes the mean (first cumulant) of the cost function. However, there are other statistical properties of the cost function such as variance (second cumulant), skewness (third cumulant) and kurtosis (fourth cumulant), which affect system performance. In this technical note, we extend the theory of traditional stochastic control by deriving the Hamilton-Jacobi-Bellman (HJB) equation as the necessary conditions for optimality. Furthermore, we derived the verification theorem, which is the sufficient condition, for higher order statistical control, and construct the optimal controller. In addition, we utilize neural networks to numerically solve HJB partial differential equations. Finally, we provide simulation results for an oscillator system to demonstrate our method.

Solution of the Feedback Control Problem in the Mathematical Model of Leukaemia Therapy

A mathematical model of leukaemia therapy based on the Gompertzian law of cell growth is investigated. The effect of the medicine on the leukaemia and normal cells is described in terms of therapy functions. A feedback control problem with the purpose of minimizing the number of the leukaemia cells while retaining as much as possible the number of normal cells is considered. This problem is reduced to solving the nonlinear Hamilton–Jacobi–Bellman partial differential equation. The feedback control synthesis is obtained by constructing an exact analytical solution to the corresponding Hamilton–Jacobi–Bellman equation.

Survey of State-Dependent Riccati Equation in Nonlinear Optimal Feedback Control Synthesis

A EROSPACE engineering applications greatly stimulated the development of optimal control theory during the 1950s and 1960s, where the objective was to drive the system states in such a way that some defined cost was minimized. This turned out to have very useful applications in the design of regulators (where some steady state is to be maintained) and in tracking control strategies (where some predetermined state trajectory is to be followed). Among such applications was the problem of optimal flight trajectories for aircraft and space vehicles. Linear optimal control theory in particular has been very well documented and widely applied, where the plant that is controlled is assumed linear and the feedback controller is constrained to be linear with respect to its input. However, the availability of powerful low-cost microprocessors has spurred great advantages in the theory and applications of nonlinear control. The competitive era of rapid technological change, particularly in aerospace exploration, now demands stringent accuracy and cost requirements in nonlinear control systems. This has motivated the rapid development of nonlinear control theory for application to challenging, complex, dynamical real-world problems, particularly those that bear major practical significance in aerospace, marine, and defense industries. Infinite-time horizon nonlinear optimal control (ITHNOC) presents a viable option for synthesizing stabilizing controllers for nonlinear systems by making a state-input tradeoff, where the objective is to minimize the cost given by a performance index. The original theory of nonlinear optimal control dates from the 1960s. Various theoretical and practical aspects of the problem have been addressed in the literature over the decades since. In particular, the continuous-time nonlinear deterministic optimal control problem associated with autonomous (time-invariant) nonlinear regulator systems that are affine (linear) in the controls has been studied by many authors. The long-established theory of optimal control offers quite mature and well-documented techniques for solving this control-affine nonlinear optimization problem, based on dynamic programming or calculus of variations, but their application is generally a very tedious task. Bellman’s dynamic programming approach reduces to solving a nonlinear first-order partial differential equation (PDE), expressed by the Hamilton–Jacobi–Bellman (HJB) equation. The solution to the HJB equation gives the optimal performance/cost value (or storage) function and determines an optimal control in feedback form under some smoothness assumptions. Alternatively, in the classical calculus of variations, optimal control problems can be characterized locally in terms of the Hamiltonian dynamics arising from Pontryagin’s minimum principle. These are the characteristic equations of the HJB PDE, which result in a nonlinear, constrained two-point boundary value problem (TPBVP) that, in general, can only be solved by successive approximation of the optimal control input using iterative numerical techniques for each set of initial conditions. Numerically, even though the nonlinear TPBVP is somewhat easier to solve than the HJBPDE, control signals can only be determined offline and are thus best suited for feedforward control of plants for which the state trajectories are known a priori. Therefore, contrary to the dynamic programming approach, the resultant control law is not generally in feedback form. Open-loop control, however, is sensitive to random disturbances and requires that the initial state be on the optimal trajectory. In contrast, nonlinear optimal feedback has inherent robustness properties (inherent in the sense that it is obtained by ignoring uncertainty and disturbances). The potential difficulty with the HJB approach is that no efficient algorithm is available to solve the PDE when it is nonlinear and the problem dimension is high, making it impossible to derive exact expressions for optimal controls for most nontrivial problems of interest. The optimal can only be computed in special cases, such as linear dynamics and quadratic cost, or very low-dimensional systems. In particular, if the plant is linear time invariant (LTI) and the (infinite-time) performance index is quadratic, then the corresponding HJB equation for this infamous linear-quadratic regulator (LQR) problem reduces to an algebraic Riccati equation (ARE). Contrary to the well-developed and widely applied theory and computational tools for theRiccati equation (for example, see [1]), theHJB equation is difficult, if not impossible, to solve for most practical applications. The exact solution for the optimal control policies is very complex

Convergence Rate for a Curse-of-Dimensionality-Free Method for a Class of HJB PDEs

In previous work of the first author and others, max-plus methods have been explored for solution of first-order, nonlinear Hamilton–Jacobi–Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. Although max-plus basis expansion and max-plus finite-element methods can provide substantial computational-speed advantages, they still generally suffer from the curse-of-dimensionality. Here we consider HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We consider a previously obtained numerical method not subject to the curse-of-dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with kernel obtained from the originating semigroup. One considers repeated application of the dual-space semigroup to obtain the solution. Although previous work indicated that the method was not subject to the curse-of-dimensionality, it did not indicate any error bounds or convergence rate. Here we obtain specific error bounds.

Differential Transformation and Its Application to Nonlinear Optimal Control

A novel numerical method based on the differential transformation is proposed for solving nonlinear optimal control problems in this paper. The differential transformation is a linear operator that transforms a function from the original time and/or space domain into another domain in order to simplify the differential calculations. The optimality conditions for the optimal control problems can be represented by algebraic and differential equations. Using the differential transformation, these algebraic and differential equations with their boundary conditions are first converted into a system of nonlinear algebraic equations. Then the numerical optimal solutions are obtained in the form of finite-term Taylor series by solving the system of nonlinear algebraic equations. The differential transformation algorithm is similar to the spectral element methods in that the computational region splits into several subregions but it uses polynomials of high degrees by keeping a small number of subregions. The differential transformation algorithm could solve the finite- (or infinite-) time horizon optimal control problems formulated as either the algebraic and ordinary differential equations using Pontryagin’s minimum principle or the Hamilton–Jacobi–Bellman partial differential equation using dynamic programming in one unified framework. In addition, the differential transformation algorithm can efficiently solve optimal control problems with the piecewise continuous dynamics and/or nonsmooth control. The performance is demonstrated through illustrative examples.

Convergence Rate for a Curse-of-dimensionality-Free Method for Hamilton–Jacobi–Bellman PDEs Represented as Maxima of Quadratic Forms

In previous work of the author and others, max-plus methods have been explored for solution of first-order, nonlinear Hamilton–Jacobi–Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. Although max-plus basis expansion and max-plus finite-element methods provide computational-speed advantages, they still generally suffer from the curse-of-dimensionality. Here we consider HJB PDEs where the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to a solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We consider a previously obtained numerical method not subject to the curse-of-dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with kernel obtained from the originating semigroup. One employs repeated application of the dual-space semigroup to obtain the solution. Here, we consider constituent Hamiltonians which contain linear and constant terms as well as purely quadratic terms. This greatly expands the allowable class of problems. However, there are solution existence issues, and the error bounds are more difficult.

Dirichlet Problems for some Hamilton–Jacobi Equations with Inequality Constraints

We use viability techniques for solving Dirichlet problems with inequality constraints (obstacles) for a class of Hamilton–Jacobi equations. The hypograph of the “solution” is defined as the “capture basin” under an auxiliary control system of a target associated with the initial and boundary conditions, viable in an environment associated with the inequality constraint. From the tangential condition characterizing capture basins, we prove that this solution is the unique “upper semicontinuous” solution to the Hamilton–Jacobi–Bellman partial differential equation in the Barron-Jensen/Frankowska sense. We show how this framework allows us to translate properties of capture basins into corresponding properties of the solutions to this problem. For instance, this approach provides a representation formula of the solution which boils down to the Lax–Hopf formula in the absence of constraints.

A Curse-of-Dimensionality-Free Numerical Method for Solution of Certain HJB PDEs

In previous works of the author and others, max-plus methods have been explored for the solution of first-order, nonlinear Hamilton–Jacobi–Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. These methods exploit the max-plus linearity of the associated semigroups. In particular, although the problems are nonlinear, the semigroups are linear in the max-plus sense. These methods have been used successfully to compute solutions. Although they provide certain computational-speed advantages, they still generally suffer from the curse of dimensionality. Here we consider HJB PDEs in which the Hamiltonian takes the form of a (pointwise) maximum of linear/quadratic forms. The approach to the solution will be rather general, but in order to ground the work, we consider only constituent Hamiltonians corresponding to long-run average-cost-per-unit-time optimal control problems for the development. We obtain a numerical method not subject to the curse of dimensionality. The method is based on construction of the dual-space semigroup corresponding to the HJB PDE. This dual-space semigroup is constructed from the dual-space semigroups corresponding to the constituent linear/quadratic Hamiltonians. The dual-space semigroup is particularly useful due to its form as a max-plus integral operator with a kernel obtained from the originating semigroup. One considers repeated application of the dual-space semigroup to obtain the solution.

Max-plus summation of Fenchel-transformed semigroups for solution of nonlinear Bellman equations

Max-plus methods have been explored for solution of first-order, nonlinear Hamilton–Jacobi–Bellman partial differential equations (HJB PDEs) and corresponding nonlinear control problems. These methods exploit the max-plus linearity of the associated semigroups. In particular, although the problems are nonlinear, the semigroups are linear in the max-plus sense. These methods have been used successfully to compute solutions. Although they provide certain advantages, they still generally suffer from the curse-of-dimensionality. The natural analog to the Laplace transform in ordinary spaces is the Fenchel transform over max-plus spaces, the range space being referred to as the dual space. One can transform the semigroup operators into operators on the dual space. There are natural operations on the transformed operators which may be used to construct solutions of nonlinear control problems. Natural building blocks correspond to transforms of operators for linear/quadratic problems. In this paper, a method for exploiting operations in the Fenchel transform space is used to develop a method for certain problems such that the computational growth in the most time-consuming portion of the computations can be hugely reduced. Although the curse-of-dimensionality is not entirely avoided, the computational cost reductions are very high for some classes of problems.

A numerical method to approximate optimal production and maintenance plan in a flexible manufacturing system

Abstract The simultaneous planning of the production and the maintenance in a flexible manufacturing system is considered in this paper. The manufacturing system is composed of one machine that produces a single product. There is a preventive maintenance plan to reduce the failure rate of the machine. This paper is different from the previous researches in this area in two separate ways. First, the failure rate of the machine is supposed to be a function of its age. Second, we assume that the demand of the manufacturing product is time dependent and its rate depends on the level of advertisement on that product. The objective is to maximize the expected discounted total profit of the firm over an infinite time horizon. In the process of finding a solution to the problem, we first characterize an optimal control by introducing a set of Hamilton–Jacobi–Bellman partial differential equations. Then we realize that under practical assumptions, this set of equations can not be solved analytically. Thus to find a suboptimal control, we approximate the original stochastic optimal control model by a discrete-time deterministic optimal control problem. Then proposing a numerical method to solve the steady state Riccati equation, we approximate a suboptimal solution to the problem.

Global optimal feedback control for general nonlinear systems with nonquadratic performance criteria

Optimal control of general nonlinear nonaffine controlled systems with nonquadratic performance criteria (that permit state- and control-dependent time-varying weighting parameters), is solved classically using a sequence of linear- quadratic and time-varying problems. The proposed method introduces an “approximating sequence of Riccati equations” (ASRE) to explicitly construct nonlinear time-varying optimal state-feedback controllers for such nonlinear systems. Under very mild conditions of local Lipschitz continuity, the sequences converge (globally) to nonlinear optimal stabilizing feedback controls. The computational simplicity and effectiveness of the ASRE algorithm is an appealing alternative to the tedious and laborious task of solving the Hamilton–Jacobi–Bellman partial differential equation. So the optimality of the ASRE control is studied by considering the original nonlinear-nonquadratic optimization problem and the corresponding necessary conditions for optimality, derived from Pontryagin's maximum principle. Global optimal stabilizing state-feedback control laws are then constructed. This is compared with the optimality of the ASRE control by considering a nonlinear fighter aircraft control system, which is nonaffine in the control. Numerical simulations are used to illustrate the application of the ASRE methodology, which demonstrate its superior performance and optimality.

Explicit Solution for a Network Control Problem in the Large Deviation Regime

We consider optimal control of a stochastic network, where service is controlled to prevent buffer overflow. We use a risk-sensitive escape time criterion, which in comparison to the ordinary escape time criteria heavily penalizes exits which occur on short time intervals. A limit as the buffer sizes tend to infinity is considered. In [2] we showed that, for a large class of networks, the limit of the normalized cost agrees with the value function of a differential game. In this game, one player controls the service discipline (who to serve and whether to serve), and the other player chooses arrival and service rates in the network. The game's value is characterized in [2] as the unique solution to a Hamilton–Jacobi–Bellman Partial Differential Equation (PDE). In the current paper we apply this general theory to the important case of a network of queues in tandem. Our main results are: (i) the construction of an explicit solution to the corresponding PDE, and (ii) drawing out the implications for optimal risk-sensitive and robust regulation of the network. In particular, the following general principle can be extracted. To avoid buffer overflow there is a natural competition between two tendencies. One may choose to serve a particular queue, since that will help prevent its own buffer from overflowing, or one may prefer to stop service, with the goal of preventing overflow of buffers further down the line. The solution to the PDE indicates the optimal choice between these two, specifying the parts of the state space where each queue must be served (so as not to lose optimality), and where it can idle. Referring to those queues which must be served as bottlenecks, one can use the solution to the PDE to explicitly calculate the bottleneck queues as a function of the system's state, in terms of a simple set of equations.

Max-Plus Methods for Nonlinear H∞ Control: Operating in the Transform Space

The solution of some forms of nonlinear H∞/L2-gain problems can be obtained via solution of the corresponding Hamilton-Jacobi-Bellman (HJB) partial differential equations (PDEs). Alternatively, the solution of some classes of HJB PDEs have representations as solutions of L2-gain problems. Both can be obtained through solution of corresponding fixed-point problems - where the operators are the semigroups associated with the PDEs. In the linear/quadratic case, the solutions of these problems can be obtained simply by solution of associated Riccati equations. Here, an exploration of a way in which the operators for linear/quadratic problems can be combined (in the semiconvex dual space) to obtain operators, and hence solutions, for more general problems is begun.