Infinite Horizon Optimal Control Problem Research Articles

Neural‐network‐based online optimal control for uncertain non‐linear continuous‐time systems with control constraints

In this study, an online adaptive optimal control scheme is developed for solving the infinite-horizon optimal control problem of uncertain non-linear continuous-time systems with the control policy having saturation constraints. A novel identifier-critic architecture is presented to approximate the Hamilton-Jacobi-Bellman equation using two neural networks (NNs): an identifier NN is used to estimate the uncertain system dynamics and a critic NN is utilised to derive the optimal control instead of typical action-critic dual networks employed in reinforcement learning. Based on the developed architecture, the identifier NN and the critic NN are tuned simultaneously. Meanwhile, unlike initial stabilising control indispensable in policy iteration, there is no special requirement imposed on the initial control. Moreover, by using Lyapunov's direct method, the weights of the identifier NN and the critic NN are guaranteed to be uniformly ultimately bounded, while keeping the closed-loop system stable. Finally, an example is provided to demonstrate the effectiveness of the present approach.

Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints

In this paper, an adaptive reinforcement learning-based solution is developed for the infinite-horizon optimal control problem of constrained-input continuous-time nonlinear systems in the presence of nonlinearities with unknown structures. Two different types of neural networks (NNs) are employed to approximate the Hamilton–Jacobi–Bellman equation. That is, an recurrent NN is constructed to identify the unknown dynamical system, and two feedforward NNs are used as the actor and the critic to approximate the optimal control and the optimal cost, respectively. Based on this framework, the action NN and the critic NN are tuned simultaneously, without the requirement for the knowledge of system drift dynamics. Moreover, by using Lyapunov’s direct method, the weights of the action NN and the critic NN are guaranteed to be uniformly ultimately bounded, while keeping the closed-loop system stable. To demonstrate the effectiveness of the present approach, simulation results are illustrated.

Optimal control for a polynomial system with a quadratic criterion over infinite horizon

This paper proposes an optimal control algorithm for a polynomial system with a quadratic criterion over infinite horizon. The designed regulator gives a closed-form solution to the infinite horizon optimal control problem for a polynomial system with a quadratic criterion. The obtained solution consists of a feedback control law obtained by solving a Riccati algebraic equation dependent on the state. Numerical simulations in the example show advantages of the developed algorithm.

Optimal advertising strategies with age-structured goodwill

The problem of a firm willing to optimally promote and sell a single product on the market is here undertaken. The awareness of such product is modeled by means of a Nerlove–Arrow goodwill as a state variable, differentiated jointly by means of time and of age of the segments in which the consumers are clustered. The problem falls into the class of infinite horizon optimal control problems of PDEs with age structure that have been studied in various papers either in cases when explicit solutions can be found or using Maximum Principle techniques. Here, assuming an infinite time horizon, we use some dynamic programming techniques in infinite dimension to characterize both the optimal advertising effort and the optimal goodwill path in the long run. An interesting feature of the optimal advertising effort is an anticipation effect with respect to the segments considered in the target market, due to time evolution of the segmentation. We analyze this effect in two different scenarios: in the first, the decision-maker can choose the advertising flow directed to different age segments at different times, while in the second she/he can only decide the activation level of an advertising medium with a given age-spectrum.

Pontryagin principles for bounded discrete-time processes

We establish necessary conditions and sufficient conditions of optimality in the form of Pontryagin principles for infinite-horizon discrete-time optimal control problems governed by a difference inequation.

Stable iterative adaptive dynamic programming algorithm with approximation errors for discrete-time nonlinear systems

In this paper, a novel iterative adaptive dynamic programming (ADP) algorithm is developed to solve infinite horizon optimal control problems for discrete-time nonlinear systems. When the iterative control law and iterative performance index function in each iteration cannot be accurately obtained, it is shown that the iterative controls can make the performance index function converge to within a finite error bound of the optimal performance index function. Stability properties are presented to show that the system can be stabilized under the iterative control law which makes the present iterative ADP algorithm feasible for implementation both on-line and off-line. Neural networks are used to approximate the iterative performance index function and compute the iterative control policy, respectively, to implement the iterative ADP algorithm. Finally, two simulation examples are given to illustrate the performance of the present method.

Existence of solutions for a class of nonconcave infinite horizon optimal control problems

In this article, we establish the existence of optimal solutions for a large class of nonconvex infinite horizon discrete-time optimal control problems. This class contains optimal control problems arising in economic dynamics which describe a model with nonconcave utility functions representing the preferences of the planner.

Policy Iteration Algorithm for Singular Controlled Diffusion Processes

In this paper, the infinite horizon optimal control problems for singular diffusion processes are considered from the viewpoints of Markov decision processes and perturbation analysis, where the singularity of diffusion means that the covariance matrix of the system noise is allowed to be degenerate. A formula of performance difference under two different controls is derived and leads to a comparison theorem. By the comparison theorem, starting from a control, a so-called better control can be selected. Therefore, a control policy iteration algorithm is developed, by which the performance improves step by step and converges to the optimal one. When this applies to the stochastic affine nonlinear regulator and stochastic linear quadratic optimal control problems, better control can be constructed in a closed form. It is also shown that when the considered stochastic systems degenerate to the deterministic ones, the proposed algorithm reduces to the adaptive dynamic programming algorithm [J. J. Murray, C. J. Cox, G. G. Lendaris, and R. Saeks, Adaptive dynamic programming, IEEE Trans. Systems Man Cybernet., 32 (2002), pp. 140--153] for the affine nonlinear systems and to the well-known Kleinman algorithm [D. L. Kleinman, On an iterative technique for Riccati equation computation, IEEE Trans. Automat. Control, 13 (1968), pp. 114--115] for the linear quadratic optimal control problem.

Infinite Horizon Optimal Control of Stochastic Delay Evolution Equations in Hilbert Spaces

The aim of the present paper is to study an infinite horizon optimal control problem in which the controlled state dynamics is governed by a stochastic delay evolution equation in Hilbert spaces. The existence and uniqueness of the optimal control are obtained by means of associated infinite horizon backward stochastic differential equations without assuming the Gâteaux differentiability of the drift coefficient and the diffusion coefficient. An optimal control problem of stochastic delay partial differential equations is also given as an example to illustrate our results.

The intrinsic comparative dynamics of infinite horizon optimal control problems with a time-varying discount rate and time-distance discounting

The intrinsic comparative dynamics of a ubiquitous class of optimal control problems with a time-varying discount rate and time-distance discounting are derived and shown to be characterized by a positive semidefinite matrix. It is also shown that the said comparative dynamics are invariant to the functional form of the discount rate function and the type of agent. Consequently, if one limits econometric testing to the basic comparative dynamics of the given class of control problems, one cannot determine (i) the functional form of the discount rate function used by an agent, and thus if an agent is a time-consistent or time-inconsistent decision maker, or (ii) if an agent commits to a plan of action or takes into account the changing nature of his preferences when choosing a plan.

A novel actor–critic–identifier architecture for approximate optimal control of uncertain nonlinear systems

An online adaptive reinforcement learning-based solution is developed for the infinite-horizon optimal control problem for continuous-time uncertain nonlinear systems. A novel actor–critic–identifier (ACI) is proposed to approximate the Hamilton–Jacobi–Bellman equation using three neural network (NN) structures—actor and critic NNs approximate the optimal control and the optimal value function, respectively, and a robust dynamic neural network identifier asymptotically approximates the uncertain system dynamics. An advantage of using the ACI architecture is that learning by the actor, critic, and identifier is continuous and simultaneous, without requiring knowledge of system drift dynamics. Convergence of the algorithm is analyzed using Lyapunov-based adaptive control methods. A persistence of excitation condition is required to guarantee exponential convergence to a bounded region in the neighborhood of the optimal control and uniformly ultimately bounded (UUB) stability of the closed-loop system. Simulation results demonstrate the performance of the actor–critic–identifier method for approximate optimal control.

On the Existence of a Classical Optimal Solution and of an Almost Strongly Optimal Solution for an Infinite-Horizon Control Problem

We consider an infinite-horizon optimal control problem with the cost functional described either by an integral over an unbounded interval (a Lebesgue integral) or by a limit of integrals (an improper Lebesgue integral). We prove some theorems on the existence of solutions to such problems. The proofs are based on appropriate lower closure theorems and some extensions of Olech’s theorem on the lower semicontinuity of an integral functional; these extensions cover the cases of functionals described by an integral over an unbounded interval and by a limit of integrals.

Stabilizing the Hamiltonian system for constructing optimal trajectories

An infinite-horizon optimal control problem based on an economic growth model is studied. The goal in the problem is to optimize the mechanisms of investment in basic production assets in order to increase the growth rate of the consumption level. The main output variable-the gross domestic product (GDP)-depends on three production factors: capital stock, human capital, and useful work. The first two factors are endogenous variables of the model, and the useful work is an exogenous factor. The dependence of the GDP on the production factors is described by the Cobb-Douglas power-type production function. The economic system under consideration is assumed to be closed, so the GDP is distributed between consumption and investment in the capital stock and human capital. The optimal control problem consists in determining optimal investment strategies that maximize the integral discounted relative consumption index on an infinite time interval. A solution to the problem is constructed on the basis of the Pontryagin maximum principle adapted to infinite-horizon problems. We examine the questions of existence and uniqueness of a solution, verify necessary and sufficient optimality conditions, and perform a qualitative analysis of Hamiltonian systems on the basis of which we propose an algorithm for constructing optimal trajectories. This algorithm uses information on solutions obtained by means of a nonlinear regulator. Finally, we estimate the accuracy of the algorithm with respect to the integral cost functional of the control process.

Infinite horizon optimal control problems with multiple thermostatic hybrid dynamics

We study an optimal control problem for a hybrid system exhibiting several internal switching variables whose discrete evolutions are governed by some delayed thermostatic laws. By the dynamic programming technique we prove that the value function is the unique viscosity solution of a system of several Hamilton–Jacobi equations, suitably coupled. The method involves a contraction principle and some suitably adapted results for exit-time problems with discontinuous exit cost.

Performance bounds for λ policy iteration and application to the game of Tetris

We consider the discrete-time infinite-horizon optimal control problem formalized by Markov decision processes (Puterman, 1994; Bertsekas and Tsitsiklis, 1996). We revisit the work of Bertsekas and...

Online optimal control of nonlinear discrete-time systems using approximate dynamic programming

In this paper, the optimal control of a class of general affine nonlinear discrete-time (DT) systems is undertaken by solving the Hamilton Jacobi-Bellman (HJB) equation online and forward in time. The proposed approach, referred normally as adaptive or approximate dynamic programming (ADP), uses online approximators (OLAs) to solve the infinite horizon optimal regulation and tracking control problems for affine nonlinear DT systems in the presence of unknown internal dynamics. Both the regulation and tracking controllers are designed using OLAs to obtain the optimal feedback control signal and its associated cost function. Additionally, the tracking controller design entails a feedforward portion that is derived and approximated using an additional OLA for steady state conditions. Novel update laws for tuning the unknown parameters of the OLAs online are derived. Lyapunov techniques are used to show that all signals are uniformly ultimately bounded and that the approximated control signals approach the optimal control inputs with small bounded error. In the absence of OLA reconstruction errors, an optimal control is demonstrated. Simulation results verify that all OLA parameter estimates remain bounded, and the proposed OLA-based optimal control scheme tunes itself to reduce the cost HJB equation.

LQ control of behavior systems in kernel representation

In this paper we consider linear systems given through a kernel representation P together with a quadratic cost functional H ̃ . We will see that the solvability of the associated infinite-horizon optimal control problem is equivalent to cyclo-dissipativity and that the solution of the optimal control problem can be obtained from the kernel representation of a para-Hermitian matrix polynomial which can be easily constructed from P and H ̃ . In contrast to previous work on this topic we avoid image representations as far as possible. The results can be applied to descriptor systems (see Section 4) to obtain results which so far were only known for regular descriptor systems with index 1 .

Pseudospectral methods for solving infinite-horizon optimal control problems

An important aspect of numerically approximating the solution of an infinite-horizon optimal control problem is the manner in which the horizon is treated. Generally, an infinite-horizon optimal control problem is approximated with a finite-horizon problem. In such cases, regardless of the finite duration of the approximation, the final time lies an infinite duration from the actual horizon at t = + ∞ . In this paper we describe two new direct pseudospectral methods using Legendre–Gauss (LG) and Legendre–Gauss–Radau (LGR) collocation for solving infinite-horizon optimal control problems numerically. A smooth, strictly monotonic transformation is used to map the infinite time domain t ∈ [ 0 , ∞ ) onto a half-open interval τ ∈ [ − 1 , 1 ) . The resulting problem on the finite interval is transcribed to a nonlinear programming problem using collocation. The proposed methods yield approximations to the state and the costate on the entire horizon, including approximations at t = + ∞ . These pseudospectral methods can be written equivalently in either a differential or an implicit integral form. In numerical experiments, the discrete solution exhibits exponential convergence as a function of the number of collocation points. It is shown that the map ϕ : [ − 1 , + 1 ) → [ 0 , + ∞ ) can be tuned to improve the quality of the discrete approximation.

Advertising and exogenous interference in a segmented market

We propose a model of a monopolist firm which advertises a product in a segmented market where a constant exogenous interference is present. Using the framework of Nerlove-Arrow advertising model, we describe the interference using a constant negative addendum in the goodwill ODE. This effect may vary over the different market segments. Hence, we admit that the firm goodwill concerning any segment may become negative and we associate 0 demand with negative goodwill values. By using the simplest demand model, i.e. a piecewise linear function, we formulate a nonsmooth infinite horizon optimal control problem. The features of an optimal advertising policy depend on the different effects in each segment of the advertising effort and of the exogenous interference. We describe an optimal solution for a single wide-spectrum medium, which requires a constant advertising effort after some finite time. We characterize the segment sets which may be a possible target for the firm using a given advertising medium.

Dynamic Programming Techniques for Feedback Control

AbstractWe consider a dynamic programming approach for solving optimal control problems of sampled continuous-time systems. Robustness to bounded noise and model uncertainties is provided by formulating the problem in the framework of differential games. We use a semi-Lagrangian scheme for the computation of the value function and the state feedback control law for constrained infinite horizon optimal control problems. The approach is illustrated with the optimal control of linear systems and nonlinear systems subject to bounded disturbances.