Infinite Horizon Optimal Control Problem Research Articles

Locally bounded variations epigraph property of the value function to infinite horizon optimal control problems under state constraints

Locally bounded variations epigraph property of the value function to infinite horizon optimal control problems under state constraints

Optimal control of Boolean control networks with state-triggered impulses

In the real world, abrupt changes caused by external environment and system interaction can exhibit a pattern resembling an “impulse” response. A novel model, known as the hybrid-index model, has recently been introduced for Boolean control networks (BCNs) to characterize such impulsive responses. The optimal control problem is a fundamental issue in control theory, and many control problems can be classified as special cases of optimal control problems. Nevertheless, the optimal control problem for the hybrid-index model remains unresolved. This study delves into the optimal control problem of impulsive Boolean control networks (IBCNs) under the hybrid-index model framework. Forward completeness serves as a crucial condition to avert Zeno behavior in the hybrid-index model of IBCNs. In contrast to prior studies on the hybrid-index model of IBCNs, the study shows that the optimal control problem of IBCNs can be addressed even in cases where the system is not forward complete. Leveraging the theory of semi-tensor product, both finite-horizon and infinite-horizon optimal control problems of IBCNs are tackled using the quotient mapping method and the recursive method for optimal control of Boolean control networks. The study yields the optimal cost and optimal control law through state feedback. To illustrate the efficacy of the proposed approach, a mathematical example and a biological example known as the λ switch are presented.

An analytical adaptive optimal control approach without solving HJB equation for nonlinear systems with input constraints

AbstractThis paper presents an analytical method to solve the optimal control problem for affine nonlinear systems with unknown drift dynamics. A new non‐quadratic cost function over an infinite horizon is presented that considers input constraints and includes the cost of the feed‐forward component of the control law. The mean value theorem for vector‐valued functions has been used to derive an integral form of this theorem. Based on this theorem, a rigorous proof is provided demonstrating that the cost function can be converted into another form. In the presence of input constraints, this converted form enables extracting the optimal control solution without solving the HJB equation. Additionally, unknown nonlinearity effects in drift dynamics are compensated in the control input. This is accomplished by estimating the unknown drift dynamics via an adaptive neural network (NN) approach. It is proven that the states and weights of NN are uniformly ultimately bounded based on a Lyapunov technique. The necessary and sufficient conditions are provided that ensure the optimality of the infinite horizon optimal control problem with a discount factor. As a result, it is demonstrated that the proposed approach satisfies the optimality criteria. To evaluate the effectiveness of the proposed approach, simulation examples are provided.

Reinforcement learning for optimal control of linear impulsive systems with periodic impulses

This paper focuses on the finite- and infinite-horizon optimal control problems of linear impulsive systems with periodic impulses under the quadratic performance index. Necessary and sufficient conditions for the optimal impulsive system are derived in terms of hybrid Riccati equations by utilizing the variational method and the collocation method combined with a time-varying Lyapunov function. Different from the existing model-based impulsive control schemes, three reinforcement learning (RL)-based algorithms are proposed to solve the optimal impulsive controller and hybrid controller for the impulsive system without the exact knowledge of the system dynamics. The asymptotical stability of the impulsive control system and the convergence of the RL-based algorithms are proved rigorously. Finally, a numerical simulation illustrates the effectiveness of the proposed control methods.

Policy gradient methods for discrete time linear quadratic regulator with random parameters

This paper studies an infinite horizon optimal control problem for discrete-time linear system and quadratic criteria, both with random parameters which are independent and identically distributed with respect to time. In this general setting, we apply the policy gradient method, a reinforcement learning technique, to search for the optimal control without requiring knowledge of statistical information of the parameters. We investigate the sub-Gaussianity of the state process and establish global linear convergence guarantee for this approach based on assumptions that are weaker and easier to verify compared to existing results. Numerical experiments are presented to illustrate our result.

Dissipativity in Infinite Horizon Optimal Control and Dynamic Programming

In this paper we extend dynamic programming techniques to the study of discrete-time infinite horizon optimal control problems on compact control invariant sets with state-independent best asymptotic average cost. To this end we analyse the interplay of dissipativity and optimal control, and propose novel recursive approaches for the solution of so called shifted Bellman Equations.

Compactification method in linear programming approach to infinite-horizon optimal control problems with a noncompact state constraint

Compactification method in linear programming approach to infinite-horizon optimal control problems with a noncompact state constraint

Direct integral pseudospectral and integral spectral methods for solving a class of infinite horizon optimal output feedback control problems using rational and exponential Gegenbauer polynomials

This study is concerned with the numerical solution of a class of infinite-horizon linear regulation problems with state equality constraints and output feedback control. We propose two numerical methods to convert the optimal control problem into nonlinear programming problems (NLPs) using collocations in a semi-infinite domain based on rational Gegenbauer (RG) and exponential Gegenbauer (EG) basis functions. We introduce new properties of these basis functions and derive their quadratures and associated truncation errors. A rigorous stability analysis of the RG and EG interpolations is also presented. The effects of various parameters on the accuracy and efficiency of the proposed methods are investigated. The performance of the developed integral spectral method is demonstrated using two benchmark test problems related to a simple model of a divert control system and the lateral dynamics of an F-16 aircraft. Comparisons of the results of the current study with available numerical solutions show that the developed numerical scheme is efficient and exhibits faster convergence rates and higher accuracy.

Conditional Cost Function and Necessary Optimality Conditions for Infinite Horizon Optimal Control Problems

Conditional Cost Function and Necessary Optimality Conditions for Infinite Horizon Optimal Control Problems

CONDITIONAL COST FUNCTION AND NECESSARY OPTIMALITY CONDITIONS FOR INFINITE HORIZON OPTIMAL CONTROL PROBLEMS

Infinite horizon optimal control problem with general endpoint constraints is reduced to a family of standard problems on finite time intervals containing the value of the conditional cost of the phase vector as a terminal term. New version of the Pontryagin maximum principle containing an explicit characterization of the adjoint variable is obtained for the problem with a general asymptotic endpoint constraint. In the case of the problem with free final state this approach leads to a normal form version of the maximum principle formulated completely in the terms of the conditional cost function.

Mathematical Model of Human Capital Dynamics

A mathematical description of household economic behavior is studied. On the one hand, households are consumers that seek to maximize the discounted utility function in an imperfect market of savings and consumer loans. On the other hand, households are workers in the labor market; they receive a wage and seek to enhance their skills to receive a higher wage. An increase in the level of worker’s skill is achieved via investment in human capital. In this paper, a mathematical model of the worker’s behavior in the labor market is represented in the form of an infinite-horizon optimal control problem. A solution existence theorem is proved, and necessary optimality conditions are obtained in the form of Pontryagin’s maximum principle. The model is identified using Russian statistical data for various social layers.

Necessary Conditions for the Optimality and Sustainability of Solutions in Infinite-Horizon Optimal Control Problems

The paper deals with an infinite-horizon optimal control problem with general asymptotic endpoint constraints. The fulfillment of constraints of this type can be viewed as the minimal necessary condition for the sustainability of solutions. A new version of the Pontryagin maximum principle with an explicitly specified adjoint variable is developed. The proof of the main results is based on the fact that the restriction of the optimal process to any finite time interval is a solution to the corresponding finite-horizon problem containing the conditional cost of the phase vector as a terminal term.

On optimal control for linear distributed time-delay systems

This article presents the solution to the infinite horizon optimal control problem for linear distributed time-delay systems. It is based on the use of the Cauchy solution for distributed time-delay systems. In contrast to previous results for systems with concentrated delays, the form of the functional and its properties are formally justified. An important property is demonstrated: the Bellman functional admits upper and lower bounds; thus, it is proved to be positive definite. Additionally, some experimental results of a temperature process are presented.

Infinite Horizon Optimal Control Problems with Discount Factor on the State, Part II: Analysis of the Control Problem

This is the continuation of our work on infinite horizon optimal control problems with a discount factor on the state variable and nonlinear partial differential equations as constraints. Existence of a solution is proven, and first as well as second order optimality conditions are derived. They are used to analyze the approximation of the infinite horizon problem by finite horizon problems.

Infinite Horizon Optimal Control Problems with Discount Factor on the State, Part I: Analysis of the Controlled State Equation

This is the first part of an investigation of infinite horizon optimal control problems subject to semilinear parabolic equations. A discount factor on the state variable is introduced in the cost. This allows for the treatment of infinite horizon problems without stabilizability assumptions. The nonlinearities can be of polynomial type, thus covering reaction-diffusion equations which are important for applications. The control-to-state mapping and its regularity are analyzed in detail. This involves the relation between the type of the nonlinearity and the discount factor.

Infinite Horizon Optimal Control for a General Class of Semilinear Parabolic Equations

A class of infinite horizon optimal control problems subject to semilinear parabolic equations is investigated. First and second order optimality conditions are obtained, in the presence of constraints on the controls, which can be either pointwise in space-time, or pointwise in time and L^2 in space. These results rely on a new L^infty estimate for nonlinear parabolic equations in an essential manner.

A novel actor–critic–identifier architecture for nonlinear multiagent systems with gradient descent method

The infinite-horizon optimal consensus control problem for continuous-time unknown nonlinear multiagent systems (MASs) is solved via an online adaptive reinforcement learning approach. Three different neural network (NN) structures are proposed as part of a novel actor–critic–identifier (ACI) to approximate the Hamilton–Jacobi–Bellman (HJB) equations. The actor and critic NNs approximate the optimal controls and optimal value functions, respectively, and identifier NN approximates the unknown system model. The ACI architecture is simpler than most known architectures, which has the benefit of allowing actor, critic, and identifier learning to occur concurrently and continuously without the system model. It is important to note that the tuning laws for the three different NNs are designed using gradient descent method. Adaptive control techniques based on Lyapunov method are used to examine the algorithm’s convergence. To verify the viability of the proposed approach, we offer a simulated example.

Reduced order model predictive control for parametrized parabolic partial differential equations

Model Predictive Control (MPC) is a well-established approach to solve infinite horizon optimal control problems. Since optimization over an infinite time horizon is generally infeasible, MPC determines a suboptimal feedback control by repeatedly solving finite time optimal control problems. Although MPC has been successfully used in many applications, applying MPC to large-scale systems – arising, e.g., through discretization of partial differential equations – requires the solution of high-dimensional optimization problems and thus poses immense computational effort. We consider systems governed by parametrized parabolic partial differential equations and employ the reduced basis method as a low-dimensional surrogate model for the finite time optimal control problem. The reduced order optimal control serves as feedback control for the original large-scale system. We analyze the proposed RB-MPC approach by first developing a posteriori error bounds for the errors in the optimal control and associated cost functional. These bounds can be evaluated efficiently in an offline-online computational procedure and allow us to guarantee asymptotic stability of the closed-loop system using the RB-MPC approach in several practical scenarios. We also propose an adaptive strategy to choose the prediction horizon of the finite time optimal control problem. Numerical results are presented to illustrate the theoretical properties of our approach.

Model-Based Policy Iterations for Nonlinear Systems via Controlled Hamiltonian Dynamics

The infinite-horizon optimal control problem for nonlinear systems is studied. In the context of model-based, iterative learning strategies we propose an alternative definition and construction of the <i>temporal difference error</i> arising in Policy Iteration strategies. In such architectures the error is computed via the evolution of the Hamiltonian function (or, possibly, of its integral) along the trajectories of the closed-loop system. Herein the temporal difference error is instead obtained via two subsequent steps: first the dynamics of the underlying costate variable in the Hamiltonian system is steered by means of a (virtual) control input in such a way that the stable invariant manifold becomes externally attractive. Then, the <i>distance-from-invariance</i> of the manifold, induced by approximate solutions, yields a natural candidate measure for the <i>policy evaluation</i> step. The <i>policy improvement</i> phase is then performed by means of standard gradient descent methods that allows to correctly update the weights of the underlying functional approximator. The above architecture then yields an <i>iterative (episodic) learning</i> scheme based on a scalar, constant <i>reward</i> at each iteration, the value of which is insensitive to the length of the episode, as in the original spirit of Reinforcement Learning strategies for discrete-time systems. Finally, the theory is validated by means of a numerical simulation involving an automatic flight control problem.

Approximating the Stationary Bellman Equation by Hierarchical Tensor Products

We treat infinite horizon optimal control problems by solving the associated stationary Hamilton-Jacobi-Bellman (HJB) equation numerically to compute the value function and an optimal feedback law. The dynamical systems under consideration are spatial discretizations of non linear parabolic partial differential equations (PDE), which means that the HJB is non linear and suffers from the curse of dimensionality. Its non linearity is handled by the Policy Iteration algorithm, where the problem is reduced to a sequence of linear, hyperbolic PDEs. These equations remain the computational bottleneck due to their high dimensions. By the method of characteristics these linearized HJB equations can be reformulated via the Koopman operator in the spirit of dynamic programming. The resulting operator equations are solved using a minimal residual method. To overcome numerical infeasability we use low rank hierarchical tensor product approximation/tree-based tensor formats, in particular tensor trains (TT tensors), and multi-polynomials, together with high dimensional quadrature, e.g. Monte-Carlo. By controlling a destabilized version of viscous Burgers and a diffusion equation with unstable reaction term numerical evidences are given.