Dynamic Programming For Optimal Control Research Articles

Constrained-Cost Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems.

For discrete-time nonlinear systems, this research is concerned with optimal control problems (OCPs) with constrained cost, and a novel value iteration with constrained cost (VICC) method is developed to solve the optimal control law with the constrained cost functions. The VICC method is initialized through a value function constructed by a feasible control law. It is proven that the iterative value function is nonincreasing and converges to the solution of the Bellman equation with constrained cost. The feasibility of the iterative control law is proven. The method to find the initial feasible control law is given. Implementation using neural networks (NNs) is introduced, and the convergence is proven by considering the approximation error. Finally, the property of the present VICC method is shown by two simulation examples.

Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems With Trajectory-Based Initial Control Policy

Adaptive Dynamic Programming for Optimal Control of Discrete-Time Nonlinear Systems With Trajectory-Based Initial Control Policy

Fourier–Hermite Dynamic Programming for Optimal Control

In this paper, we propose a novel computational method for solving non-linear optimal control problems. The method is based on the use of Fourier–Hermite series for approximating the action-value function arising in dynamic programming instead of the conventional Taylor-series expansion used in differential dynamic programming (DDP). The coefficients of the Fourier–Hermite series can be numerically computed by using sigma-point methods, which leads to a novel class of sigma-point based dynamic programming methods. We also prove the quadratic convergence of the method and experimentally test its performance against other methods.

Dimension reduction based adaptive dynamic programming for optimal control of discrete-time nonlinear control-affine systems

Dynamic programming based methods have been widely used in solving discrete-time nonlinear constrained optimal control problems. However, applying these methods in real-time is challenging because a large amount of memory is needed and the associated computational cost is high. Here, a search space dimension reduction strategy is proposed for a class of nonlinear discrete-time systems that are control-affine and invertible. Specifically, a bio-inspired motion rule is combined with inverse dynamics to reduce the value iteration search space to one dimension. The corresponding suboptimal control algorithm is developed and its optimality is analysed. An adaptation rule is developed to estimate uncertainties and improve the base policy. The closed-loop system is proven to be asymptotically stable. The advantages of the algorithm including much smaller computational cost and significantly reduced memory usage are demonstrated with two simulation examples.

Data-Driven Adaptive Dynamic Programming for Optimal Control of Continuous-Time Multicontroller Systems With Unknown Dynamics

This paper investigates the optimal control of continuous-time multi-controller systems with completely unknown dynamics using data-driven adaptive dynamic programming (DD-ADP). In this investigation, all controllers take actions together as a team, and they have precisely the same cost function, which is actually a fully cooperative game. According to optimal control theory, the HJB equation corresponding to the fully cooperative game is derived. To obtain the solution to HJB equation, a model-based policy iteration (PI) algorithm is first presented. On the basis of the PI algorithm, a DD-ADP algorithm without requiring the system dynamics is developed, and the neural networks (NNs) implementation scheme of the developed DD-ADP algorithm is given. Stability and convergence analysis are derived by Lyapunov theory. Finally, numerical simulation examples on linear and nonlinear multi-controller systems demonstrate the effectiveness of the designed scheme.

Adaptive dynamic programming for optimal control of discrete‐time nonlinear system with state constraints based on control barrier function

AbstractAdaptive dynamic programming (ADP) methods have demonstrated their efficiency. However, many of the applications for which ADP offers great potential, are also safety‐critical and need to meet safety specifications in the presence of physical constraints. In this article, an optimal controller for solving discrete‐time nonlinear systems with state constraints is proposed. By introducing the control barrier function into the utility function, the problem with state constraints is transformed into an unconstrained optimal control problem, addressing state constraints which are difficult to handle by traditional ADP methods. The constructed sequence of value function is shown to be monotonically non‐increasing and converges to the optimal value. Besides, this article gives the stability proof of the developed algorithm, as well as the conditions for satisfying the state constraints. To implement and approximate the control barrier function based adaptive dynamic programming algorithm, an actor‐critic network structure is built. During the training process, two neural networks are used for approximation separately. The performance of the proposed method is validated by testing it on a simulation example.

Multi-step heuristic dynamic programming for optimal control of nonlinear discrete-time systems

Policy iteration and value iteration are two main iterative adaptive dynamic programming frameworks for solving optimal control problems. Policy iteration converges fast while requiring an initial stabilizing control policy, which is a strict constraint in practice. Value iteration avoids the requirement of initial admissible control policy while converging much slowly. This paper tries to utilize the advantages of policy iteration and value iteration, and avoids their drawbacks at the same time. Therefore, a multi-step heuristic dynamic programming (MsHDP) method is developed for solving the optimal control problem of nonlinear discrete-time systems. MsHDP speeds up value iteration and avoids the requirement of initial admissible control policy in policy iteration at the same time. The convergence theory of MsHDP is established by proving that it converges to the solution of the Bellman equation. For implementation purpose, the actor-critic neural network (NN) structure is developed. The critic NN is employed to estimate the value function and its NN weight vector is computed with a least-square scheme. The actor NN is used to estimate the control policy and a gradient descent method is proposed for updating its NN weight vector. According to the comparative simulation studies on two examples, the effectiveness and advantages of MsHDP are verified.

Model-free Adaptive Dynamic Programming for Optimal Control of Discrete-time Affine Nonlinear System

In this paper, a model-free and effective approach is proposed to solve infinite horizon optimal control problem for affine nonlinear systems based on adaptive dynamic programming technique. The developed approach, referred to as the actor-critic structure, employs two multilayer perceptron neural networks to approximate the state-action value function and the control policy, respectively. It uses data collected arbitrarily from any reasonable sampling distribution for policy iteration. In the policy evaluation phase, a novel objective function is defined for updating the critic network, and thus makes the critic network converge to the Bellman equation directly rather than iteratively. In the policy improvement phase, the action network is updated to minimize the outputs of the critic network. The two phases alternate until no more improvement of the control policy is observed, such that the optimal control policy is achieved. Two simulation examples are provided to show the effectiveness of the approach.

Some Applications of Optimal Control in Sustainable Fishing in the Baltic Sea

Issues related to the implementation of dynamic programming for optimal control of a three-dimensional dynamic model (the fish populations management problem) are presented. They belong to a class of models called Lotka-Volterra models. The existence of bionomic equilibria will be considered. The problem of optimal harvest policy is then solved for the control of various classes of its behaviour. Therefore the focus will be the optimality conditions by using the Bellman principle. Moreover, we consider a different form for the optimal value of the control vector, namely the feedback or closed-loop form of the control. Academic examples are studied in order to demonstrate the proposed methods.

On Implementation of Dynamic Programming for Optimal Control Problems with Final State Constraints

In this paper we present issues related to the implementation of dynamic programming for optimal control of a one-dimensional dynamic model, such as the hybrid electric vehicle energy management problem. A study on the resolution of the discretized state space emphasizes the need for careful implementation. A new method is presented to treat numerical issues appropriately. In particular, the method deals with numerical problems that arise due to high gradients in the optimal cost-to-go function. These gradients mainly occur on the border of the feasible state region. The proposed method not only enhances the accuracy of the final global optimum but also allows for a reduction of the state-space resolution with maintained accuracy. The latter substantially reduces the computational effort to calculate the global optimum. This allows for further applications of dynamic programming for hybrid electric vehicles such as extensive parameter studies.

Hadamard-like derivatives in Preisach modeling and control

We define a concept of derivatives of the relative boundary of the active set, on the Preisach plane, with respect to the time variable, under a time-varying input of the piecewise monotone type. These derivatives are related to the set-valued derivatives previously introduced by these authors. This concept of derivatives, which is akin to the derivatives introduced by Hadamard in the derivation of his formula on variation of a Green's function under perturbations of the domain for a boundary-value problem, allows the representation of the Preisach evolution in terms of a generalized variant of a differential equation, which in turn can be used as a basis for dynamic programming for optimal control and game theory problems for systems with hysteresis.

Partitioned Dynamic Programming for Optimal Control

Parallel algorithms for the solution of linear-quadratic optimal control problems are described. The algorithms are based on a straightforward decomposition of the domain of the problem, and are related to multiple shooting methods for two-point boundary value problems. Their arithmetic cost is approximately twice that of the serial dynamic programming approach; however, they have the advantage that they can be efficiently implemented on a wide variety of parallel architectures. Extension to the case in which there are box constraints on the controls is simple. The algorithms can be used to solve linear-quadratic subproblems arising from the application of Newton’s method or two-metric gradient projection methods to nonlinear problems.

The Application of Automatic Control for Energy Saving in Heating Furnaces

The fuel consumption in heating furnaces can be reduced, in some cases substantially, by the optimization of process control. A system for the control of energy utilization must become the part of a conventional furnace equipment. Reasonable physical assumptions about the process of heating enables us to construct simple control algorithms. These algorithms make it practical the extensive use ot microprocessors tor optimal control of energy utilization in furnaces. The development of such algorithms is studied in this paper.The optimization of heating of a lumped batch is discussed. The technological requirements and maximum permissible temperature of the refractory lining furnish the process constraints. The problem is solved by the use of maximum principle. The optimal firing rate and variation of the furnace temperature are determined by the problem solution. The optimal control algorithms for the batch furnace are given. The thermodynamic minimum of the energy consumption for given conditions is obtained and employed as the criterion for the estimation of efficiency of a control system. Simplified algorithms are constructed for the control of furnaces when heat losses are comparatively high and comparatively small. The optimal control of heating of massive bodies is discussed.The conditions of the process linearization are considered. The minimization of fuel consumption is posed as the problem of linear programming. Dynamic programming provides the most general technique for the optimization of the heating process. The paper depicts the procedure of the use of dynamic programing for the minimization of fuel consumption in furnaces. The application of linear and dynamic programming for optimal control of continuous furnaces is discussed. The information necessary for the optimal control of fuel utilization is determined.