Globalized Dual Heuristic Programming Research Articles

Efficient Online Globalized Dual Heuristic Programming With an Associated Dual Network.

Globalized dual heuristic programming (GDHP) is the most comprehensive adaptive critic design, which employs its critic to minimize the error with respect to both the cost-to-go and its derivatives simultaneously. Its implementation, however, confronts a dilemma of either introducing more computational load by explicitly calculating the second partial derivative term or sacrificing the accuracy by loosening the association between the cost-to-go and its derivatives. This article aims at increasing the online learning efficiency of GDHP while retaining its analytical accuracy by introducing a novel GDHP design based on a critic network and an associated dual network. This associated dual network is derived from the critic network explicitly and precisely, and its structure is in the same level of complexity as dual heuristic programming critics. Three simulation experiments are conducted to validate the learning ability, efficiency, and feasibility of the proposed GDHP critic design.

Convergence analysis of the deep neural networks based globalized dual heuristic programming

Globalized dual heuristic programming (GDHP) algorithm is a special form of approximate dynamic programming (ADP) method that solves the Hamilton–Jacobi–Bellman (HJB) equation for the case where the system takes control-affine form subject to the quadratic cost function. This study incorporates the deep neural networks (DNNs) as a function approximator to inherit the advantages of which to express high-dimensional function space. Elementwise error bound of the costate function sequence is newly derived and the convergence property is presented. In the approximated function space, uniformly ultimate boundedness (UUB) condition for the weights of the general multi-layer NNs weights is obtained. It is also proved that under the gradient descent method for solving the moving target regression problem, UUB gradually converges to the value, which exclusively contains the approximation reconstruction error. The proposed method is demonstrated on the continuous reactor control in aims to obtain the control policy for multiple initial states, which justifies the necessity of DNNs structure for such cases.

Approximately Optimal Control of Discrete-Time Nonlinear Switched Systems Using Globalized Dual Heuristic Programming

Based on the idea of data-driven control, a novel iterative adaptive dynamic programming (ADP) algorithm based on the globalized dual heuristic programming (GDHP) technique is used to solve the optimal control problem of discrete-time nonlinear switched systems. In order to solve the Hamilton–Jacobi–Bellman (HJB) equation of switched systems, the iterative ADP method is proposed and the strict convergence analysis is also provided. Three neural networks are constructed to implement the iterative ADP algorithm, where a novel model network is designed to identify the system dynamics, a critic network is used to approximate the cost function and its partial derivatives, and an action network is provided to obtain the approximate optimal control law. Two simulation examples are described to illustrate the effectiveness of the proposed method by comparing with the heuristic dynamic programming (HDP) and dual heuristic programming (DHP) methods.

Deep reinforcement learning based finite-horizon optimal tracking control for nonlinear system

Abstract Reinforcement learning (RL) can be used to obtain an approximate numerical solution to the Hamilton-Jacobi-Bellman (HJB) equation. Recent advances in machine learning community enable the use of deep neural networks (DNNs) to approximate high-dimensional nonlinear functions as those that occur in RL, accurately without any domain knowledge. In the standard RL setting, both system and cost structures are unknown, and the amount of data needed to obtain an accurate approximation can be impractically large. Meanwhile, when the structures are known, they can be used to solve the HJB equation efficiently. Herein, the model-based globalized dual heuristic programming (GDHP) is proposed, in which the HJB equation is separated into value, costate, and policy functions. A particular class of interest in this research is finite horizon optimal tracking control (FHOC) problem. Additional issues that arise, such as time-varying functions, terminal constraints, and delta-input formulation, are addressed in the context of FHOC. The DNN structure and training algorithm suitable for FHOC are presented. A benchmark continuous reactor example is provided to illustrate the proposed approach.

Iterative GDHP-based approximate optimal tracking control for a class of discrete-time nonlinear systems

In this paper, an iterative globalized dual heuristic programming (GDHP) method is developed to deal with the approximate optimal tracking control for a class of discrete-time nonlinear systems. The optimal tracking control problem is formulated by solving the discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation. Then, it is approximately solved by the developed iterative GDHP-based algorithm with convergence analysis. The iterative GDHP algorithm is implemented by constructing three neural networks to approximate the error system dynamics, the cost function with its derivative, and the control policy in each iteration, respectively. The information of the cost function and its derivative is provided during iteration calculation. Two simulation examples are investigated to verify the performance of the proposed approximate optimal tracking control approach.

Simple and fast calculation of the second-order gradients for globalized dual heuristic dynamic programming in neural networks.

We derive an algorithm to exactly calculate the mixed second-order derivatives of a neural network's output with respect to its input vector and weight vector. This is necessary for the adaptive dynamic programming (ADP) algorithms globalized dual heuristic programming (GDHP) and value-gradient learning. The algorithm calculates the inner product of this second-order matrix with a given fixed vector in a time that is linear in the number of weights in the neural network. We use a "forward accumulation" of the derivative calculations which produces a much more elegant and easy-to-implement solution than has previously been published for this task. In doing so, the algorithm makes GDHP simple to implement and efficient, bridging the gap between the widely used DHP and GDHP ADP methods.

Accommodating controller malfunctions through fault tolerant control architecture

In previous work we have proposed a supervised globalized dual heuristic programming (GDHP) controller as a solution to the fault tolerant control (FTC) problem of nonlinear plants subject to abrupt and incipient faults capable of drastically modifying the system dynamics to maintain stability and performance. The neural network (NN) based adaptive critic controller presented the best choice for the flexibility and power necessary to accomplish the task, however no success guarantees can be made for the online training of neural weights for the unrestricted fault recovery problem. Built on the existing framework, we propose a novel supervisory system capable of detecting controller malfunctions before the stability of the plant is compromised. Furthermore, due to its ability to discern between controller malfunctions and faults within the plant, the proposed supervisor acts in a specific fashion in the event of a controller malfunction to provide new avenues with a greater probability of convergence using information from a dynamic model bank. The classification and distinction of controller malfunctions from the faults in the plant itself is achieved through an advanced decision logic based on three independent quality indexes. Proof-of-the-concept simulations over a nonlinear plant demonstrate the validity of the approach.

Improving the Performance of Globalized Dual Heuristic Programming for Fault Tolerant Control Through an Online Learning Supervisor

An advanced reconfigurable controller enhanced by a multiple model architecture is proposed as a tool to achieve fault tolerance in complex nonlinear systems. The most complete adaptive critic design, globalized dual heuristic programming (GDHP), constitutes a highly flexible nonlinear adaptive controller responsible for the generation of new control solutions for novel plant dynamics introduced by unknown faults. The main contribution of the presented work focuses on a novel fault tolerant control supervisor. Working on a higher hierarchical level, the proposed supervisor makes use of two quality indices to perform fault detection, identification and isolation based on the knowledge stored in a dynamic model bank (DMB). In the event of abrupt known faults, such knowledge is then used to greatly reduce the reconfiguration time of the GDHP controller. The synergy of the proposed supervisor and GDHP goes beyond, as solutions designed by the controller to previously unknown faults are autonomously added to the model bank. The fine interrelations between the algorithm's subsystems and its advanced capabilities are illustrated through extensive numerical simulations of a single-input single-output (SISO) linear system and of a multiple-input multiple-output (MIMO) nonlinear system, both subject to a series of fault scenarios involving expected and unexpected, abrupt and incipient faults. Note to Practitioners-The raising complexity of physical plants and control missions inevitably leads to increasing occurrence, diversity and severity of faults. Take automated production process as an example, the extent of time a plant is capable of maintaining acceptable performance levels is now considered to be the single factor with the highest impact on profitability. For a growing number of plants, it has become impractical to list all possible fault scenarios in order to take the necessary steps to assure continuous healthy operation. Therefore, it is now essential to have a control algorithm dedicated to the provision of tailored control solutions capable of maintaining stability and as much performance as possible during the occurrence of faults. This is the goal of fault tolerant control. In the presented paper, the most complete adaptive critic design, globalized dual heuristic programming (GDHP), is responsible for the generation of new control solutions for novel plant dynamics introduced by faults unknown at design time. A highly flexible nonlinear adaptive controller, GDHP is capable of dealing with both abrupt and incipient (gradually changing dynamics) faults. Working on a higher hierarchical level, a novel fault supervisor is introduced which makes use of two quality indices to perform fault detection, identification and isolation based on the knowledge stored in a dynamic model bank (DMB). In the event of abrupt known faults, such knowledge is then used to greatly reduce the convergence time of the GDHP controller. The synergy of the proposed supervisor and GDHP goes further, as solutions designed by the controller to previously unknown faults are autonomously added to the model bank in fact allowing the supervisor to learn new fault scenarios and their solutions as they occur. The fine interrelations between the algorithm's subsystems and its advanced capabilities are illustrated through extensive numerical simulations.

Adaptive critic designs

We discuss a variety of adaptive critic designs (ACDs) for neurocontrol. These are suitable for learning in noisy, nonlinear, and nonstationary environments. They have common roots as generalizations of dynamic programming for neural reinforcement learning approaches. Our discussion of these origins leads to an explanation of three design families: heuristic dynamic programming, dual heuristic programming, and globalized dual heuristic programming (GDHP). The main emphasis is on DHP and GDHP as advanced ACDs. We suggest two new modifications of the original GDHP design that are currently the only working implementations of GDHP. They promise to be useful for many engineering applications in the areas of optimization and optimal control. Based on one of these modifications, we present a unified approach to all ACDs. This leads to a generalized training procedure for ACDs.