Direct Heuristic Dynamic Programming Research Articles

Online Reinforcement Learning Control by Direct Heuristic Dynamic Programming: From Time-Driven to Event-Driven.

In this work, time-driven learning refers to the machine learning method that updates parameters in a prediction model continuously as new data arrives. Among existing approximate dynamic programming (ADP) and reinforcement learning (RL) algorithms, the direct heuristic dynamic programming (dHDP) has been shown an effective tool as demonstrated in solving several complex learning control problems. It continuously updates the control policy and the critic as system states continuously evolve. It is therefore desirable to prevent the time-driven dHDP from updating due to insignificant system event such as noise. Toward this goal, we propose a new event-driven dHDP. By constructing a Lyapunov function candidate, we prove the uniformly ultimately boundedness (UUB) of the system states and the weights in the critic and the control policy networks. Consequently, we show the approximate control and cost-to-go function approaching Bellman optimality within a finite bound. We also illustrate how the event-driven dHDP algorithm works in comparison to the original time-driven dHDP.

Toward reliable designs of data-driven reinforcement learning tracking control for Euler–Lagrange systems

This paper addresses reinforcement learning based, direct signal tracking control with an objective of developing mathematically suitable and practically useful design approaches. Specifically, we aim to provide reliable and easy to implement designs in order to reach reproducible neural network-based solutions. Our proposed new design takes advantage of two control design frameworks: a reinforcement learning based, data-driven approach to provide the needed adaptation and (sub)optimality, and a backstepping based approach to provide closed-loop system stability framework. We develop this work based on an established direct heuristic dynamic programming (dHDP) learning paradigm to perform online learning and adaptation and a backstepping design for a class of important nonlinear dynamics described as Euler–Lagrange systems. We provide a theoretical guarantee for the stability of the overall dynamic system, weight convergence of the approximating nonlinear neural networks, and the Bellman (sub)optimality of the resulted control policy. We use simulations to demonstrate significantly improved design performance of the proposed approach over the original dHDP.

Virtual inertia control parameter regulator of doubly fed induction generator based on direct heuristic dynamic programming

The most widely applied doubly-fed induction generator (DFIG) leads to insufficient frequency regulation ability of power system. The virtual inertia control can adjust the active power of DFIG according to frequency fluctuation. However, the frequent changes in power flow and operating point bring great challenges to traditional virtual inertia controller with fixed parameters. Direct Heuristic Dynamic Programming (Direct HDP) based control parameter regulator is proposed in this paper to obtain the optimal control parameters when the operation condition changes. Compared with the initial parameters, the response has been significantly improved after adaptive optimization by the proposed method when the system operating point changes.

Robotic Knee Tracking Control to Mimic the Intact Human Knee Profile Based on Actor-Critic Reinforcement Learning

We address a state-of-the-art reinforcement learning (RL) control approach to automatically configure robotic prosthesis impedance parameters to enable end-to-end, continuous locomotion intended for transfemoral amputee subjects. Specifically, our actor-critic based RL provides tracking control of a robotic knee prosthesis to mimic the intact knee profile, This is a significant advance from our previous RL based automatic tuning of prosthesis control parameters which have centered on regulation control with a designer prescribed robotic knee profile as the target. In addition to presenting the tracking control algorithm based on direct heuristic dynamic programming (dHDP), we provide a control performance guarantee including the case of constrained inputs. We show that our proposed tracking control possesses several important properties, such as weight convergence of the learning networks, Bellman (sub) optimality of the cost-to-go value function and control input, and practical stability of the human-robot system. We further provide a systematic simulation of the proposed tracking control using a realistic human-robot system simulator, the OpenSim, to emulate how the dHDP enables level ground walking, walking on different terrains and at different paces. These results show that our proposed dHDP based tracking control is not only theoretically suitable, but also practically useful.

Self-Learning Controllers in the Oil and Gas Industry

Recently, solving the optimization-control problems by using artificial intelligence has widelyappeared in the petroleum fields in exploration and production. This paper presents the stateof-the-art reinforcement-learning algorithm applying in the petroleum optimization-controlproblems, which is called a direct heuristic dynamic programming (DHDP). DHDP has twointeractive artificial neural networks, which are the critic network (provider acritique/evaluated signal) and the actor network (provider a control signal). This paper focuseson a generic on-line learning control system in Markov decision process principles.Furthermore, DHDP is a model-free learning design that does not require prior knowledgeabout a dynamic model; therefore, DHDP can be appllied with any petroleum equipment ordevise directly without needed to drive a mathematical model. Moreover, DHDP learns byitself (self-learning) without human intervention via repeating the interaction between anequipment and environment/process. The equipment receives the states of theenvironment/process via sensors, and the algorithm maximizes the reward by selecting thecorrect optimal action (control signal). A quadruple tank system (QTS) is taken as a benchmarktest problem, that the nonlinear model responses close to the real model, for three reasons:First, QTS is widely used in the most petroleum exploration/production fields (entire system orparts), which consists of four tanks and two electrical-pumps with two pressure control valves.Second, QTS is a difficult model to control, which has a limited zone of operating parametersto be stable; therefore, if DHDP controls on QTS by itself, DHDP can control on otherequipment in a fast and optimal manner. Third, QTS is designed with a multi-input-multioutput (MIMO) model for analysis in the real-time nonlinear dynamic system; therefore, theQTS model has a similar model with most MIMO devises in oil and gas field. The overalllearning control system performance is tested and compared with a proportional integralderivative (PID) via MATLAB programming. DHDP provides enhanced performancecomparing with the PID approach with 99.2466% improvement.

Optimal Adaptive Super-Twisting Sliding-Mode Control Using Online Actor-Critic Neural Networks for Permanent-Magnet Synchronous Motor Drives

In this paper, a novel optimal adaptive-gains super-twisting sliding-mode control (OAGSTSMC) using actor-critic approach is proposed for a high-speed permanent-magnet synchronous motor (PMSM) drive system. First, the super-twisting sliding-mode controller (STSMC) is adopted for reducing the chattering phenomenon and stabilizing the PMSM drive system. However, the control performance may be destroyed due external disturbances and parameter variations of the drive system. In addition, the conservative selection of the STSMC gains may affect the control performance. Therefore, for enhancing the standard super-twisting approach performance via avoiding the constraints on knowing the disturbances as well as uncertainties upper bounds, and to achieve the drive system robustness, the direct heuristic dynamic programming (HDP) is utilized for optimal tuning of STSMC gains. Consequently, an online actor-critic algorithm with HDP is designed for facilitating the online solution of the Hamilton-Jacobi-Bellman (HJB) equation via a critic neural network while pursuing an optimal control via an actor neural network at the same time. Furthermore, based on Lyapunov approach, the stability of the closed-loop control system is assured. A real-time implementation is performed for verifying the proposed OAGSTSMC efficacy. The experimental results endorse that the proposed OAGSTSMC control approach achieves the PMSM superior dynamic performance regardless of unknown uncertainties as well as exterior disturbances.

Online Reinforcement Learning Control for the Personalization of a Robotic Knee Prosthesis.

Robotic prostheses deliver greater function than passive prostheses, but we face the challenge of tuning a large number of control parameters in order to personalize the device for individual amputee users. This problem is not easily solved by traditional control designs or the latest robotic technology. Reinforcement learning (RL) is naturally appealing. The recent, unprecedented success of AlphaZero demonstrated RL as a feasible, large-scale problem solver. However, the prosthesis-tuning problem is associated with several unaddressed issues such as that it does not have a known and stable model, the continuous states and controls of the problem may result in a curse of dimensionality, and the human-prosthesis system is constantly subject to measurement noise, environmental change and human-body-caused variations. In this paper, we demonstrated the feasibility of direct heuristic dynamic programming, an approximate dynamic programming (ADP) approach, to automatically tune the 12 robotic knee prosthesis parameters to meet individual human users' needs. We tested the ADP-tuner on two subjects (one able-bodied subject and one amputee subject) walking at a fixed speed on a treadmill. The ADP-tuner learned to reach target gait kinematics in an average of 300 gait cycles or 10 min of walking. We observed improved ADP tuning performance when we transferred a previously learned ADP controller to a new learning session with the same subject. To the best of our knowledge, our approach to personalize robotic prostheses is the first implementation of online ADP learning control to a clinical problem involving human subjects.

A New Powered Lower Limb Prosthesis Control Framework Based on Adaptive Dynamic Programming

This brief presents a novel application of adaptive dynamic programming (ADP) for optimal adaptive control of powered lower limb prostheses, a type of wearable robots to assist the motor function of the limb amputees. Current control of these robotic devices typically relies on finite state impedance control (FS-IC), which lacks adaptability to the user's physical condition. As a result, joint impedance settings are often customized manually and heuristically in clinics, which greatly hinder the wide use of these advanced medical devices. This simulation study aimed at demonstrating the feasibility of ADP for automatic tuning of the twelve knee joint impedance parameters during a complete gait cycle to achieve balanced walking. Given that the accurate models of human walking dynamics are difficult to obtain, the model-free ADP control algorithms were considered. First, direct heuristic dynamic programming (dHDP) was applied to the control problem, and its performance was evaluated on OpenSim, an often-used dynamic walking simulator. For the comparison purposes, we selected another established ADP algorithm, the neural fitted Q with continuous action (NFQCA). In both cases, the ADP controllers learned to control the right knee joint and achieved balanced walking, but dHDP outperformed NFQCA in this application during a 200 gait cycle-based testing.

Adaptive fuzzy optimal control using direct heuristic dynamic programming for chaotic discrete-time system

In this paper, we aim to solve the optimal tracking control problem for the Henon Mapping chaotic system using the direct heuristic dynamic programming (DHDP) setting with filtered tracking error. The fuzzy logic system is used to approximate the long-term utility function. Compared with the results for chaotic discrete-time system, the cost of the controller is reduced. The Lyapunov analysis approach is utilized to prove the stability of the chaotic system. It is shown that the tracking error, the adaptation law and the control input retain the property of uniformly ultimate boundedness. A simulation example is given to demonstrate the effectiveness of the proposed approach.

Direct heuristic dynamic programming based on an improved PID neural network

In this paper, an improved PID-neural network (IPIDNN) structure is proposed and applied to the critic and action networks of direct heuristic dynamic programming (DHDP). As one of online learning algorithm of approximate dynamic programming (ADP), DHDP has demonstrated its applicability to large state and control problems. Theoretically, the DHDP algorithm requires access to full state feedback in order to obtain solutions to the Bellman optimality equation. Unfortunately, it is not always possible to access all the states in a real system. This paper proposes a solution by suggesting an IPIDNN configuration to construct the critic and action networks to achieve an output feedback control. Since this structure can estimate the integrals and derivatives of measurable outputs, more system states are utilized and thus better control performance are expected. Compared with traditional PIDNN, this configuration is flexible and easy to expand. Based on this structure, a gradient decent algorithm for this IPIDNN-based DHDP is presented. Convergence issues are addressed within a single learning time step and for the entire learning process. Some important insights are provided to guide the implementation of the algorithm. The proposed learning controller has been applied to a cart-pole system to validate the effectiveness of the structure and the algorithm.

A boundedness result for the direct heuristic dynamic programming

Approximate/adaptive dynamic programming (ADP) has been studied extensively in recent years for its potential scalability to solve large state and control space problems, including those involving continuous states and continuous controls. The applicability of ADP algorithms, especially the adaptive critic designs has been demonstrated in several case studies. Direct heuristic dynamic programming (direct HDP) is one of the ADP algorithms inspired by the adaptive critic designs. It has been shown applicable to industrial scale, realistic and complex control problems. In this paper, we provide a uniformly ultimately boundedness (UUB) result for the direct HDP learning controller under mild and intuitive conditions. By using a Lyapunov approach we show that the estimation errors of the learning parameters or the weights in the action and critic networks remain UUB. This result provides a useful controller convergence guarantee for the first time for the direct HDP design.

Direct Heuristic Dynamic Programming for Nonlinear Tracking Control With Filtered Tracking Error

This paper makes use of the direct heuristic dynamic programming design in a nonlinear tracking control setting with filtered tracking error. A Lyapunov stability approach is used for the stability analysis of the tracking system. It is shown that the closed-loop tracking error and the approximating neural network weight estimates retain the property of uniformly ultimate boundedness under the presence of neural network approximation error and bounded unknown disturbances under certain conditions.

Performance Evaluation of Direct Heuristic Dynamic Programming using Control-Theoretic Measures

Approximate dynamic programming (ADP) has been widely studied from several important perspectives: algorithm development, learning efficiency measured by success or failure statistics, convergence rate, and learning error bounds. Given that many learning benchmarks used in ADP or reinforcement learning studies are control problems, it is important and necessary to examine the learning controllers from a control-theoretic perspective. This paper makes use of direct heuristic dynamic programming (direct HDP) and three typical benchmark examples to introduce a unique analytical framework that can be applied to other learning control paradigms and other complex control problems. The sensitivity analysis and the linear quadratic regulator (LQR) design are used in the paper for two purposes: to quantify direct HDP performances and to provide guidance toward designing better learning controllers. The use of LQR however does not limit the direct HDP to be a learning controller that addresses nonlinear dynamic system control issues. Toward this end, applications of the direct HDP for nonlinear control problems beyond sensitivity analysis and the confines of LQR have been developed and compared whenever appropriate to an LQR design.

Direct Heuristic Dynamic Programming for Damping Oscillations in a Large Power System

This paper applies a neural-network-based approximate dynamic programming method, namely, the direct heuristic dynamic programming (direct HDP), to a large power system stability control problem. The direct HDP is a learning- and approximation-based approach to addressing nonlinear coordinated control under uncertainty. One of the major design parameters, the controller learning objective function, is formulated to directly account for network-wide low-frequency oscillation with the presence of nonlinearity, uncertainty, and coupling effect among system components. Results include a novel learning control structure based on the direct HDP with applications to two power system problems. The first case involves static var compensator supplementary damping control, which is used to provide a comprehensive evaluation of the learning control performance. The second case aims at addressing a difficult complex system challenge by providing a new solution to a large interconnected power network oscillation damping control problem that frequently occurs in the China Southern Power Grid.

Supplementary Damping Controller Design using Direct Heuristic Dynamic Programming in Complex Power Systems

In modern, large scale interconnected power grids, low-frequency oscillation is a key roadblock to improved power transmission capacity. Supplementary generator control, flexible AC transmission system (FACTS), and high voltage direct currents (HVDC) are engineered devices designed to damp such low frequency swings. In this paper a neural network-based approximate dynamic programming method, namely direct heuristic dynamic programming (direct HDP), is applied to power system stability enhancement. Direct HDP is a learning and approximation based approach to addressing nonlinear system control problems under uncertainty, and it is also a model-free design strategy. The action and critic networks of the direct HDP are implemented using multi-layer perceptrons; learning is carried out based on the interactions between the controller and the power system. For this design approach, real time system responses are provided through wide-area measurement system (WAMS). The controller learning objective is formulated as a reward function that reflects global characteristics of the power system under low frequency oscillation, as well as tight coupling effects among system components. Direct HDP control design is illustrated by case studies, which are also used to demonstrate the learning control performance. The proposed direct HDP learning control is also developed as a new solution to a large scale system coordination problem by using the China Southern Power Grid as a major test bed.