Adaptive Dynamic Programming Controller Research Articles

Vibration Suppression of a Flexible Beam Structure Coupled with Liquid Sloshing via ADP Control Based on FBG Strain Measurement

In this study, an adaptive dynamic programming (ADP) control strategy based on the strain measurement of a fiber Bragg grating (FGB) sensor array is proposed for the vibration suppression of a complicated flexible-sloshing coupled system, which usually exists in aerospace engineering, such as launch vehicles with a large amount of liquid propellant as well as a flexible beam structure. To simplify the flexible-sloshing coupled dynamics model, the equivalent spring-mass-damper (SMD) model of liquid sloshing is employed, and a finite-element method (FEM) dynamic model for the beam structure coupled with the liquid sloshing is mathematically established. Then, a strain-based vibration dynamic model is derived by employing a transformation matrix based on the relationship between displacement and strain of the beam structure. To facilitate the design of a strain-based control, a tracking differentiator is designed to provide the strains’ derivative signals as partial states’ estimations. Feeding the system with the strain measurements and their derivatives’ estimations, an ADP controller with an action-dependent heuristic dynamic programming structure is proposed to suppress the vibration of the flexible-sloshing coupled system, and the corresponding Lyapunov stability of the closed-loop system is theoretically guaranteed. Numerical results show the proposed method can effectively suppress coupled vibration depending on limited strain measurements irrespective of external disturbances.

Pinning consensus control for switched multi-agent systems: A switched adaptive dynamic programming method

Stability is fundamental to ensure the operation of control system, but optimality is the ultimate goal to achieve the maximum performance. This paper investigates an event-triggered pinning optimal consensus control for switched multi-agent system (SMAS) via a switched adaptive dynamic programming (ADP) method. The technical contribution mainly lies in two aspects. On the one hand, in order to optimize the control performance and ensure the consensus, the switched local value function (SLVF) and the minimum-error switching law are constructed. Based on SLVF, an algorithm of switched ADP policy iteration is proposed, and its convergence and optimality are proved. On the other hand, considering that it is impractical to install a controller for each agent in reality, a pinning strategy is developed to guide the setting of the ADP controller, which can reduce the waste of control resources. A new condition is constructed to determine the minimum number of controlled vertices of the SMAS. Lastly, a numerical example is given to verify the effectiveness of the proposed method.

An Improved Adaptive Dynamic Programming Algorithm Based on Fuzzy Extended State Observer for Dissolved Oxygen Concentration Control

To solve the anti-disturbance control problem of dissolved oxygen concentration in the wastewater treatment plant (WWTP), an anti-disturbance control scheme based on reinforcement learning (RL) is proposed. An extended state observer (ESO) based on the Takagi–Sugeno (T-S) fuzzy model is first designed to estimate the the system state and total disturbance. The anti-disturbance controller compensates for the total disturbance based on the output of the observer in real time, online searches the optimal control policy using a neural-network-based adaptive dynamic programming (ADP) controller. For reducing the computational complexity and avoiding local optimal solutions, the echo state network (ESN) is used to approximate the optimal control policy and optimal value function in the ADP controller. Further analysis demonstrates the observer estimation errors for system state and total disturbance are bounded, and the weights of ESNs in the ADP controller are convergent. Finally, the effectiveness of the proposed ESO-based ADP control scheme is evaluated on a benchmark simulation model of the WWTP.

Neural-Network-Based Control With Dynamic Event-Triggered Mechanisms Under DoS Attacks and Applications in Load Frequency Control

The paper is concerned with the supplementary control based on adaptive dynamic programming (ADP) for a class of discrete-time networked system with the simultaneous presence of dynamic event-triggered mechanisms and Denial-of-Service (DoS) attacks. The dynamic behavior of DoSs is described by a model with the appropriate frequency and durations. A neural network (NN)-based observer is first designed to estimate system states in order to resolve the limitation in ADP-based control due mainly to data sparsity. The performance analysis and gain design of the NN-based observer are systematically discussed in light of the switched system theory combined with the average dwell-time method. Subsequently, the policy iteration algorithm with an actor-critic structure is developed to implement the designed supplementary ADP controller, and the corresponding condition on learning rates in weight updating rules is derived by virtue of the well-known Lyapunov stability. Finally, the effectiveness of the developed approach is demonstrated by an application in load frequency control of power systems.

Optimal trajectory-tracking guidance for reusable launch vehicle based on adaptive dynamic programming

An optimal trajectory-tracking guidance method for Reusable Launch Vehicle (RLV) based on neural Adaptive Dynamic Programming (ADP) is proposed. Firstly, a reentry reference trajectory and the corresponding steady-state control are generated based on the Gauss Pseudo-spectral Method (GPM) with saturation constraints on the amplitude, rate and acceleration of flow angles. A single-critic ADP controller is designed for optimal feedback control, which is combined with the steady-state control to realize the trajectory-tracking guidance. An innovative weight iteration algorithm for the critic neural network is proposed to reduce training computation and improve guidance accuracy. Simulation results show that with initial state errors and system uncertainty, the terminal position error is within 2.1 km in a reentry flight whose range covers more than 5500 km. Involved with the innovative weight iteration method, the training computation is reduced by 95% compared with the traditional gradient descent method and the guidance performance is also improved.

Regulating Control of In-Pipe Intelligent Isolation Plugging Tool Based on Adaptive Dynamic Programming

In-pipe intelligent isolation plugging tools (IPTs) are crucial in pipeline maintenance. During the plugging process, the flow field around the IPT changes drastically, resulting in vibration and instability of the plugging process. Therefore, three foldable spoilers were designed at the tail of the IPT to reduce the vibration of the IPT. First, a disturbing flow experiment of IPT with spoilers was designed. A mathematical model of the pneumatic spoiler control system was established to regulate the spoiler angles. Second, based on the experimental data, a bidirectional long short-term memory (Bi-LSTM) neural network predictor between the plugging states, the spoiler angles, and the pressure gradient was established. Then, an adaptive dynamic programming controller was designed to select the optimal control action for each plugging state, thereby reducing the pressure gradient. Finally, Python and MATLAB/Simulink were used for simulation. The results showed that prediction errors were controlled within 9%, and the controller could reduce the pressure gradient during the plugging process by an average of 25.94%, which alleviated the vibration of the IPT and achieved a smooth plugging operation.

Policy Gradient Adaptive Critic Design With Dynamic Prioritized Experience Replay for Wastewater Treatment Process Control

With the industrialization of modern society, the pollution of water resources becomes more and more serious. Although purifying urban sewage through the wastewater treatment plants eases the burden of fragile ecosystems, the nonlinearities and uncertainties of biochemical reactions are difficult to address. In this article, a dynamic prioritized policy gradient adaptive dynamic programming (ADP) method is developed to solve the optimal control problem of nonaffine nonlinear discrete-time systems, along with convergence analysis of the algorithm. To the best of our knowledge, it is indispensable to conduct system modeling during the previous ADP research on wastewater treatment process control. By introducing the dynamic prioritized replay buffer and neural networks, the proposed ADP controller can track the setpoints of the wastewater treatment plant and alleviate the effects of disturbance without system modeling. The test results verify that the devised control method outperforms the proportional-integral-derivative strategy with less oscillation when unknown interference occurred.

Event-Triggered Optimal Control for the Continuous Stirred Tank Reactor System

Continuous stirred tank reactor (CSTR) is essential equipment found in chemical processing industries. Control of CSTR process has long been a challenging issue due to the high complexity and strong nonlinearity of the chemical process. In this paper, a novel event-triggered (ET) adaptive dynamic programming (ADP) optimal control (OC) algorithm is developed for continuous stirred tank reactor (CSTR) system. In order to reduce the computational load and the communication data between the controller and the actuator, a new event-triggering condition based on Taylor series expansion is designed for the ET controller. The control policy does not update unless the event-triggering condition is not satisfied. Compared with the time-triggering mechanism, the update frequency of the controller can be obviously reduced. Then, an identifer-actor-critic structure is developed to implement the ET ADP controller. In particular, to overcome the challenge of establishing an exact dynamic for CSTR system, an identifier neural network (NN) is employed to reconstruct the unknown system dynamic based on offline data. Furthermore, the actor-critic structure is developed to obtain the ET control law and the value function. In actor and critic NNs, weights are turned just at the triggered instant and remained constant during the inter event times. Finally, the developed ET ADP controller is applied to the CSTR system. Experimental results show that the developed ET approach can cut down the update frequency to 66\%, which is very significant for the real CSTR system.

Adaptive dynamic programing design for the neural control of hypersonic flight vehicles

An adaptive dynamic programming controller based on backstepping method is designed for the optimal tracking control of hypersonic flight vehicles. The control input is divided into two parts namely stable control and optimal control. First, the back-stepping method is exploited via neural networks (NNs) to estimate unknown functions. Then, the computational load is reduced by the minimal-learning-parameter (MLP) scheme. To avoid the problem of “explosion of terms”, a first-order filter is adopted. Next, the optimal controller is designed based on the adaptive dynamic programming. In order to solve the Hamiltonian equation, NNs estimators are introduced to approximate performance indicators, achieving the approximate optimal control of hypersonic flight vehicles. Finally, the effectiveness and advantages of the control method are verified by simulation results.

Reinforcement Learning-Based Structural Control of Floating Wind Turbines

The structural control of floating wind turbines using active tuned mass damper is investigated in this article. To our knowledge, this is for the first time that reinforcement learning-based control approach is employed to this type of application. Specifically, an adaptive dynamic programming (ADP) algorithm is used to derive the optimal control law based on the nonlinear structural dynamics, and the large-scale machine learning platform Tensorflow is employed for the design and implementation of the neural network (NN) structure. Three fully connected NNs, i.e., a plant network, a critic network, and an action network, are included in the proposed NN structure. Their training requires the gradient information flowing through the whole network, which is tackled by automatic differentiation, a popular technique for deriving the gradients of complex networks automatically. While to our knowledge, the network structures in the existing literature are rather simple and the training of the hidden layer is usually ignored. This allows their gradients to be derived analytically, which is infeasible with complex network structures. Thus, automatic differentiation greatly improves the employed ADP algorithm’s ability in solving complex problems. The simulation results of structural control of floating wind turbines show that ADP controller performs very well in both normal and extreme conditions, with the standard deviation of the platform pitch displacement being reduced by around 40%. A clear advantage of ADP controllers over the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> controller is observed, especially in extreme conditions. Moreover, our design considers the tradeoff between the control performance and power consumption.

Online policy iteration ADP-based attitude-tracking control for hypersonic vehicles

An online adaptive dynamic programming (ADP) attitude-tracking controller based on policy iteration is proposed, aiming to approach the optimal control of hypersonic vehicles (HVs). The Bellman equation, known as the principal recursive dynamic programming formula, is provided to obtain the controller. In particular, the control action is generated by the ADP controller to track the attitude trajectory. In order to approach optimal control in the uncertain nonlinear HVs system, we use policy iteration to approximate the Bellman equation and build an actor-predictor-critic framework, in which the action network, state estimator and critic network are adopted to implement the policy iteration. Meanwhile, an offline learning method is provided to approach the initial value of iterative computations and improve the efficiency of online learning. The comparative simulations demonstrate the good performance of PIADP with aerodynamic parameter perturbations and random disturbances.

Event-Triggered Adaptive Dynamic Programming for Unmatched Uncertain Nonlinear Continuous-Time Systems

In this article, an event-triggered adaptive dynamic programming (ADP) method is proposed to solve the robust control problem of unmatched uncertain systems. First, the robust control problem with unmatched uncertainties is transformed into the optimal control design for an auxiliary system. Subsequently, to reduce controller executions and save computational and communication resources, an event-triggering mechanism is introduced. By using a critic neural network (NN) to approximate the value function, novel concurrent learning is developed to learn NN weights, which avoids the requirement of an initial admissible control and the persistence of excitation condition. Moreover, it is proven that the developed event-triggered ADP controller guarantees the robustness of the uncertain system and the uniform ultimate boundedness of the NN weight estimation error. Finally, by using the F-16 aircraft and the inverted pendulum with unmatched uncertainties as examples, the simulation results show the effectiveness of the developed event-triggered ADP method.

Optimal Control for Aluminum Electrolysis Process Using Adaptive Dynamic Programming

Optimal control of aluminum electrolysis production process (AEPP) has long been a challenging industrial issue due to its inherent difficulty in establishing an accurate dynamic model. In this paper, a novel robust optimal control algorithm based on adaptive dynamic programming (ADP) is proposed for the AEPP, where the system subjects to input constraints. First, to establish an accurate dynamic model for the AEPP system, recursive neural network (RNN) is employed to reconstruct the system dynamic using the input-output production data. To ensure input constraints are not to exceed the bound of the actuator, the optimal control problem of the AEPP is formulated under a new nonquadratic form performance index function. Then, considering the perturbation of the AEPP, the robust control problem is effectively converted to the constrained optimal control problem via system transformation. Furthermore, a single critic network framework is developed to obtain the approximate solution of the Hamilton-Jacobi-Bellman (HJB) equation. Finally, the proposed ADP controller is applied to the AEPP system to validate the effectiveness and performance.

Data-Based Optimal Tracking Control for Natural Gas Desulfurization System

Desulfurization control of natural gas has long been a challenging industrial issue owing to its inherent difficulty in establishing accurate mathematical model for the nonlinear and strong coupling process. In this paper, a data-based adaptive dynamic programming (ADP) algorithm is presented to solve optimal control for natural gas desulfurization. First, neural network (NN) is used to reconstruct the dynamics of the desulfurization system via the input and output production data. Then, an improved unscented Kalman filter (IUKF) aided ADP method is presented to solve optimal control problem for desulfurization system, where IUKF algorithm is developed as a new weight-updating strategy for the action network and the critic network. The IUKF aided algorithm can improve the convergence speed as well as the anti-interference ability of the ADP controller. Furthermore, the proposed IUKF-ADP algorithm is implemented using the heuristic dynamic programming (HDP) structure. Finally, the effectiveness of the proposed IUKF-ADP algorithm is demonstrated through experiments of the natural gas desulfurization system.

Prioritizing Useful Experience Replay for Heuristic Dynamic Programming-Based Learning Systems.

The adaptive dynamic programming controller usually needs a long training period because the data usage efficiency is relatively low by discarding the samples once used. Prioritized experience replay (ER) promotes important experiences and is more efficient in learning the control process. This paper proposes integrating an efficient learning capability of prioritized ER design into heuristic dynamic programming (HDP). First, a one time-step backward state-action pair is used to design the ER tuple and, thus, avoids the model network. Second, a systematic approach is proposed to integrate the ER into both critic and action networks of HDP controller design. The proposed approach is tested for two case studies: a cart-pole balancing task and a triple-link pendulum balancing task. For fair comparison, we set the same initial weight parameters and initial starting states for both traditional HDP and the proposed approach under the same simulation environment. The proposed approach improves the required average number of trials to succeed by 60.56% for cart-pole, and 56.89% for triple-link balancing tasks, in comparison with the traditional HDP approach. Also, we have added results of ER-based HDP for comparison. Moreover, theoretical convergence analysis is presented to guarantee the stability of the proposed control design.

Online adaptive dynamic programming based on echo state networks for dissolved oxygen control

To solve the control problem of dissolved oxygen concentration in wastewater treatment process (WWTP), an online adaptive dynamic programming (ADP) scheme based on echo state networks (ESN) is proposed. The ESN-based ADP controller contains three modules composed of echo state networks, and the three modules approximate respectively the critic index, the system model and the optimal control policy. Further, an online learning algorithm on the basis of recursive least squares (RLS) is designed to train the ESN-based ADP controller, and also the convergence of the online learning algorithm is investigated. The design and operation of the ESN-based ADP controller depend mainly on the online data observed from controlled system, and minor prior knowledge is required. The proposed ESN-based ADP controller’s performance is tested and evaluated on Benchmark Simulation Model no. 1 (BSM1)—a general WWTP benchmark. Experimental results demonstrate that the proposed controller can achieve high performance on dissolved oxygen level control in WWTP.

A new history experience replay design for model-free adaptive dynamic programming

An adaptive dynamic programming (ADP) controller is a powerful control technique that has been investigated, designed and tested in a wide range of applications for solving optimal control problems in complex systems. The performance of the ADP controller is usually obtained by long training periods because the data usage efficiency is low as it discards the samples once used. History experience, also known as experience replay, is a powerful technique showing potential to accelerate the training process of learning and control. However, the existing design of history experience cannot be directly used for the model-free ADP design, because the existing work focuses on the forward temporal difference (TD) information (e.g., state-action pair). This information is between the current time step and the future time step and will need a model network for future information prediction. This paper proposes a new history experience replay design to avoid the usage of the model network or identifier of the system/environment. Specifically, we designed the experience tuple with one step backward state-action information and the TD can be achieved by a previous time step and a current time step. In addition, a systematic approach is proposed to integrate history experience in both the critic and action networks of the ADP controller design. The proposed approach is tested for two case studies: a cart-pole balancing task and a triple-link pendulum balancing task. For fair comparison, we set the same initial starting states and initial weight parameters for both approaches under the same simulation environment. The statistical results show that the proposed approach can improve the required average number of trials to succeed as well as the success rate. In general, the proposed approach improved the required average trial to succeed by 26.5% for cart-pole and 43% for triple-link balancing tasks.

An Event-Triggered Approach for Load Frequency Control With Supplementary ADP

The modern power system is evolving towards a new generation of smart grid, with significant benefits from the latest computer-based communication network technologies. Furthermore, as the incremental deployment of phase measurements units (PMUs) and the use of Smart Meters, there will be a substantial increase of the real-time system measurements. Under this trend, event-triggered control (ETC) will play an important role in reducing the communication and computation cost. In this paper, a novel ETC architecture design for load frequency control (LFC) with supplementary adaptive dynamic programming (ADP) is presented. The primary proportional-integral (PI) controller uses different proportional and integral thresholds for updating the actions, while the supplementary ADP controller is updated in an aperiodic manner. A strategy for the parameters calculation is introduced in a systematic way, and theoretical analysis of the ultimate boundedness for the closed-loop event-triggered system is also included. Simulation studies are carried out on one-area and three-area IEEE LFC benchmarks, and the results demonstrate the efficiency and effectiveness of the proposed design.

Adaptive control of powered transfemoral prostheses based on adaptive dynamic programming.

In this study, we developed and tested a novel adaptive controller for powered transfemoral prostheses. Adaptive dynamic programming (ADP) was implemented within the prosthesis control to complement the existing finite state impedance control (FS-IC) in a prototypic active-transfemoral prosthesis (ATP). The ADP controller interacts with the human user-prosthesis system, observes the prosthesis user's dynamic states during walking, and learns to personalize user performance properties via online adaptation to meet the individual user's objectives. The new ADP controller was preliminarily tested on one able-bodied subject walking on a treadmill. The test objective was for the user to approach normative knee kinematics by tuning the FS-IC impedance parameters via ADP. The results showed the ADP was able to adjust the prosthesis controller to generate the desired normative knee kinematics within 10 minutes. In the meantime, the FS-IC impedance parameters converged at the end of the adaptive tuning procedure while maintaining the desired human-prosthesis performance. This study demonstrated the feasibility of ADP for adaptive control of a powered lower limb prosthesis. Future research efforts will address several important issues in order to validate the system on amputees. To achieve this goal, human user-centered performance objective functions will be developed, tested, and used in this adaptive controller design.

Reactive power control of grid-connected wind farm based on adaptive dynamic programming

Optimal control of large-scale wind farm has become a critical issue for the development of renewable energy systems and their integration into the power grid to provide reliable, secure, and efficient electricity. Among many enabling technologies, the latest research results from both the power and energy community and computational intelligence (CI) community have demonstrated that CI research could provide key technical innovations into this challenging problem. In this paper, a neural network based controller is presented for the reactive power control of wind farm with doubly fed induction generators (DFIG). Specifically, we investigate the on-line learning and control approach based on adaptive dynamic programming (ADP) for wind farm control and integration with the grid. This controller can effectively dampen the oscillation of the wind farm system after the ground fault of the grid. Compared to previous control strategies, this controller is on-line and “model free”, and therefore, can reduce the control complexity. Simulation studies are carried out in Matlab/Simulink and the results demonstrated the effectiveness of the ADP controller.