Adaptive Dynamic Programming Control Research Articles

Adaptive dynamic programming base on MMC device of a flexible high-altitude long endurance aircraft

Flexible high-altitude long endurance (HALE) aircraft face challenges such as the delay effect caused by aeroelasticity and low efficiency during high-altitude cruising. This paper addresses a new active control scheme that replaces traditional aerodynamic control surfaces with longitudinal and lateral moving masses and adopts an observer-based adaptive dynamic programming (ADP) control strategy, aiming to solve the attitude control of HALE aircraft with flexible wings. Firstly, considering the account of unsteady aerodynamics, attitude angular velocity, and moving masses, complete dynamic model is established for a flexible aircraft, in order to more accurately describe the state of the HALE aircraft. This model includes two moving masses to replace traditional ailerons and elevators. It is difficult to design attitude control strategies considering the delay effects and unknown dynamic disturbances of HALE aircraft. To this end, the actor-critic structure of the ADP approach is designed to achieve the near-optimal control strategy of the system, which eliminates the need for prior knowledge of model parameters. A fixed time disturbance observer (FTDO) is proposed to estimate the future tracking error information and unknown disturbances to improve the control performance of the attitude. Finally, the stability of the system is proved via the Lyapunov technique and a comprehensive simulation of HALE aircraft is provided. The simulation results show that the proposed control algorithm can enable the aircraft attitude angles to quickly response and maintain stability under gust wind. Compared with a single ADP control strategy, this paper's algorithm has advantages in response speed and error.

Data-Driven Event-Triggered Adaptive Dynamic Programming Control for Nonlinear Systems With Input Saturation.

This article is devoted to data-driven event-triggered adaptive dynamic programming (ADP) control for nonlinear systems under input saturation. A global optimal data-driven control law is established by the ADP method with a modified index. Compared with the existing constant penalty factor, a dynamic version is constructed to accelerate error convergence. A new triggering mechanism covering existing results as special cases is set up to reduce redundant triggering events caused by emergent factors. The uniformly ultimate boundedness of error system is established by the Lyapunov method. The validity of the presented scheme is verified by two examples.

Online Adaptive Dynamic Programming for Optimal Self-Learning Control of VTOL Aircraft Systems With Disturbances

In this paper, a novel online adaptive dynamic programming control algorithm is developed to solve optimal control problems for Vertical Take Off and Landing (VTOL) aircraft systems. Considering the input disturbances of the VTOL systems and adding perturbation interference term to the value function, a robust policy iteration-based ADP method will be introduced to obtain the optimal controller with an approximate optimal control approach. The convergence property is developed to ensure the value function converges to a finite neighborhood of the optimal one. The uniform ultimate boundedness (UUB) of the iterative errors will be strictly proved by Lyapunov stability theory. Finally, we will give mathematical simulation results with the developed method. Compared with sliding mode control (SMC) and linear quadratic regulator (LQR) methods, the simulation results illustrate better overall performance of the novel method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —As an effective tool to solve the optimal control problem of complex nonlinear systems, adaptive dynamic programming (ADP) can effectively overcome the computational problems caused by “curse of dimensionality”. VTOL aircraft system is a typical strongly coupled, underactuate, non-minimum phase system, and the sudden random disturbances will make the control task more challenging. Aiming at the particularity of VTOL system, a new online policy iteration-based ADP algorithm is developed to obtain optimal control with an approximate optimal control strategy in this paper. Decoupling and disturbances can be effectively avoided. convergence will be analyzed to make sure the value function converge to a finite neighborhood of the optimal value function. The UUB property of the system will be proved by the Lyapunov approach.

Event-Triggering-Learning-Based ADP Control for Post-Stall Pitching Maneuver of Aircraft.

In this article, an improved event-triggering-learning (ETL)-based adaptive dynamic programming (ADP) method for the post-stall pitching maneuver of aircraft is proposed to achieve the robust optimal control and reduce the computational cost. First, a feedforward control with the nonlinear disturbance observer (NDO) technique is designed to attenuate the adverse effects caused by the unsteady aerodynamic disturbances. Subsequently, the ADP method with a critic neural network which is constructed to approximate the value function in the Hamilton-Jacobi-Bellman equation is employed to conduct the optimal control of aircraft. In addition, to reduce the computational cost of learning, the event-triggering (ET) mechanism with an improved ET condition is applied. The Lyapunov stability theory is utilized to prove that all signals in the closed-loop control system are uniformly ultimately bounded. Finally, simulation results are presented to illustrate the effectiveness of the proposed ETL-based ADP method.

Vibration Suppression of a Flexible Beam Structure Coupled with Liquid Sloshing via ADP Control Based on FBG Strain Measurement

In this study, an adaptive dynamic programming (ADP) control strategy based on the strain measurement of a fiber Bragg grating (FGB) sensor array is proposed for the vibration suppression of a complicated flexible-sloshing coupled system, which usually exists in aerospace engineering, such as launch vehicles with a large amount of liquid propellant as well as a flexible beam structure. To simplify the flexible-sloshing coupled dynamics model, the equivalent spring-mass-damper (SMD) model of liquid sloshing is employed, and a finite-element method (FEM) dynamic model for the beam structure coupled with the liquid sloshing is mathematically established. Then, a strain-based vibration dynamic model is derived by employing a transformation matrix based on the relationship between displacement and strain of the beam structure. To facilitate the design of a strain-based control, a tracking differentiator is designed to provide the strains’ derivative signals as partial states’ estimations. Feeding the system with the strain measurements and their derivatives’ estimations, an ADP controller with an action-dependent heuristic dynamic programming structure is proposed to suppress the vibration of the flexible-sloshing coupled system, and the corresponding Lyapunov stability of the closed-loop system is theoretically guaranteed. Numerical results show the proposed method can effectively suppress coupled vibration depending on limited strain measurements irrespective of external disturbances.

Event-Triggered Output Feedback Control for a Class of Nonlinear Systems via Disturbance Observer and Adaptive Dynamic Programming

An event-triggered output feedback control approach is proposed via a disturbance observer and adaptive dynamic programming (ADP). The solution starts from constructing a nonlinear disturbance observer which only depends on the measurement of system output. A state observer is then developed based on approximation information of system dynamics via neural networks. In order to avoid continuous transmission and reduce communication burden in the closed-loop system, an event-triggered mechanism is introduced such that the control signal is updated only at specific instant that a triggered condition is violated. By virtue of the disturbance observer and state observer, an output-feedback ADP control approach then is developed, where only a critic network is employed to estimate the value function. Based on Lyapunov stability theory, stability of the closed-loop system is rigorously analyzed, and the effectiveness of the proposed control approach is verified by two simulation examples.

Adaptive dynamic programming based composite control for profile tracking with multiple constraints

A composite control method based on adaptive dynamic programming (ADP) is developed to solve the profile tracking problem of the hypersonic gliding vehicle (HGV) with multiple constraints (flight and input constraints). First, based on the framework of the standard profile guidance method, a standard drag acceleration–velocity profile is planned as the tracking target, which satisfies the flight constraints. Second, a composite controller based on ADP and dynamic surface control (DSC) is proposed to track the planned profile and satisfy both the terminal conditions and the input constraint. An ADP method is developed to realize the optimal control through a critic neural network (NN) approximation for the solution of Hamilton–Jacobi–Bellman (HJB) equation. By using DSC method, the steady control is realized. Finally, the states of the closed-loop system and the weight estimation error are proved to be uniformly ultimately bounded, and the effectiveness of the proposed composite control method is verified via simulations.

A Kernel-Based Real-Time Adaptive Dynamic Programming Method for Economic Household Energy Systems

Modern home energy management systems (HEMSs) have great flexibility of energy consumption for customers, but at the same time, bear a range of problems, such as the high system complexity, uncertainty and time-varying nature of load consumptions, and renewable sources generation. This has brought great challenges for the real-time control. To solve these problems, we propose an HEMS that integrates a kernel-based real-time adaptive dynamic programming (K-RT-ADP) with a new preprocessing short-term prediction technique. For the preprocessing short-term prediction, we propose a gated recurrent unit-bidirectional encoder representations from the transformer (GRU-BERT) model to improve the forecasting accuracy of electrical loads and renewable energy generation. In particular, we classify household appliances into the temperature-sensitive loads, human activity sensitive loads, and insensitive/constant loads. The GRU-BERT model can incorporate weather and human activity information to predict load consumption and solar generation. For real-time control, we propose and employ the K-RT-ADP HEMS based on the GRU-BERT prediction algorithm. The objective of the K-RT-ADP HEMS is to minimize the electricity cost and maximize the solar energy utilization. To enhance the nonlinear approximation ability and generalization ability of the adaptive dynamic programming (ADP) algorithm, the K-RT-ADP algorithm leverages kernel mapping instead of neural networks. Hardware-in-the-loop experiments demonstrate the superiority of the proposed K-RT-ADP HEMS over the traditional ADP control through comparison.

Learning-Based Neural Dynamic Surface Predictive Control for MMC

Reinforcement learning technique was developed recently as an interesting topic in designing adaptive optimal controllers. This technique explicitly provided a feasible solution to circumvent the “curse of dimensionality” and requiring a system model inherent in the classical dynamic programming algorithm. By virtue of this property, in our work, by introducing this technique into a predictor-based online adaptive neural dynamic surface predictive control architecture, we concentrate on a novel robust predictive control framework subject to system uncertainties. To be specific, in this presented framework, an adaptive dynamic programming control strategy utilizing a critic neural network point of view is developed to learn the optimal control policy. Our modification is able to facilitate the alleviation of performance deterioration caused by system uncertainties and enable the smooth and fast learning, while keeping the merits of the finite control-set model predictive control. Finally, the interest and applicability of the proposed control methodology are verified by performance evaluation.

An Improved Adaptive Dynamic Programming Algorithm Based on Fuzzy Extended State Observer for Dissolved Oxygen Concentration Control

To solve the anti-disturbance control problem of dissolved oxygen concentration in the wastewater treatment plant (WWTP), an anti-disturbance control scheme based on reinforcement learning (RL) is proposed. An extended state observer (ESO) based on the Takagi–Sugeno (T-S) fuzzy model is first designed to estimate the the system state and total disturbance. The anti-disturbance controller compensates for the total disturbance based on the output of the observer in real time, online searches the optimal control policy using a neural-network-based adaptive dynamic programming (ADP) controller. For reducing the computational complexity and avoiding local optimal solutions, the echo state network (ESN) is used to approximate the optimal control policy and optimal value function in the ADP controller. Further analysis demonstrates the observer estimation errors for system state and total disturbance are bounded, and the weights of ESNs in the ADP controller are convergent. Finally, the effectiveness of the proposed ESO-based ADP control scheme is evaluated on a benchmark simulation model of the WWTP.

Online and hard constrained adaptive dynamic programming algorithm for energy storage control in smart buildings

AbstractThe high computational complexity caused by static optimization is the key factor to hinder the development of energy management systems. Adaptive dynamic programming (ADP) is an effective dynamic optimization method to break through the bottleneck. However, the hard constraints of energy system have not been fully considered due to the non‐convexity and nonlinearity of the value function, which makes the theoretical analysis complicated and brings safe security problems in battery systems. In this article, a systematic online ADP control framework is proposed for smart buildings control to ensure hard constraints to be satisfied. The second‐order local expansion at the current state is used to replace the nonlinear value function to simplify the theoretical analysis with the error of the reminder term. Based on the local property of value function, a method for the determination of adaptive parameters is first designed. It is proven that the solution of adaptive parameters not only prevents over‐charged and over‐discharged of the battery but also limits the charging and discharging power of the battery to be less than the rated power. In addition, long‐short term memory (LSTM) neural networks, a type dynamic network with memory characteristics, are used for the implementation of the present algorithm instead of the static networks to help realize the algorithm online. Due to the hidden state of LSTM, the performance of the online algorithm is improved after running the energy system. Numerical results verify the effectiveness of the proposed online control algorithm.

State consensus cooperative control for a class of nonlinear multi-agent systems with output constraints via ADP approach

A state consensus cooperative adaptive dynamic programming (ADP) control strategy is proposed for a nonlinear multi-agent system (MAS) with output constraints. On the basis of the transformation function, state models of leader and followers are transformed into affine ones. By using a monotonically increasing mapping function, the state-consensus cooperative control problem for an MAS with output constraints is equivalently transformed into a cooperative approximately optimal control one for an affine MAS. Then, a neural network observer is constructed for estimation of inner states, and, by graph theory and ADP method, the state consensus cooperative ADP control strategy is developed. The proposed strategy guarantees the performance index of the transformed system is approximately optimal. Furthermore, the stability analysis of whole closed-loop system is presented. Through the Lyapunov Theorem, we prove that the states of the MAS achieve consensus and the output signals of the followers satisfy the constraints. Also, all signals of the closed-loop MAS are bounded, and the trajectory of the leader node is cooperative bounded. The theoretical analysis and effectiveness of the strategy are verified by both a physical and a numerical example.

Data-driven active flutter control of airfoil with input constraints based on adaptive dynamic programming method

The stable flight region can be extended by adding control flap at the wing trailing edge and combined with active control technology. We studied the active flutter control by considering the input constraints. By designing the data-driven optimal controller, the limit cycle oscillations of a typical two-dimensional airfoil wing can be suppressed with single trailing edge control surface. The traditional control methods always need a precise mathematical model of the system, which put high requirements on system modeling. In this study, a novel data-driven optimal controller is proposed by using the input–output data and without depending on the nonlinear system dynamic model. This model-free approach avoids the effects of modeling errors and system uncertainty. When the data-driven controller is applied, the limit cycle oscillation phenomenon of the airfoil wing is eliminated within several seconds. It can be seen from the numerical simulation result that the data-driven adaptive dynamic programming control method possess superiority and feasibility.

H∞ Tracking Control of Discrete-Time System With Delays via Data-Based Adaptive Dynamic Programming

In this article, the H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control problem for a class of discrete time-delay systems is studied using databased adaptive dynamic programming (ADP) algorithm. First, the controlled system and the reference system are combined to form an augmented discrete time-delay system. Second, we transform it into a discrete time-delay system represented by system measured data, which allows the system state to be completely replaced by the data generated during system operation. Third, a novel data-based Bellman equation is derived according to the Bellman optimality principle. Then, an ADP-based H <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">∞</sub> tracking control method is designed by means of the measured data. The simulation example demonstrates the effectiveness of the data-based ADP control method proposed in this article.

Event-triggered ADP control of a class of non-affine continuous-time nonlinear systems using output information

An event-triggered adaptive dynamic programming (ADP) approach is proposed for a class of non-affine continuous-time nonlinear systems with unknown internal states. A neural networks (NNs)-based observer is designed to reconstruct internal states of the system using output information, and then, by the estimation signals, an output feedback ADP control approach is developed in an event-triggered manner. The proposed approach samples the states and updates the control signal only when the triggered condition is violated, and critic NNs are designed to approximate the performance index. Compared with the traditional ADP one under a fixed sampling mechanism, the event-triggered control approach reduces the computation resource and transmission load in the learning process. The stability analysis of the closed-loop system is provided based on the Lyapunov’s theorem. Two simulation results also verify the theoretical claims.

Learning Human–Robot Interaction for Robot-Assisted Pedestrian Flow Optimization

Due to the fast-is-slower phenomenon in emergency escape, it is desirable to regulate pedestrian flows at the exit or a bottleneck. We propose a new robot-assisted pedestrian regulation and study passive human–robot interaction (HRI). A learning-based motion control approach is presented for a robot to efficiently interact with pedestrians for desirable collective motion. We first formulate the problem into an optimal control framework using the pedestrian dynamics description based on existing social force models with embedded HRI forces. To solve the defined optimal control problem, we propose an adaptive dynamic programming (ADP) approach to provide adjustable motion parameters of the robot to efficiently interact with pedestrians so that the regulated pedestrian flow tracks a desired velocity. The ADP control process only uses observed flow information rather than the models of pedestrians, and the ADP method provides feedback control with online learning and control capability. Simulation results demonstrate that the proposed approach can regulate pedestrian flows to desirable speeds by online learning.

Event-Triggered Globalized Dual Heuristic Programming and Its Application to Networked Control Systems

Networked control systems (NCSs) provide many benefits, such as higher control accuracy and better robustness with the successively increasing computational complexity and communication burden. This results in the traditional adaptive dynamic programming control method having difficulty meeting the real-time requirements of industrial systems. In this paper, a novel event-triggered globalized dual heuristic programming method is proposed to reduce the required samples while guaranteeing the stability of the system. In the proposed method, the NCSs can communicate and update the control law only when the designed event-triggered condition is violated. Furthermore, the Elman neural network, which is a dynamic feedback network with a memory function is implemented to reconstruct the state variables as an approximator, and it depends only on the input and output data. To obtain fewer event-triggered times, two optimization methods, i.e., the unscented Kalman filter and the multiobjective quantum particle swarm optimization, are used to optimize the initial weights of the networks and the positive constant in the event-triggered condition, respectively. The simulation results on industrial system of aluminum electrolysis production are included to verify the performance of the controller.

Fuzzy adaptive optimal control for nonlinear switched systems with actuator hysteresis

SummaryThis article considers the issue of fuzzy adaptive dynamic programming control of nonlinear switched systems with arbitrary switchings and unknown uncertain functions and actuator hysteresis nonlinearities. The whole control approach is made of switching feedforward controller and optimal switching feedback controller. To get over the hardness of arbitrary switching structure and the issue of “explosion of complexity”, the common Lyapunov function theory and dynamic surface control method are utilized in the recursive design technique. By using fuzzy logic systems to model unknown inner dynamics and unknown cost functions, a novel fuzzy adaptive optimal switching control strategy is developed. Meanwhile, uniformly ultimately boundedness of all weights in the controlled systems are proved by the proposed control method, and the tracking performance is guaranteed in an optimal manner. Subsequently, a numerical simulation study is used to test the effectiveness of the presented control strategy.

A New Powered Lower Limb Prosthesis Control Framework Based on Adaptive Dynamic Programming

This brief presents a novel application of adaptive dynamic programming (ADP) for optimal adaptive control of powered lower limb prostheses, a type of wearable robots to assist the motor function of the limb amputees. Current control of these robotic devices typically relies on finite state impedance control (FS-IC), which lacks adaptability to the user's physical condition. As a result, joint impedance settings are often customized manually and heuristically in clinics, which greatly hinder the wide use of these advanced medical devices. This simulation study aimed at demonstrating the feasibility of ADP for automatic tuning of the twelve knee joint impedance parameters during a complete gait cycle to achieve balanced walking. Given that the accurate models of human walking dynamics are difficult to obtain, the model-free ADP control algorithms were considered. First, direct heuristic dynamic programming (dHDP) was applied to the control problem, and its performance was evaluated on OpenSim, an often-used dynamic walking simulator. For the comparison purposes, we selected another established ADP algorithm, the neural fitted Q with continuous action (NFQCA). In both cases, the ADP controllers learned to control the right knee joint and achieved balanced walking, but dHDP outperformed NFQCA in this application during a 200 gait cycle-based testing.

An Event-Triggered ADP Control Approach for Continuous-Time System With Unknown Internal States.

This paper proposes a novel event-triggered adaptive dynamic programming (ADP) control method for nonlinear continuous-time system with unknown internal states. Comparing with the traditional ADP design with a fixed sample period, the event-triggered method samples the state and updates the controller only when it is necessary. Therefore, the computation cost and transmission load are reduced. Usually, the event-triggered method is based on the system entire state which is either infeasible or very difficult to obtain in practice applications. This paper integrates a neural-network-based observer to recover the system internal states from the measurable feedback. Both the proposed observer and the controller are aperiodically updated according to the designed triggering condition. Neural network techniques are applied to estimate the performance index and help calculate the control action. The stability analysis of the proposed method is also demonstrated by Lyapunov construct for both the continuous and jump dynamics. The simulation results verify the theoretical analysis and justify the efficiency of the proposed method.