Adaptive Dynamic Programming Approach Research Articles

Robust optimal control for uncertain wheeled mobile robot based on reinforcement learning: ADP approach

This paper presents a robust optimal control approach for the wheel mobile robot system, which considers the effects of external disturbances, uncertainties, and wheel slipping. The proposed method utilizes an adaptive dynamic programming (ADP) technique in conjunction with a disturbance observer. Initially, the system's state space model is formulated through the utilization of kinematic and dynamic models. Subsequently, the ADP method is employed to establish an online adaptive optimal controller, which solely relies on a single neural network for the purpose of function approximation. The utilization of the disturbance observer in conjunction with the compensation controller serves to alleviate the effects of disturbances. The Lyapunov theorem establishes the stability of the complete closed-loop system and the convergence of the weights of the neural network. The proposed approach has been shown to be effective through simulation under the effect of the disturbances and the change of the desired trajectory.

Optimal captured power control of variable speed wind turbine systems: Adaptive dynamic programming approach

SummaryAn adaptive optimal controller is proposed in this paper to maximize the captured power of a variable speed wind power system. The proposed controller is a combination of optimal and adaptive control components. The Adaptive Dynamic Programming technique is used to design the optimal control component to overcome the nonlinear problem of system dynamics and ensure stability. While the neural network is used to approximate unknown disturbances and system uncertainties. After that, the adaptive control component fully compensates for the effects of these unknown elements. Neither optimal nor adaptive control components necessitate prior knowledge of system dynamics. Furthermore, the approximation network updates only the weight matrix norm rather than the weight matrix of the neural network in each interval time, which significantly reduces computation. The stability analysis of the closed‐loop system is obtained using Lyapunov stability theory. The correctness and robustness of the control scheme are validated in two different scenarios using MATLAB/Simulink. The presented robust adaptive optimal controller is also compared to other existing controllers to demonstrate its benefits.

An Effective PQ-Decoupling Control Scheme Using Adaptive Dynamic Programming Approach to Reducing Oscillations of Virtual Synchronous Generators for Grid Connection With Different Impedance Types

An Effective <i>PQ</i>-Decoupling Control Scheme Using Adaptive Dynamic Programming Approach to Reducing Oscillations of Virtual Synchronous Generators for Grid Connection With Different Impedance Types

Min–max adaptive dynamic programming for zero-sum differential games

In this paper, a min–max adaptive dynamic programming approach is studied for a zero-sum differential game problem with unknown nonlinear dynamics. The unknown nonlinear dynamics is learned through the online Recurrent Neural Network (RNN). The error between the learned RNN model and the dynamics of the system is proved to converge to zero. The policy iteration algorithm is utilised to solve the Hamilton–Jacobi–Isaacs (HJI) equation associated with the differential game. Furthermore, the value function in the HJI equation is approximated through a critical Neural Network with its weights and activation functions updated online iteratively. The Uniform Ultimate Bounded stability of the closed-loop system is proved based on the Lyapunov theory. Finally, the effectiveness of the proposed solution method is demonstrated by applying it to three examples.

Guaranteed cost tracking control of constrained input nonlinear uncertain systems using event-based ADP

This article investigates the guaranteed cost robust tracking control problem of nonlinear systems subjected to input constraint and unmatched uncertainty. The event-based adaptive dynamic programming (ADP) approach is utilized to address this problem. First, the tracking error and reference trajectory are combined to form an augmented uncertain system. Then, by decomposing the uncertainty into the matched and unmatched parts, the original tracking problem is converted into the optimal regulation problem of an auxiliary system. The cost function for the auxiliary system is defined, and the associated Hamilton–Jacobi–Bellman (HJB) equation is solved using a single critic neural network (NN). Moreover, a novel event-triggering rule is formulated, and it is shown that the designed event-based controller guarantees that the tracking error is uniformly ultimately bounded. The derivation of event-based guaranteed cost and its relation with the time-based counterpart is presented. The exclusion of the infamous Zeno behavior is guaranteed. The uniform ultimate boundedness of the critic weight estimation error is shown. Finally, the effectiveness of the proposed event-triggered ADP method is illustrated through simulations of the spring–mass–damper system and Van der Pol’s oscillator with unmatched uncertainty.

Event-Based Robust Optimal Consensus Control for Nonlinear Multiagent System With Local Adaptive Dynamic Programming.

This article investigates the robust optimal consensus for nonlinear multiagent systems (MASs) through the local adaptive dynamic programming (ADP) approach and the event-triggered control method. Due to the nonlinearities in dynamics, the first part defines a novel measurement error to construct a distributed integral sliding-mode controller, and the consensus errors can approximately converge to the origin in a fixed time. Then, a modified cost function with augmented control is proposed to deal with the unmatched disturbances for the event-based optimal consensus controller. Specifically, a single network local ADP structure with novel concurrent learning is presented to approximate the optimal consensus policies, which guarantees the robustness of the MASs and the uniform ultimate boundedness (UUB) of the neural network (NN) weights' estimation error and relaxes the requirement of initial admissible control. Finally, an illustrative simulation verifies the effectiveness of the method.

Robust adaptive fault-tolerant control for path maneuvering of autonomous surface vehicles with actuator faults based on the noncooperative game strategy

This paper investigates a parameterized path-guided following/maneuvering control problem of an autonomous surface vehicle (ASV) subject to actuator faults via a noncooperative game strategy. Unlike the existing path maneuvering control methods, the noncooperative game of this paper includes two sub-games. On one hand, the path update is considered to be against the effort of kinematic control. On the other hand, the kinetic control is considered to be against the total disturbances consisting of actuator faults and external disturbances. A robust adaptive fault-tolerant path maneuvering controller is designed based on a noncooperative game approach. Specifically, we design a kinematic control law using an improved dynamic surface control approach to achieve the geometric objective. An improved neural predictor is constructed, in which a concurrent learning-based adaptation is designed to identify unknown fault coefficients. A path update law and a kinetic control law are calculated to cover noncooperative games by using an adaptive dynamic programming approach. Theoretical analysis shows that the closed-loop system is input-to-state stable, and the proposed method can meet the geometric objective, the dynamic objective, and the fault-tolerant objective. Finally, simulation results demonstrate the efficacy of the proposed robust adaptive fault-tolerant control method for path maneuvering.

Fault-Tolerant Multiplayer Tracking Control for Autonomous Vehicle via Model-Free Adaptive Dynamic Programming

This article investigates the completely unknown autonomous vehicle tracking issues with actuator faults through model-free adaptive dynamic programming (MFADP) approaches. Because partial parameters are measured difficultly or inaccurately, the model-based control theories are imperfect for the vehicles. Therefore, the proposed multiplayer optimal control method in this work, which is not necessary to know the prior system knowledge, achieves the purpose of unknown vehicle tracking control via a novel MFADP theory. Besides, the control strategies are robust, which contain adaptive regulators to eliminate the disturbance of the vehicle systems caused by actuator faults, modeling errors, and curvature interference. To reduce the computational burden of control, a single neural network (NN) architecture is constructed with minimal computational cost and fast response speed. In addition, the convergence analysis of the NN structure, the stability and robustness analysis of identification, and the control schemes in this work are supplied. Finally, two driving scenario simulations are shown to prove the effectiveness of the established controller.

Model-free frequency regulation in islanded microgrids: An event-triggered adaptive dynamic programming approach

This paper is concerned with frequency regulation in islanded microgrids via an event-triggered adaptive dynamic programming method. Comparatively to the existing works, we relax the typical limitations on the system, namely, exactly known models, constant load impedances, and single operation mode. The adopted frequency regulation scheme can reach frequency synchronization within the acceptable bounds in the presence of the model uncertainties, operation mode switching, and time-varying loads with no need for any prior information on the system parameters. By employing an event-triggered scheme, the secondary controller is updated only at the aperiodic triggering instants which contributes to less transmission cost. Besides, the use of measurements can help the adaptive weights to be adjusted online in the approximation process, which allows the proposed controller to adapt to the disturbances in real time. Finally, tests including the islanding process and load variation are performed in MATLAB/SimPowerSystem to show the effectiveness of the proposed method.

Neural-Network-Based Immune Optimization Regulation Using Adaptive Dynamic Programming

This article investigates optimal regulation scheme between tumor and immune cells based on the adaptive dynamic programming (ADP) approach. The therapeutic goal is to inhibit the growth of tumor cells to allowable injury degree and maximize the number of immune cells in the meantime. The reliable controller is derived through the ADP approach to make the number of cells achieve the specific ideal states. First, the main objective is to weaken the negative effect caused by chemotherapy and immunotherapy, which means that the minimal dose of chemotherapeutic and immunotherapeutic drugs can be operational in the treatment process. Second, according to the nonlinear dynamical mathematical model of tumor cells, chemotherapy and immunotherapeutic drugs can act as powerful regulatory measures, which is a closed-loop control behavior. Finally, states of the system and critic weight errors are proved to be ultimately uniformly bounded with the appropriate optimization control strategy and the simulation results are shown to demonstrate the effectiveness of the cybernetics methodology.

Model free optimal control of unknown nonaffine nonlinear systems with input quantization and DoS attack

This paper is devoted to model free optimal control of unknown nonaffine nonlinear systems with input quantization and DoS attacks. Without model information, the system is presented by a modified compact form utilizing the quantized and attacked input. With this presentation, an optimisation criterion is used to approximate the unknown pseudo partial derivative parameter. Resorting to the adaptive dynamic programming approach, a single neural network-based weighting estimation law with variable learning rate is constructed to approximate the optimal cost function. Based on the approximated parameter and cost, an optimal control law is derived by applying the stationary condition. Sufficient conditions are established to make weighting approximation error and system state be uniformly ultimately bounded. The validity of the proposed model free optimal strategy is verified by illustrative examples.

Dynamic event-triggered adaptive control for zero-sum games of nonlinear stochastic systems

This paper considers solving the zero-sum games (ZSGs) of nonlinear stochastic systems under the architecture of dynamic event-triggered mechanism (DETM). The adaptive dynamic programming (ADP) approach with great approximate computing power is proposed to seek the saddle-point equilibrium solution of ZSGs. First, the Hamilton–Jacobi–Isaacs equation of stochastic version is constructed which is the key to solving the two-player ZSGs. Then, the ADP method is employed to search the approximated solution under the dynamic triggering mechanism. On the foundation of static event-triggered mechanism (SETM), the DETM is derived through introducing an internal dynamic invariable to further lower the update frequency for the strategies. Additionally, the stability of the closed system is demonstrated in detail via the Lyapunov theory. Finally, two representative examples are provided to present the availability and reliability of this approach from different perspectives.

Designing an optimal ship course-keeping system for an unknown object model via adaptive dynamic programming approach

The paper discusses the problem of ship autopilot design based on the adaptive (approximate) dynamic programming approach (ADP). The main objective of this paper is to propose the design of a ship's course controller within the framework of computational intelligence techniques, i.e. (in our case) based on model-free optimal regulation. This method is attractive in that it does not require knowledge of the system model to generate optimal process control. The key idea is to iteratively solve the Hamilton-Jacobi-Bellman (HJB) equation based on a simple value function approximator structure. In this way, by introducing such basic elements as evaluation and improvement, we are able to carry out an intelligent optimization process based only on the data received online. Thus, this is a kind of data-driven control methodology. The simulations of the ship course change process confirmed the high efficiency of the proposed controller. An important feature of this approach is also the elimination of the tuning problem.

Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming

&lt;abstract&gt;&lt;p&gt;The fault-tolerant control problem of a hydraulic servo actuator in the presence of actuator faults is studied utilizing adaptive dynamic programming. This task is challenging because of unknown system dynamics, uncertain disturbances or unmeasurable system states of such highly nonlinear systems in real applications. The aim is to achieve asymptotic tracking and actuator faults compensation by minimizing some predefined performance index. The discrete-time algebraic Riccati equation is iteratively solved by the adaptive dynamic programming approach. For practical reasons, adaptive dynamic programming techniques and fault compensation are integrated to iteratively compute an approximated optimal fault-tolerant control using real-time input/output data without any a priori knowledge of the system dynamics and unmeasurable states. As a result, a fault-tolerant control of hydraulic servo actuator is then designed based on adaptive dynamic programming via output feedback. Also, the convergence analysis of a data-driven fault-tolerant control is theoretically shown as well. Finally, intensive simulation results are given to prove the validity and merits of the developed data-driven fault-tolerant control strategy.&lt;/p&gt;&lt;/abstract&gt;

ADP-Based Attitude-Tracking Control with Prescribed Performance for Hypersonic Vehicles

This paper proposes an observer-based adaptive dynamic programming (ADP) approach to handle the optimal attitude-tracking control tasks for hypersonic vehicles with prescribed performance constraints. Herein, the biggest challenge is how to design an optimal attitude-tracking controller which could balance performance constraints and optimization capabilities with the presence of partially unknown dynamics and actuator saturation. To this end, the transformation function is designed to convert the constrained tracking error into the equivalent unconstrained one. After that, a constraint-aware neural network-based observer is proposed to estimate the future tracking error information required by the optimal controller, which avoids the requirement of the prior nonlinear dynamics knowledge, and its estimation errors are compensated through online feedback. Taking the estimated future attitude-tracking error into account, the critic-only ADP approach is designed via the value iteration to achieve the near-optimal control strategy for equivalent systems with a simplified optimized control structure. Furthermore, a novel experience replay-driven weight updating law is provided to circumvent the conventional persistent excitation condition with an online-verification condition, and the algorithm convergence and closed-loop stability are discussed with the help of the Lyapunov technique and induction. Finally, a comprehensive simulation of an attitude-tracking control system for hypersonic vehicles is provided to verify the effectiveness and superiority of the proposed approach.

Adaptive Learning Security Control for Networked Switched Systems Subject to Resource Constraints and Attacks

We employ the adaptive dynamic programming (ADP) approach with a resilient event-triggering mechanism to handle the optimal control problems for a switched system, which is constrained by network resources and vulnerable to denial-of-service (DoS) attacks. The proposed approach is able to guarantee satisfactory system performances even when the data transmission is interrupted intermittently. A key step is to develop an updating method for the neural network (NN) weights of ADP in response to the triggered events and experienced attacks. The consideration of lowering computational cost and mitigating attack influences is integrated into the design of the system switching law for which a cost function is utilized to reduce unnecessary switching. In fact, the optimal control policy and optimal switching policy can be obtained as the outcome of the converging ADP iterative process. Furthermore, the uniformly ultimately boundedness of system state and NN weights is proven; more importantly, we illustrate in the dynamical processes, how the event-triggering, system switching, and DoS attacks affect one another. Finally, a numerical example is provided to verify the effectiveness of the proposed method.

Data-driven control of hydraulic servo actuator: An event-triggered adaptive dynamic programming approach.

Hydraulic servo actuators (HSAs) are often used in the industry in tasks that request great power, high accuracy and dynamic motion. It is well known that an HSA is a highly complex nonlinear system, and that the system parameters cannot be accurately determined due to various uncertainties, an inability to measure some parameters and disturbances. This paper considers an event-triggered learning control problem of the HSA with unknown dynamics based on adaptive dynamic programming (ADP) via output feedback. Due to increasing practical application of the control algorithm, a linear discrete model of HSA is considered and an online learning data driven controller is used, which is based on measured input and output data instead of unmeasurable states and unknown system parameters. Hence, the ADP-based data driven controller in this paper requires neither the knowledge of the HSA dynamics nor exosystem dynamics. Then, an event-based feedback strategy is introduced to the closed-loop system to save the communication resources and reduce the number of control updates. The convergence of the ADP-based control algorithm is also theoretically shown. Simulation results verify the feasibility and effectiveness of the proposed approach in solving the optimal control problem of HSAs.

Adaptive, Optimal, Virtual Synchronous Generator Control of Three-Phase Grid-Connected Inverters Under Different Grid Conditions—An Adaptive Dynamic Programming Approach

This article proposes an adaptive, optimal, data-driven control approach based on reinforcement learning and adaptive dynamic programming to the three-phase grid-connected inverter employed in virtual synchronous generators (VSGs). This article takes into account unknown system dynamics and different grid conditions, including balanced/unbalanced grids, voltage drop/sag, and weak grids. The proposed method is based on value iteration, which does not rely on an initial admissible control policy for learning. Considering the premise that the VSG control should stabilize the closed-loop dynamics, the VSG outputs are optimally regulated through the adaptive, optimal control strategy proposed in this article. Comparative simulations and experimental results validate the proposed method's effectiveness and reveal its practicality and implementation.

Adaptive dynamic programming‐based adaptive‐gain sliding mode tracking control for fixed‐wing unmanned aerial vehicle with disturbances

AbstractThis article proposes an adaptive dynamic programming‐based adaptive‐gain sliding mode control (ADP‐ASMC) scheme for a fixed‐wing unmanned aerial vehicle (UAV) with matched and unmatched disturbances. Starting from the dynamic of fixed‐wing UAV, the control‐oriented model composed of attitude subsystem and airspeed subsystem is established. According to the different issues in two subsystems, two novel adaptive‐gain generalized super‐twisting (AGST) algorithms are developed to eliminate the effects of disturbances in two subsystems and make the system trajectories tend to the designed integral sliding manifolds in finite time. Then, based on the expected equivalent sliding‐mode dynamics, the modified adaptive dynamic programming approach with actor‐critic structure is utilized to generate the nearly optimal control laws and achieve the nearly optimal performance of the sliding‐mode dynamics. Furthermore, through the Lyapunov stability theorem, the tracking errors and the weight estimation errors of two neural networks are all uniformly ultimately bounded. Finally, comparative simulations demonstrate the superior performance of the proposed control scheme for the fixed‐wing UAV.

Reinforcement Learning-Based Cooperative Optimal Output Regulation via Distributed Adaptive Internal Model.

In this article, a data-driven distributed control method is proposed to solve the cooperative optimal output regulation problem of leader-follower multiagent systems. Different from traditional studies on cooperative output regulation, a distributed adaptive internal model is originally developed, which includes a distributed internal model and a distributed observer to estimate the leader's dynamics. Without relying on the dynamics of multiagent systems, we have proposed two reinforcement learning algorithms, policy iteration and value iteration, to learn the optimal controller through online input and state data, and estimated values of the leader's state. By combining these methods, we have established a basis for connecting data-distributed control methods with adaptive dynamic programming approaches in general since these are the theoretical foundation from which they are built.