Robust Adaptive Dynamic Programming Research Articles

Robust Policy Iteration of Uncertain Interconnected Systems With Imperfect Data

This paper investigates the robust optimal control problem of a class of continuous-time, partially linear, interconnected systems. In addition to the dynamic uncertainties resulted from the interconnected dynamic system, unknown bounded disturbances are taken into account throughout the learning process, wherein the system’s dynamics and the disturbances are assumed unknown. These challenges lead the collected online data to be imperfect. In this scenario, traditional data-driven control techniques, such as adaptive dynamic programming (ADP) and robust ADP, encounter a challenge in approximating the optimal control policy precisely due to imperfect data and computational errors. In this paper, a novel data-driven robust policy iteration method is proposed to simultaneously solve the robust optimal control problems. Without relying on the knowledge of the system’s dynamics, the external disturbances or the complete state, the implementation of the proposed method only needs to access the input and partial state information. Based on the small-gain theorem, the notions of strong unboundedness observability and input-to-output stability, it is guaranteed that the learned robust optimal control gain is stabilizing and that the solution of the closed-loop system is uniformly ultimately bounded despite the existence of dynamic uncertainties and unknown external disturbances. The simulation results reveal the efficiency and practicality of the proposed data-driven control method. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This work is motivated by the use of reinforcement learning to improve the quality of designing adaptive optimal controllers for engineering applications. Adaptive dynamic programming methods, in particular policy iteration (PI), are widely used in solving optimal control problems. However, due to the iterative nature of PI, the approximated optimal control policy may be inaccurate and imprecise. Especially, when using imperfect system’s measurements instead of the modelling information. This can result in causing the learned control policy to deviate from the actual optimal policy. This becomes more challenging in the existence of dynamic uncertainties and unknown external disturbances which corrupt the measurements and result in imperfect data. This work investigates the conditions on the uncertainties such that the proposed novel data-driven PI algorithm is robust to system’s uncertainties, unknown external disturbances and imperfect measurements. The approximated robust optimal control policy performs robustly in the existences of imperfect data and uncertainties, and at the same time is close enough to the optimal control policy.

Data-Based Robust Adaptive Dynamic Programming for Balancing Control Performance and Energy Consumption in Wastewater Treatment Process

Data-Based Robust Adaptive Dynamic Programming for Balancing Control Performance and Energy Consumption in Wastewater Treatment Process

Output information-based intermittent optimal control for continuous-time nonlinear systems with unmatched uncertainties via adaptive dynamic programming

Intermittent control stands as a valuable strategy for resource conservation and cost reduction across diverse systems. Nonetheless, prevailing research is intractable to address the challenges posed by robust optimal intermittent control of nonlinear input-affine systems with unmatched uncertainties. This paper aims to fill this gap. Initially, we introduce an enhanced finite-time intermittent control approach to ensure stability within nonlinear dynamic systems harboring bounded errors. A neural networks (NNs) state observer is constructed to estimate system information. Subsequently, an optimal intermittent controller that operates within a finite time span, guaranteeing system stability by employing the Hamilton–Jacobi–Bellman (HJB) methodology. Furthermore, we devise an output information-based event-triggered intermittent (ETI) approach rooted in the robust adaptive dynamic programming (ADP) algorithm, furnishing an optimal intermittent control law. In this process, a critic NNs is introduced to estimate the cost function and optimal intermittent controller. Simulation results show that our proposed method is superior to existing intermittent control strategies.

Event-Triggered Robust Adaptive Dynamic Programming for Multiplayer Stackelberg-Nash Games of Uncertain Nonlinear Systems.

In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is developed to solve a class of multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. Considering the different roles of players in the MSNG, the hierarchical decision-making process is described as the designed value functions for the leader and all followers, which assist to transform the robust control problem of the uncertain nonlinear system into an optimal regulation problem of the nominal system. Then, an online policy iteration algorithm is formulated to solve the derived coupled Hamilton-Jacobi equation. Meanwhile, an event-triggered mechanism is designed to alleviate computational and communication burdens. Moreover, critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control polices for all players, which constitute the Stackelberg-Nash equilibrium of the MSNG. By using Lyapunov's direct method, the stability of the closed-loop uncertain nonlinear system is guaranteed under the ETRADP-based control scheme in the sense of uniform ultimate boundedness. Finally, a numerical simulation is provided to demonstrate the effectiveness of the present ETRADP-based control scheme.

Event-Triggered Robust Adaptive Dynamic Programming With Output Feedback for Large-Scale Systems

In this article, an event-triggered output-feedback adaptive optimal control approach is proposed for large-scale systems with parametric and dynamic uncertainties through robust adaptive dynamic programming and small-gain techniques. By using the input and output data, the unmeasurable states are reconstructed instead of designing a Luenberger observer. To save the communication resources and reduce the number of control updates, an event-based feedback control policy is learned based on policy iteration and value iteration, respectively. The closed-loop stability and the convergence of the proposed algorithms are analyzed by using Lyapunov stability theory and small-gain techniques. A practical example of multimachine power systems with governor controllers is given to demonstrate the effectiveness of the proposed methods.

Learning Control for Flexible Manipulators with Varying Loads: A Composite Method with Robust Adaptive Dynamic Programming and Robust Sliding Mode Control

This paper focuses on the learning-based motion control for flexible manipulators with varying loads via the singularly perturbed technique. Considering the two-timescale feature of the flexible manipulator, system dynamics are decomposed into fast and slow subsystems, and corresponding sub-controllers are designed with robust adaptive dynamic programming (RADP) and robust sliding mode control (RSMC) methods, respectively. In the proposed composite control framework, an RADP-based sub-controller is developed to realize the trajectory tracking and alleviate the parametric uncertainty utilizing rotating angles in the slow timescale, while an RSMC sub-controller is introduced to improve the vibration suppression in the fast timescale. Finally, the stability of the closed-loop system is guaranteed, and simulations are carried out to show the effectiveness of the proposed control algorithm.

Robust ADP-Based Sliding-Mode Fault-Tolerant Control for Nonlinear Systems with Application to Spacecraft

This paper considers a novel fault-tolerant control (FTC) scheme for a category of cascade nonlinear systems with mismatched uncertainties and unknown actuator faults. The robust adaptive dynamic programming (RADP) is used to design a novel optimal sliding surface (SS) off-line, which renders the corresponding sliding-mode dynamics able to obtain robustness of stability to mismatched uncertainties. Subsequently, a simple sliding-mode control (SMC) with the adaptive fault compensation is developed to guarantee reachability of the sliding mode. Then, it is proven that the weight errors of neural networks (NN) of RADP and the closed-loop system are stable based on Lyapunov stability theory. In the simulation section, this proposed scheme is used to deal with the attitude FTC of a spacecraft, and simulation results verify the effectiveness of the proposed novel control scheme.

Data‐based robust optimal control of discrete‐time systems with uncertainties via adaptive dynamic programming

AbstractIn this article, a new data‐based adaptive dynamic programming algorithm is proposed to solve the optimal control policy for discrete‐time systems with uncertainties. Firstly, for uncertain systems, the corresponding Hamiltonian function is designed, and then the robust adaptive dynamic programming algorithm is obtained. Next, by using the input and output data of the system, the data‐based Bellman equation is constructed, and the data‐based robust adaptive dynamic programming algorithm is derived, which does not require the accurate model of the system. Finally, a simulation example shows the effectiveness of the proposed algorithm.

Policy Iteration and Event-Triggered Robust Adaptive Dynamic Programming for Large-Scale Systems

Abstract In this paper, an event-triggered robust optimal control approach is proposed for large-scale systems with both parametric and dynamic uncertainties through robust adaptive dynamic programming, policy iteration, and small-gain techniques. By using the input and output data, the unmeasurable states are reconstructed instead of constructing a Luenberger observer. Starting from an admissible control policy, an event-based feedback control policy is learned to save the communication resources and reduce the number of control updates. The closed-loop stability and the convergence of the proposed algorithm are analyzed by using Lyapunov and small-gain techniques. A practical example of multimachine power systems with governor controllers is given to demonstrate the effectiveness of the proposed method.

Online Adaptive Critic Learning Control of Unknown Dynamics With Application to Deep Submergence Rescue Vehicle

As a powerful tool for nonlinear systems robust controller design, robust adaptive dynamic programming (RADP) methods require initial admissible control and prior knowledge of disturbance to be effective. As the most effective approach to provide robustness to uncertainties, active disturbance attenuation (ADA), was rarely considered in RADP literatures. To combine ADA with RADP, a neural-network identifier was developed initially to approximate the plant dynamics and the imposed external disturbance. System states was extended with the approximated disturbance to establish ADA actor-critic learning. To relax the initial admissible control constraint, a novel auxiliary system were created based on the identifier dynamics. Theoretical analysis and simulations on unstable nonlinear system show that the approximated control law with respect to the auxiliary system and a newly proposed cost function itself could guarantee asymptotic stability of the original system. Simulations and comparison with other model-free control techniques demonstrated the excellent performance and robustness of the proposed method. Applicability of the proposed method was validated by applying it to trajectory tracking control of a deep submergence rescue vehicle.

An adaptive dynamic programming method for torque ripple minimization of PMSM

<p style='text-indent:20px;'>The imperfect sinusoidal flux distribution, cogging torque, and current measurement errors can cause periodic torque ripple in the permanent magnet synchronous motor (PMSM). These ripples are reflected in the periodic oscillation of the motor speed and torque, causing vibration at low speeds and noise at high speeds. As a high-precision tracking application, ripple degrades the application performance of PMSM. In this paper, an adaptive dynamic programming (ADP) scheme is proposed to reduce the periodic torque ripples. An optimal controller is designed by iterative control algorithm using robust adaptive dynamic programming theory and strategic iterative technique. ADP is combined with the existing Proportional-Integral (PI) current controller and generates compensated reference current iteratively from cycle to cycle so as to minimize the mean square torque error. As a result, an optimization problem is constructed and an optimal controller is obtained. The simulation results show that the robust adaptive dynamic programming achieves lower torque ripple and shorter dynamic adjustment time during steady-state operation, thus meeting the requirements of steady speed state and the dynamic performance of the regulation system.

Event-Triggered Integral Sliding-Mode Control for Nonlinear Constrained-Input Systems With Disturbances via Adaptive Dynamic Programming

This article proposed a novel event-triggered integral sliding-mode control (ISMC) strategy for nonlinear system with disturbances via robust adaptive dynamic programming (ADP) considering control constrains. A mixed event-triggered ISMC scheme with two different trigger parts is developed and the optimal performance of sliding-mode dynamics with input constrains is ensured. By guaranteeing that the system trajectory converges to the sliding-mode surface with removing the input disturbances, a discontinuous part triggering rule is presented together with the existence analysis of a lower trigger interval time bound. Then, the optimal event-triggered control of sliding-mode dynamics is converted into a discounted factor-based $H_{\infty }$ constrained control problem under continuous part triggering condition. To solve the event-triggered HJI equation, a critic-only neural network (NN)-based ADP scheme is developed by applying a concurrent learning. The NN weight is updated by analyzing the uniformly ultimately bounded (UUB) stability of sliding-mode dynamics considering the event-triggered condition via the Lyapunov technique. Finally, the validity of our control strategy is verified by simulation.

Data-Driven Cooperative Output Regulation of Multi-Agent Systems via Robust Adaptive Dynamic Programming

This brief studies the cooperative output regulation problems of multi-agent systems with parametric and dynamic uncertainties. By means of robust adaptive dynamic programming, a model-free distributed controller is developed via online input and state data. The cyclic-small-gain theorem is applied to ensure the asymptotic stability of the closed-loop system and thus solves the cooperative output regulation problem. Each follower is able to achieve asymptotic tracking and non-vanishing disturbance rejection. A numerical example is utilized to illustrate the effectiveness of the proposed algorithm.

Optimal control for a class of complex singular system based on adaptive dynamic programming

This paper presents a new design approach to achieve decentralized optimal control of high-dimension complex singular systems with dynamic uncertainties. Based on robust adaptive dynamic programming &#x0028 robust ADP &#x0029 method, controllers for solving the singular systems optimal control problem are designed. The proposed algorithm can work well when the system model is not exactly known but the input and output data can be measured. The policy iteration of each controller only uses their own states and input information for learning, and do not need to know the whole system dynamics. Simulation results on the New England 10-machine 39-bus test system show the effectiveness of the designed controller.

Robust Optimal Control of Wave Energy Converters Based on Adaptive Dynamic Programming

This paper presents a robust adaptive optimal control strategy for wave energy converters (WECs). We first propose a new estimator in a simple form to address modeling uncertainties and formulate the control of WECs as an optimal control problem. Then, a novel energy maximization control strategy is developed based on the concept of adaptive dynamic programming (ADP), where a critic neural network (NN) is used to approximate the time-dependent optimal cost value. To achieve guaranteed convergence, a recently proposed adaptive law based on the parameter estimation error is further tailored to update the weights of critic NN online. Consequently, the critic NN output, e.g., the costate, can be used to compute the optimal feedback control. The proposed robust ADP WEC control method is not only effective in handling dynamic uncertainties, but also computationally efficient with a very fast online convergence rate for the weights of the critic NN (less than 20 seconds for irregular sea waves as demonstrated in the simulations). These advantages significantly enhance the real-time applicability of the proposed method. Simulation results show that this approach is robust to model uncertainties and has significantly reduced computational costs for implementation.

Robust ADP Design for Continuous-Time Nonlinear Systems With Output Constraints.

In this paper, a novel robust adaptive dynamic programming (RADP)-based control strategy is presented for the optimal control of a class of output-constrained continuous-time unknown nonlinear systems. Our contribution includes a step forward beyond the usual optimal control result to show that the output of the plant is always within user-defined bounds. To achieve the new results, an error transformation technique is first established to generate an equivalent nonlinear system, whose asymptotic stability guarantees both the asymptotic stability and the satisfaction of the output restriction of the original system. Furthermore, RADP algorithms are developed to solve the transformed nonlinear optimal control problem with completely unknown dynamics as well as a robust design to guarantee the stability of the closed-loop systems in the presence of unavailable internal dynamic state. Via small-gain theorem, asymptotic stability of the original and transformed nonlinear system is theoretically guaranteed. Finally, comparison results demonstrate the merits of the proposed control policy.

Stochastic and adaptive optimal control of uncertain interconnected systems: A data-driven approach

This paper provides a novel non-model-based, data-driven stochastic H∞ control design for linear continuous-time stochastic interconnected systems with unknown dynamics. Our contributions are three-fold. First, we develop a tool to show how to assign an arbitrarily small input-to-output stochastic L2 gain of the closed-loop system, by combining the gain assignment technique with the zero-sum dynamic game-based H∞ control design. Second, robustness to dynamic uncertainties is tackled using the small-gain theory. Third, we develop a non-model-based stochastic robust adaptive dynamic programming (RADP) algorithm for adaptive optimal controller design. In sharp contrast to the existing methods, the obtained algorithm is based on value iteration (VI), and the knowledge of an initial stabilizing control policy is no longer needed. An example of a power electronic system is adopted to illustrate the obtained results.

Adaptive Dynamic Programming for Stochastic Systems With State and Control Dependent Noise

In this technical note, the adaptive optimal control problem is investigated for a class of continuous-time stochastic systems subject to multiplicative noise. A novel non-model-based optimal control design methodology is employed to iteratively update the control policy on-line by using directly the data of the system state and input. Both adaptive dynamic programming (ADP) and robust ADP algorithms are developed, along with rigorous stability and convergence analysis. The effectiveness of the obtained methods is illustrated by an example arising from biological sensorimotor control.

Output-feedback adaptive optimal control of interconnected systems based on robust adaptive dynamic programming

This paper studies the adaptive and optimal output-feedback problem for continuous-time uncertain systems with nonlinear dynamic uncertainties. Data-driven output-feedback control policies are developed by approximate/adaptive dynamic programming (ADP) based on both policy iteration and value iteration methods. The obtained adaptive and optimal output-feedback controllers differ from the existing literature on the ADP in that they are derived from sampled-data systems theory and are guaranteed to be robust to dynamic uncertainties. A small-gain condition is given under which the overall system is globally asymptotically stable at the origin. An application to power systems is given to test the effectiveness of the proposed approaches.

Robust adaptive dynamic programming based online tracking control algorithm for real wheeled mobile robot with omni-directional vision system

This paper proposes a new method to design an online robust adaptive dynamic programming algorithm (RADPA) for a wheeled mobile robot which is equipped with an omni-directional vision system. To integrate kinematic and dynamic controllers into the unique controller, we transform the strict feedback system dynamics into tracking error dynamics. Then, we propose a control scheme which uses only one neural network rather than three proposed in the actor-critic-based control schemes for the two-player zero-sum game problem. A neural network weight update law is designed for approximating the solution of the Hamilton–Jacobi–Isaacs equation without knowing knowledge of internal system dynamics. To implement the scheme, we propose the online RADPA, in which control and disturbance laws are updated simultaneously in an iterative loop. The convergence and stability of the online RADPA are proven by Lyapunov techniques. Simulations and experiments on a wheeled mobile robot testbed are carried out to verify the effectiveness of the proposed algorithm.