Adaptive Dynamic Programming Technique Research Articles

Robust optimal control for uncertain wheeled mobile robot based on reinforcement learning: ADP approach

This paper presents a robust optimal control approach for the wheel mobile robot system, which considers the effects of external disturbances, uncertainties, and wheel slipping. The proposed method utilizes an adaptive dynamic programming (ADP) technique in conjunction with a disturbance observer. Initially, the system's state space model is formulated through the utilization of kinematic and dynamic models. Subsequently, the ADP method is employed to establish an online adaptive optimal controller, which solely relies on a single neural network for the purpose of function approximation. The utilization of the disturbance observer in conjunction with the compensation controller serves to alleviate the effects of disturbances. The Lyapunov theorem establishes the stability of the complete closed-loop system and the convergence of the weights of the neural network. The proposed approach has been shown to be effective through simulation under the effect of the disturbances and the change of the desired trajectory.

Adaptive dynamic programming and distributionally robust optimal control of linear stochastic system using the Wasserstein metric

SummaryIn this paper, we consider the optimal control of unknown stochastic dynamical system for both the finite‐horizon and infinite‐horizon cases. The objective of this paper is to find an optimal controller to minimize the expected value of a function which depends on the random disturbance. Throughout this paper, it is assumed that the mean vector and covariance matrix of the disturbance distribution is unknown. An uncertainty set in the space of mean vector and the covariance matrix is introduced. For the finite‐horizon case, we derive a closed‐form expression of the unique optimal policy and the opponents policy that generates the worst‐case distribution. For the infinite‐horizon case, we simplify the Riccati equation obtained in the finite‐hozion setting to an algebraic Riccati equation, which can guarantee the existence of the solution of the Riccati equation. It is shown that the resulting optimal policies obtained in these two cases can stabilize the expected value of the system state under the worst‐case distribution. Furthermore, the unknown system matrices can also be explicitly computed using the adaptive dynamic programming technique, which can help compute the optimal control policy by solving the algebraic Riccati equation. Finally, a simulation example is presented to demonstrate the effectiveness of our theoretical results.

Optimized Backstepping Cooperative Control for Output-Constrained Stochastic Nonlinear Network Systems via a Multibridge-Hole Function.

In this article, a new leader-following tracking control approach is investigated for stochastic multiagent systems with multibridge-hole output constraints. The multibridge-hole output constraints mean that the output of the system is constrained in some intervals and unconstrained in other intervals. The constrained and unconstrained intervals can be set arbitrarily. By designing a new shift function to construct the barrier Lyapunov function, the optimal controller is constructed by combining the backstepping technique with the adaptive dynamic programming technique. The model network is used to estimate the unknown disturbances and uncertainty terms in the system. The critic network and the actor network are constructed such that the designed controller adheres to the Bellman optimality principle and gives the optimal solution of the system. The proposed control method is versatile and compatible with various types of output constrained control problems, such as unconstrained control problems, constrained control problems, and delay constrained problems without changing the structure of the controller. Finally, some simulation results are given to verify the effectiveness of the method.

Adaptive fault-tolerant optimal control for hypersonic vehicles with state constrains based on adaptive dynamic programming

This paper addresses a novel adaptive fault-tolerant control (FTC) design based on adaptive dynamic programming (ADP) technique for hypersonic vehicle (HSV) subject to actuator fault and state constrains. The total control input is constructed by the combination of backstepping-based adaptive FTC and the ADP-based adaptive optimal control to improve the tracking performance and fault-tolerant capacity. By introducing a barrier Lyapunov function (BLF) to deal with the state constraints, a backstepping-based FTC scheme is designed to transform the tracking control problem into an equivalent optimal control problem. Subsequently, an adaptive optimal control strategy is developed by using the ADP technique to provide a supplementary control action. A critic network is constructed to solve the Hamilton–Jacobi–Bellman (HJB) equation online. The convergence properties of the closed-loop system are developed by utilizing the Lyapunov stability theory. Finally, simulation results are carried out to illustrate the effectiveness and efficiency of the proposed adaptive ADP-based FTC strategy.

Optimal captured power control of variable speed wind turbine systems: Adaptive dynamic programming approach

SummaryAn adaptive optimal controller is proposed in this paper to maximize the captured power of a variable speed wind power system. The proposed controller is a combination of optimal and adaptive control components. The Adaptive Dynamic Programming technique is used to design the optimal control component to overcome the nonlinear problem of system dynamics and ensure stability. While the neural network is used to approximate unknown disturbances and system uncertainties. After that, the adaptive control component fully compensates for the effects of these unknown elements. Neither optimal nor adaptive control components necessitate prior knowledge of system dynamics. Furthermore, the approximation network updates only the weight matrix norm rather than the weight matrix of the neural network in each interval time, which significantly reduces computation. The stability analysis of the closed‐loop system is obtained using Lyapunov stability theory. The correctness and robustness of the control scheme are validated in two different scenarios using MATLAB/Simulink. The presented robust adaptive optimal controller is also compared to other existing controllers to demonstrate its benefits.

Reinforcement Learning for Dual-Control Aircraft Six-Degree-of-Freedom Attitude Control with System Uncertainty

This article proposes a near-optimal control strategy based on reinforcement learning, which is applied to the six-degree-of-freedom (6-DoF) attitude control of dual-control aircraft. In order to solve the problem that the existing reinforcement learning is difficult to apply to the high-dimensional multiple-input multiple-output (MIMO) systems, the Long Short-Term Memory (LSTM) neural network is introduced to replace the polynomial network in the adaptive dynamic programming (ADP) technique. Meanwhile, based on the Lyapunov method, a novel online adaptive updating law of LSTM neural network weights is given, and the stability of the system is verified. In the simulation process, the algorithm proposed in this article is applied to the six-degree-of-freedom attitude control problem of dual-control aircraft with system uncertainty. The simulation results show that the algorithm can achieve near-optimal control.

Time-varying gain extended state observer-based adaptive optimal control for disturbed unmanned helicopter

In this paper, the robust adaptive optimal tracking control problem is addressed for the disturbed unmanned helicopter based on the time-varying gain extended state observer (TVGESO) and adaptive dynamic programming (ADP) methods. Firstly, a novel TVGESO is developed to tackle the unknown disturbance, which can overcome the drawback of initial peaking phenomenon in the traditional linear ESO method. Meanwhile, compared with the nonlinear ESO, the proposed TVGESO possesses easier and rigorous stability analysis process. Subsequently, the optimal tracking control issue for the original unmanned helicopter system is transformed into an optimization stabilization problem. By means of the ADP and neural network techniques, the feedforward controller and optimal feedback controller are skillfully designed. Compared with the conventional backstepping approach, the designed anti-disturbance optimal controller can make the unmanned helicopter accomplish the tracking task with less energy. Finally, simulation comparisons demonstrate the validity of the developed control scheme.

Reinforcement Learning-Based Event-Triggered FCS-MPC for Power Converters

This article aims to first focus on an improvement of finite control-set model predictive control strategy for power converters that is based on reinforcement learning event-triggered predictive control architecture with the help of adaptive dynamic programming technique and event-triggered mechanism subject to system uncertainties. Our development, endowed with the merits of reinforcement learning and event-triggered control as well as predictive control solution, is able to alleviate the issues of parametric uncertainties and high switching frequency inherent in the existing scheme, while retaining the merits of the finite control-set model predictive control. Finally, this proposal is experimentally evaluated, where robust performance tests confirm the interest and applicability of the proposed control methodology.

Two‐layer leader‐follower optimal affine formation maneuver control for networked unmanned surface vessels with input saturations

AbstractThis article studies the two‐layer leader‐follower optimal affine formation maneuver (AFM) control problem for multiple unmanned surface vessels (USV). The basic idea of the developed two‐layer control framework is to realize the time‐varying formation for the leader group in advance, and then let the follower USVs agilely respond to the target formation affinely localized by leaders. To realize this goal, a time‐varying formation reference system is first synthesized to describe leaders' target formation in the affine map of the nominal configuration. The reference system is set up by virtual vessel velocity command and unrotated geometric command, which make it possesses an intuitive application structure for operators to plan AFM commands in real‐time sensed surrounding environments. Then, by means of optimized backstepping design and the adaptive dynamic programming technique, actor‐only reinforcement learning controllers are established for USVs in both two layers to realize the optimal target formation tracking objective. For the proposed optimal controllers, the persistent excitation condition that is required by common optimal control schemes is also removed, which endows the presented scheme with lower computational complexity. Finally, theoretical analysis and simulation results illustrate the closed‐loop stability and the effectiveness of the presented scheme.

Nearly optimal fault-tolerant constrained tracking for multi-axis servo system via practical terminal sliding mode and adaptive dynamic programming

In this paper, a nearly optimal tracking control is proposed for n-links robotic manipulators subject to parameter uncertainties, time-profile failures, and input saturation constraints. Firstly, the practical terminal sliding-mode (PTSM) manifold with a linear additional term is proposed to combine the system states related to joint rotation, such that the controlled states quickly fall into a tiny neighborhood of the equilibrium once they reach the PTSM manifold. Secondly, a nearly optimal sliding-mode reaching law is designed by using the adaptive dynamic programming (ADP) technique. Benefiting from a non-quadratic positive defined mapping of the proposed performance index, which relates to the derivative of the sliding-mode function, reduced-order system dynamics can be constrained to a desired region. For the bounded actuator fault caused by various inducements such as the power supply fluctuation and the wear of parts, a radial basis function neural network (RBFNN) is introduced to compensate for this, and the input saturation constraints of the controlled plant are also compensated at the same time. Innovatively, the node weights of RBFNN are updated by the critic network of the ADP framework, such that the integrity of the proposed control strategy is improved. Simulations verify the main conclusions.

Integral sliding mode‐based event‐triggered nearly optimal tracking control for uncertain nonlinear systems

SummaryIn this paper, an event‐triggered nearly optimal tracking control method is investigated for a class of uncertain nonlinear systems by integrating adaptive dynamic programming (ADP) and integral sliding mode (ISM) control techniques. An ISM‐based discontinuous control law with a neural network (NN) adaptive term is designed to eliminate the influence of the uncertainties and obtain the sliding mode dynamics which is equivalent to the tracking error dynamics without uncertainties, and relax the known upper‐bounded condition of uncertainties. In order to guarantee the stability of tracking error system and the considerable optimality, under the ADP technique, a critic NN is applied to approximate the optimal value function for solving the event‐triggered Hamilton‐Jacobi‐Bellman equation and the event‐triggered nearly optimal feedback control is obtained. The feedback control law is updated and transmitted to plant only when events occur, thus both the communication and the computational resources can be saved. Furthermore, the stability of tracking error is proven thanks to Lyapunov's direct method. Finally, we provide two simulation examples to validate the developed control scheme.

Hierarchical Multiagent Formation Control Scheme via Actor-Critic Learning.

This article presents a nearly optimal solution to the cooperative formation control problem for large-scale multiagent system (MAS). First, multigroup technique is widely used for the decomposition of the large-scale problem, but there is no consensus between different subgroups. Inspired by the hierarchical structure applied in the MAS, a hierarchical leader-following formation control structure with multigroup technique is constructed, where two layers and three types of agents are designed. Second, adaptive dynamic programming technique is conformed to the optimal formation control problem by the establishment of performance index function. Based on the traditional generalized policy iteration (PI) algorithm, the multistep generalized policy iteration (MsGPI) is developed with the modification of policy evaluation. The novel algorithm not only inherits the advantages of high convergence speed and low computational complexity in the generalized PI algorithm but also further accelerates the convergence speed and reduces run time. Besides, the stability analysis, convergence analysis, and optimality analysis are given for the proposed multistep PI algorithm. Afterward, a neural network-based actor-critic structure is built for approximating the iterative control policies and value functions. Finally, a large-scale formation control problem is provided to demonstrate the performance of our developed hierarchical leader-following formation control structure and MsGPI algorithm.

A Unified Framework for Data-Driven Optimal Control of Connected Vehicles in Mixed Traffic

This paper presents a unified approach to the problem of learning-based optimal control of connected human-driven and autonomous vehicles in mixed-traffic environments including both the freeway and ring road settings. The stabilizability of a string of connected vehicles including multiple autonomous vehicles (AVs) and heterogeneous human-driven vehicles (HDVs) is studied by a model reduction technique and the Popov-Belevitch-Hautus (PBH) test. For this problem setup, a linear quadratic regulator (LQR) problem is formulated and a solution based on adaptive dynamic programming (ADP) techniques is proposed without a priori knowledge on model parameters. To start the learning process, an initial stabilizing control law is obtained using the small-gain theorem for the ring road case. It is shown that the obtained stabilizing control law can achieve general <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mathcal {L}_{p}$</tex-math></inline-formula> string stability under appropriate conditions. Besides, to minimize the impact of external disturbance, a linear quadratic zero-sum game is introduced and solved by an iterative learning-based algorithm. Finally, the simulation results verify the theoretical analysis and the proposed methods achieve desirable performance for control of a mixed-vehicular network.

Adaptive fuzzy dynamic programming (AFDP) technique for linear programming problems lps with fuzzy constraints

Adaptive fuzzy dynamic programming (AFDP) technique for linear programming problems lps with fuzzy constraints

Neural network-based adaptive optimal containment control for non-affine nonlinear multi-agent systems within an identifier-actor-critic framework

This paper addresses the adaptive optimal containment control issue for non-affine nonlinear multi-agent systems in the presence of periodic disturbances. To deal with the disturbed internal dynamics, a fourier series expansion-neural networks-based adaptive identifier is designed for each follower, such that the restrictions posed on the system dynamics are released. Then,an adaptive dynamic programming technique is adopted to acquire the optimized virtual and actual controllers under a simplified actor-critic architecture, where the critic aims to appraise control performance and the actor aims to perform control task. Note that the above updating laws are constructed by the negative gradient of a designed function, which is constructed on the basis of the partial derivative of Hamilton-Jacobi-Bellman equation. Finally, simulation results are provided to show the applicability and effectiveness of the containment control scheme.

Neural network–based optimal fault compensation control of the nonlinear multi-agent system and its application to UAVs formation flight

This article investigates the optimal consensus problem for unmanned aerial vehicle formation systems with actuator faults based on nonlinear multi-agent systems. Initially, for fault-free multi-agent system, the distributed optimal controllers are constructed based on the adaptive dynamic programming technique. A critic neural network is applied to approximate the solution of the nonlinear Hamilton–Jacobi–Bellman equations, in which the weight updating laws are built to guarantee the weight vectors of the critic neural network convergence. Second, the fault compensators and corresponding tuning laws are proposed to compensate for actuator faults. Through a combination of optimal controllers and fault compensators, the distributed optimal fault-tolerant controllers are obtained. Then, according to Lyapunov extension theorem, some stability criteria for ensuring the stability of the aircraft and the normal flight of the unmanned aerial vehicle formation are established in the event of an actuator failure. Finally, an example of an unmanned aerial vehicle formation system is introduced to verify the efficiency and reliability of the designed optimal fault-tolerant control scheme.

Further Results on Optimal Tracking Control for Nonlinear Systems With Nonzero Equilibrium via Adaptive Dynamic Programming.

This article develops a novel cost function (performance index function) to overcome the obstacles in solving the optimal tracking control problem for a class of nonlinear systems with known system dynamics via adaptive dynamic programming (ADP) technique. For the traditional optimal control problems, the assumption that the controlled system has zero equilibrium is generally required to guarantee the finiteness of an infinite horizon cost function and a unique solution. In order to solve the optimal tracking control problem of nonlinear systems with nonzero equilibrium, a specific cost function related to tracking errors and their derivatives is designed in this article, in which the aforementioned assumption and related obstacles are removed and the controller design process is simplified. Finally, comparative simulations are conducted on an inverted pendulum system to illustrate the effectiveness and advantages of the proposed optimal tracking control strategy.

Event-Triggered Robust Adaptive Dynamic Programming With Output Feedback for Large-Scale Systems

In this article, an event-triggered output-feedback adaptive optimal control approach is proposed for large-scale systems with parametric and dynamic uncertainties through robust adaptive dynamic programming and small-gain techniques. By using the input and output data, the unmeasurable states are reconstructed instead of designing a Luenberger observer. To save the communication resources and reduce the number of control updates, an event-based feedback control policy is learned based on policy iteration and value iteration, respectively. The closed-loop stability and the convergence of the proposed algorithms are analyzed by using Lyapunov stability theory and small-gain techniques. A practical example of multimachine power systems with governor controllers is given to demonstrate the effectiveness of the proposed methods.

Event-Triggered Near-Optimal Control for Unknown Discrete-Time Nonlinear Systems Using Parallel Control.

This article uses parallel control to investigate the problem of event-triggered near-optimal control (ETNOC) for unknown discrete-time (DT) nonlinear systems. First, to achieve parallel control, an augmented nonlinear system (ANS) with an augmented performance index (API) is proposed to introduce the control input into the feedback system. The control stability relationship between the ANS and the original system is analyzed, and it is shown that, by choosing a proper API, optimal control of the ANS with the API can be seen as near-optimal control of the original system with the original performance index (OPI). Second, based on parallel control, a novel event-triggered scheme is proposed, and then a novel ETNOC method is developed using the time-triggered optimal value function of the ANS with the API. The control stability is proved, and an upper bound, which is related to the design parameter, is provided for the actual performance index in advance. Then, to implement the developed ETNOC method for unknown DT nonlinear systems, a novel online learning algorithm is developed without reconstructing unknown systems, and neural network (NN) and adaptive dynamic programming (ADP) techniques are employed in the developed algorithm. The convergence of the signals in the closed-loop system (CLS) is shown using the Lyapunov approach, and the assumption of boundedness of input dynamics is not required. Finally, two simulations justify the theoretical conjectures.

Fault-tolerant control of a hydraulic servo actuator via adaptive dynamic programming

&lt;abstract&gt;&lt;p&gt;The fault-tolerant control problem of a hydraulic servo actuator in the presence of actuator faults is studied utilizing adaptive dynamic programming. This task is challenging because of unknown system dynamics, uncertain disturbances or unmeasurable system states of such highly nonlinear systems in real applications. The aim is to achieve asymptotic tracking and actuator faults compensation by minimizing some predefined performance index. The discrete-time algebraic Riccati equation is iteratively solved by the adaptive dynamic programming approach. For practical reasons, adaptive dynamic programming techniques and fault compensation are integrated to iteratively compute an approximated optimal fault-tolerant control using real-time input/output data without any a priori knowledge of the system dynamics and unmeasurable states. As a result, a fault-tolerant control of hydraulic servo actuator is then designed based on adaptive dynamic programming via output feedback. Also, the convergence analysis of a data-driven fault-tolerant control is theoretically shown as well. Finally, intensive simulation results are given to prove the validity and merits of the developed data-driven fault-tolerant control strategy.&lt;/p&gt;&lt;/abstract&gt;