Adaptive Neural Network Control of AUVs With Control Input Nonlinearities Using Reinforcement Learning

In this paper, decentralized optimal tracking control problem has been studied for multi-agent systems (MAS) with a large population, i.e., massive MAS. Due to the curse of dimensionality and limited communication resource, solving optimal tracking control for massive multi-agent systems in a decentralized manner is much more preferred but also challenging. Therefore, the emerging Mean Field game theory has been adopted and integrated with online reinforcement learning approaches and further produced a novel Actor-Critic-Mass algorithm. In the developed scheme, each agent has three neural networks (NN), i.e., 1) mass neural network (NN) that learned the MAS large population behaviors, 2) critic NN that estimated optimal cost function by using local information and mass behaviors learned from mass NN, and 3) actor NN that online solved the decentralized adaptive optimal tracking control based on information obtained from mass NN and critic NN. According to mean-field game theory, the Hamiltonian-Jacobian-Bellman (HJB) and Fokker-Planck-Kolmogorov (FPK) equations are two key components that can be used for tuning actor, critic, and mass NNs effectively. Moreover, Lyapunov theory is used to prove that all the closed-loop signals and NN weights are uniformly bounded in the meanwhile the approximated control input converges close to its near optimal cost with time. Eventually, a series of comprehensive simulation demonstrated the effectiveness of the proposed framework.

Learning‐based robust control methodologies under information constraints

A novel model-free robust saturated reinforcement learning-based controller for quadrotors guaranteeing prescribed transient and steady state performance

A novel adaptive-critic-based neural network (NN) controller in discrete time is designed to deliver a desired tracking performance for a class of nonlinear systems in the presence of actuator constraints. The constraints of the actuator are treated in the controller design as the saturation nonlinearity. The adaptive critic NN controller architecture based on state feedback includes two NNs: the critic NN is used to approximate the "strategic" utility function, whereas the action NN is employed to minimize both the strategic utility function and the unknown nonlinear dynamic estimation errors. The critic and action NN weight updates are derived by minimizing certain quadratic performance indexes. Using the Lyapunov approach and with novel weight updates, the uniformly ultimate boundedness of the closed-loop tracking error and weight estimates is shown in the presence of NN approximation errors and bounded unknown disturbances. The proposed NN controller works in the presence of multiple nonlinearities, unlike other schemes that normally approximate one nonlinearity. Moreover, the adaptive critic NN controller does not require an explicit offline training phase, and the NN weights can be initialized at zero or random. Simulation results justify the theoretical analysis.

This paper presents a novel adaptive optimal control algorithm by combining adaptive dynamic programming with nonlinear model predictive control for unknown continuous‐time affine nonlinear systems. The adaptive optimal control design is realized by the model‐critic‐actor architecture. Model neural network, critic neural network and actor neural network are constructed to approximate the system dynamics, the cost function and the optimal control law respectively. The random initialization of neural networks usually influences the control performance, so three neural networks are initialized properly to obtain the suitable initial values so that the control performance is improved. Especially, actor neural network is initialized to approximate the near‐optimal control law which is obtained from nonlinear model predictive control. The convergence of the proposed algorithm is proved by the Lyapunov theory. Finally, simulation results are provided to illustrate the effectiveness of the proposed algorithm.

In this paper, a decentralized adaptive optimal control based on the Mean Field game and self-organizing neural networks has been developed for multi-agent systems (MAS) with a large population and uncertain dynamics. This design can effectively break "Curse of dimensionality" as well as reduce the computational complexity through appropriately integrating emerging mean-field game theory with self-organizing neural networks based reinforcement learning technique. Firstly, decentralized optimal control for massive multi-agent systems has been formulated as a mean-field game. To obtain the mean-field game solution, coupled Hamiltonian-Jacobian-Bellman (HJB) equation and Fokker-Planck-Kolmogorov (FPK) equation needed to be solved simultaneously which is challenging in real-time. Therefore, a novel Actor-Critic-Mass (ACM) structure has been developed along with self-organizing neural networks. In the developed ACM structure, each agent has three neural networks (NN), which is, 1) mass NN that learns the team's overall behavior via online estimating the solution of Fokker-Planck-Kolmogorov (FPK) equation, 2) critic NN that obtains the optimal cost function via learning the Hamiltonian-Jacobian-Bellman (HJB) equation solution along with time. 3) actor NN that estimates the decentralized optimal control by using the critic and mass NNs. To reduce the NNs computational complexity, a self-organizing neural network has been adopted that can adjusting NNs' architecture based on the NN learning performance as well as the computation cost. Eventually, numerical simulation has been provided to demonstrate the effectiveness of the developed scheme.

Adaptive control for multi-agent systems with actuator fault via reinforcement learning and its application on multi-unmanned surface vehicle

In this paper, an adaptive controller is developed for a class of multi-input and multioutput nonlinear systems with neural networks (NNs) used as a modeling tool. It is shown that all the signals in the closed-loop system with the proposed adaptive neural controller are globally uniformly bounded for any external input in . In our control design, the upper bound of the NN modeling error and the gains of external disturbance are characterized by unknown upper bounds, which is more rational to establish the stability in the adaptive NN control. Filter-based modification terms are used in the update laws of unknown parameters to improve the transient performance. Finally, fault-tolerant control is developed to accommodate actuator failure. An illustrative example applying the adaptive controller to control a rigid robot arm shows the validation of the proposed controller.In this paper, an adaptive controller is developed for a class of multi-input and multioutput nonlinear systems with neural networks (NNs) used as a modeling tool. It is shown that all the signals in the closed-loop system with the proposed adaptive neural controller are globally uniformly bounded for any external input in . In our control design, the upper bound of the NN modeling error and the gains of external disturbance are characterized by unknown upper bounds, which is more rational to establish the stability in the adaptive NN control. Filter-based modification terms are used in the update laws of unknown parameters to improve the transient performance. Finally, fault-tolerant control is developed to accommodate actuator failure. An illustrative example applying the adaptive controller to control a rigid robot arm shows the validation of the proposed controller.

In this paper, a super-twisting disturbance observer (STDO)-based adaptive reinforcement learning control scheme is proposed for the straight air compound missile system with aerodynamic uncertainties and unmodeled dynamics. Firstly, neural network (NN)-based adaptive reinforcement learning control scheme with actor-critic design is investigated to deal with the tracking problems for the straight gas compound system. The actor NN and the critic NN are utilised to cope with the unmodeled dynamics and approximate the cost function that are related to control input and tracking error, respectively. In other words, the actor NN is used to perform the tracking control behaviours, and the critic NN aims to evaluate the tracking performance and give feedback to actor NN. Moreover, with the aid of the STDO disturbance observer, the problem of the control signal fluctuation caused by the mismatched disturbance can be solved well. Based on the proposed adaptive law and the Lyapunov direct method, the eventually consistent boundedness of the straight gas compound system is proved. Finally, numerical simulations are carried out to demonstrate the feasibility and superiority of the proposed reinforcement learning-based STDO control algorithm.

Discrete-time online learning control for a class of unknown nonaffine nonlinear systems using reinforcement learning

This study focuses on the novel reinforcement learning control strategy of a nonlinear two-degrees-of-freedom (2-DOF) helicopter system for tracking the desired trajectory while minimizing the tracking error. First, gradient descent algorithm is incorporated in the context of the reinforcement learning control scheme to obtain the adaptive laws. Subsequently, considering the uncertainties in the nonlinear system, radial basis function (RBF) neural networks (NNs) are exploited to approximate the unknown internal dynamics. In contrast to the previous studies, aiming at accelerating the convergence in reinforcement learning control, a barrier Lyapunov function is constructed to constrain the states to ensure that the tracking error rapidly converges to a neighborhood of zero. Under the proposed control strategy, the states of the closed-loop system are proven to be semi-globally uniformly ultimately bounded through rigorous Lyapunov analyses, and the state constraints are satisfied. Furthermore, the simulations and experiments conducted on a Quanser laboratory platform reveal that the proposed control functions are suitable and effective. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —This paper is motivated by designing a reinforcement learning control strategy to enhance online learning capability and control performance of the controller for a nonlinear 2-DOF helicopter system. The control framework is divided into the design of the critic and actor NNs, responsible primarily for evaluating the control performance and approximating uncertainties in the system separately. Unlike the adaptive NN control, the actor NN weights are updated by combining information of states and inputs from the critic NN. In addition, aiming at accelerating the convergence, a barrier Lyapunov function is constructed to constrain the states to ensure that the tracking error rapidly converges to a neighborhood of zero. Finally, the proposed control strategy is validated in simulation and experiment on the Quanser laboratory platform.

In this paper, our main goal is to solve optimal control problem by using reinforcement learning (RL) algorithm for marine surface vessel system with known dynamic. And this algorithm is an optimal control algorithm based on policy iteration (PI), and it can obtain the suitable approximations of cost function and the optimized control policy. There are two neural networks (NNs), where critic NN aims to estimate the cost-to-go and actor NN is utilized to design suitable input controller and minimize the tracking error. A novel tuning method is given for critic NN and actor NN. The stability and convergence are proven by Lyapunov's direct method. Finally, the numerical simulations are conducted to demonstrate the feasibility and superiority of presented algorithm.

This paper proposes an optimized tracking control approach using neural network (NN) based reinforcement learning (RL) for a class of nonlinear dynamic systems, which requires both tracking and optimizing to be performed simultaneously. Generally, for obtaining optimal control solution, Hamilton-Jacobi-Bellman equation is expected to be solvable, but, owing to strong nonlinearity, the equation is solved difficultly or even impossibly by analytical methods. Therefore, adaptive NN approximation based RL is usually considered. In the optimized control design, for driving output state following to the desired trajectory, an error term is split from optimal performance index function, and then both actor and critic NNs are built to perform RL algorithm. Actor NN aims to execute control behaviors, and critic NN aims to appraise control performance and make feedback to actor. The proof of stability concludes that the desired control performances are obtained. A numerical simulation is designed and implemented, and the desired results are shown.

In this paper, a novel reinforcement learning neural network (NN)-based controller, referred to adaptive critic controller, is proposed for affine nonlinear discrete-time systems with applications to nanomanipulation. In the online NN reinforcement learning method, one NN is designated as the critic NN, which approximates the long-term cost function by assuming that the states of the nonlinear systems is available for measurement. An action NN is employed to derive an optimal control signal to track a desired system trajectory while minimizing the cost function. Online updating weight tuning schemes for these two NNs are also derived. By using the Lyapunov approach, the uniformly ultimate boundedness (UUB) of the tracking error and weight estimates is shown. Nanomanipulation implies manipulating objects with nanometer size. It takes several hours to perform a simple task in the nanoscale world. To accomplish the task automatically the proposed online learning control design is evaluated for the task of nanomanipulation and verified in the simulation environment

In this paper, neural networks (NNs) and adaptive robust control (ARC) design philosophy are integrated to design performance‐oriented control laws for a class of single‐input–single‐output (SISO)nth‐order non‐ linear systems. Both repeatable (or state dependent) unknown non‐linearities and non‐repeatable unknown non‐linearities such as external disturbances are considered. In addition, unknown non‐linearities can exist in the control input channel as well. All unknown but repeatable non‐linear functions are approximated by outputs of multi‐layer neural networks to achieve a better model compensation for an improved performance. All NN weights are tuned on‐line with no prior training needed. In order to avoid the possible divergence of the on‐line tuning of neural network, discontinuous projection method with fictitious bounds is used in the NN weight adjusting laws to make sure that all NN weights are tuned within a prescribed range. By doing so, even in the presence of approximation error and non‐repeatable non‐linearities such as disturbances, a controlled learning is achieved and the possible destabilizing effect of on‐line tuning of NN weights is avoided. Certain robust control terms are constructed to attenuate various model uncertainties effectively for a guaranteed output tracking transient performance and a guaranteed final tracking accuracy in general. In addition, if the unknown repeatable model uncertainties are in the functional range of the neural networks and the ideal weights fall within the prescribed range, asymptotic output tracking is also achieved to retain the perfect learning capability of neural networks in the ideal situation. The proposed neural network adaptive control (NNARC) strategy is then applied to the precision motion control of a linear motor drive system to help to realize the high‐performance potential of such a drive technology. NN is employed to compensate for the effects of the lumped unknown non‐linearities due to the position dependent friction and electro‐magnetic ripple forces. Comparative experiments verify the high‐performance nature of the proposed NNARC. With an encoder resolution of 1 µm, for a low‐speed back‐and‐forth movement, the position tracking error is kept within ±2 µm during the most execution time while the maximum tracking error during the entire run is kept within ±5.6 µm. Copyright © 2001 John Wiley &amp; Sons, Ltd.