Unknown Continuous-time Nonlinear Systems Research Articles

Nearly optimal stabilization of unknown continuous-time nonlinear systems: A new parallel control approach

This paper develops a novel online nearly optimal control (ONOC) method for unknown continuous-time (CT) nonaffine nonlinear systems without recovering unknown systems. First, a dynamic control law is proposed for CT nonaffine nonlinear systems using parallel control. To achieve the proposed dynamic control law, an affine augmented system (AAS) is constructed according to the original system, and an augmented performance index (API) is constructed on the basis of the original performance index (OPI). Then, the stability relationship between the original system and the AAS is provided, and it is proven that, by selecting a suitable parameter in the API, optimal control of the AAS with the API is equivalent to near-optimal control of the original system with the OPI. Subsequently, based on the proposed dynamic control law, we extend integral reinforcement learning (IRL) to completely unknown CT nonaffine systems, and it is further proved that closed-loop signals are uniformly ultimately bounded (UUB) without the assumption that the input dynamics are bounded. Furthermore, the OPI can be set to an arbitrary positive-definite form, and the UUB bound for the state vector can be predetermined. Lastly, simulations are offered to exhibit the correctness of the developed ONOC method. Source code of this paper is available at: https://github.com/lujingweihh/Adaptive-dynamic-programming-algorithms/tree/main/model_free_integral_reinforcement_learning.

Robust safe reinforcement learning control of unknown continuous-time nonlinear systems with state constraints and disturbances

Industrial processes must operate safely and optimally in practice, which typically entails solving a constrained optimal control (COC) problem. Finding a control policy for such a problem, however, is challenging especially when the system dynamics are unknown. In this paper, a novel safe reinforcement learning (RL) algorithm is proposed to deal with the COC problem for the continuous-time nonlinear system with unknown dynamics and disturbances. The presented method can ensure that the system operates within a predefined safe region in the presence of system model uncertainties and external disturbances, which goes beyond the results of typical RL methods. To handle the system state constraints, the problem is first transformed into an unconstrained one via the proposed data-driven slack function approach. Then an improved model-based RL method is devised to learn a near-optimal and safe control policy in real-time. Finally, a robust compensator and the composite neural-network updating rule are designed to eliminate the influence of disturbances and uncertainties. Theoretical analysis based on the Lyapunov approach is conducted to prove the closed-loop system stability and the algorithm convergence, which further guarantees the satisfaction of safety constraints. The effectiveness of the proposed RL algorithm is verified through simulations.

Adaptive Optimal Control for Unknown Constrained Nonlinear Systems With a Novel Quasi-Model Network.

A policy-iteration-based algorithm is presented in this article for optimal control of unknown continuous-time nonlinear systems subject to bounded inputs by utilizing the adaptive dynamic programming (ADP). Three neural networks (NNs), called critic network, actor network, and quasi-model network, are utilized in the proposed algorithm to give approximations of the control law, the cost function, and the function constituted by partial derivatives of value functions with respect to states and unknown input gain dynamics, respectively. At each iteration, based on the least sum of squares method, the parameters of critic and quasi-model networks will be tuned simultaneously, which eliminates the necessity of separately learning the system model in advance. Then, the control law is improved by satisfying the necessary optimality condition. Then, the proposed algorithm's optimality and convergence properties are exhibited. Finally, the simulation results demonstrate the availability of the proposed algorithm.

Adaptive Event-Triggered Near-Optimal Tracking Control for Unknown Continuous-Time Nonlinear Systems

This paper studies the event-triggered optimal tracking control (ETOTC) problem of continuous-time (CT) unknown nonlinear systems. In order to solve the ETOTC problem, an augmented system composed of the error system dynamics and the reference dynamics is used to introduce a new discounted performance index function (DPIF). A novel event-triggered (ET) adaptive dynamic programming (ADP) method is developed to solve the ET Hamilton-Jacobi-Bellman equation (HJBE). The presented method is implemented via an identifier-critic architecture, which consists of two neural networks (NNs): an identifier NN is applied to estimate the unknown system dynamics, and a critic NN is constructed to obtain the approximate solution of the ET HJBE. The augmented closed-loop system and the critic estimation error are proved to be ultimately uniformly bounded (UUB) by the Lyapunov direct method. Finally, two simulations illustrate the effectiveness of the developed method.

Indirect adaptive fuzzy-regulated optimal control for unknown continuous-time nonlinear systems

Indirect adaptive fuzzy-regulated optimal control for unknown continuous-time nonlinear systems

Event-Triggered Adaptive Dynamic Programming for Zero-Sum Game of Partially Unknown Continuous-Time Nonlinear Systems

In this paper, the zero-sum game problem is considered for partially unknown continuous-time nonlinear systems, and an event-triggered adaptive dynamic programming (ADP) method is developed to solve the problem. First, an identifier neural network (NN) and a critic NN are applied to approximate the drift system dynamics and the optimal value function, respectively. Subsequently, an event-triggered approach is developed based on ADP, which samples the states and updates the weights of NNs at the same time when the event-triggering condition is violated, such that the computational complexity is reduced. It is proved that the states and the error of NN weights are uniformly ultimately bounded. Finally, the effectiveness of the developed ADP-based event-triggered method is verified through simulation studies.

Constrained Event-Triggered H ∞ Control Based on Adaptive Dynamic Programming With Concurrent Learning

In this article, an event-triggered <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control method is proposed based on adaptive dynamic programming (ADP) with concurrent learning for unknown continuous-time nonlinear systems with control constraints. First, a system identification technique based on neural networks (NNs) is adopted to identify completely unknown systems. Second, a critic NN is employed to approximate the value function. A novel weight updating rule is developed based on the event-triggered control law and time-triggered disturbance law, which reduces controller execution times and guarantees the stability of the system. Subsequently, concurrent learning is applied to the weight updating rule to relax the demand for the traditional persistence of excitation condition that is difficult to implement online. Finally, the comparison between the time-triggered method and event-triggered method in simulation demonstrates the effectiveness of the developed constrained event-triggered ADP method.

Observer-Based H Tracking Control Scheme and Its Application to Robot Arms

Abstract In this paper, an adaptive dynamic programming (ADP)-based H∞ tracking control method is proposed for unknown continuous-time nonlinear systems. By constructing a neural network (NN) observer with system input and output data, the dynamics of completely unknown systems can be identified online. Then, an augmented system composed of tracking error and reference trajectory dynamics is formulated, and the H∞ tracking control problem is regarded as a zero-sum game problem which is solved by using ADP technique. A critic NN is employed to approximate the cost function to solve the Hamilton-Jacobi-Isaacs equation. Furthermore, the ultimate uniform boundedness of observation errors, critic NN weights, and tracking errors are all proved through Lyapunov’s direct method. Finally, the proposed method is applied to a single link robot arm with unknown dynamics to demonstrate its effectiveness.

Adaptive Dynamics Programming for H∞ Control of Continuous-Time Unknown Nonlinear Systems via Generalized Fuzzy Hyperbolic Models

In this paper, a novel adaptive dynamic programming (ADP) algorithm is developed for the infinite-horizon ( $H_{\infty}$ ) optimal control problems with unknown continuous-time (CT) nonlinear systems subject to external disturbances. To facilitate the implementation of the algorithm, generalized fuzzy hyperbolic models (GFHMs) are utilized to establish an identifier–critic architecture, where the identifier is designed to reconstruct the unknown system dynamics, and the GFHM-based critic network is employed to approximate the value functions. The CT $H_{\infty}$ optimal control issue is converted into a two-player zero-sum game and the corresponding Hamilton–Jacobi–Isaacs equation is derived. The learning procedure of the critic design is adaptively implemented with the help of the reconstructed model, thus the requirement of the complete system dynamics is relaxed. Furthermore, by the means of Lyapunov direct method, the uniform ultimate boundedness stability analysis of the closed-loop control system is explicitly provided. Finally, to compare the control performances and disturbance attenuation properties of the proposed method and the existing ADP algorithms, two numerical examples are given.

Adaptive dynamic programming based event-triggered control for unknown continuous-time nonlinear systems with input constraints

An adaptive dynamic programming (ADP) based event-triggered control method is established in this paper to solve the optimal control problem of unknown continuous-time nonlinear systems with input constraints. First, the unknown system is identified using two neural networks (NNs). Second, the threshold for event-triggering condition is designed, which guarantees the system stability. Then, a critic NN is employed to approximate the value function and the closed-loop system is proved to be uniformly ultimately bounded. Finally, the simulation results demonstrate the feasibility of the developed ADP method. The main contributions of this paper are that the developed ADP-based event-triggered control method avoids the Zeno phenomenon effectively and updates the weights of three NNs simultaneously in the process of implementation. Meanwhile, an improved critic NN weight updating criterion is adopted, which does not require an initial admissible control.

Event-Triggered Reinforcement Learning-Based Adaptive Tracking Control for Completely Unknown Continuous-Time Nonlinear Systems.

In this paper, event-triggered reinforcement learning-based adaptive tracking control is developed for the continuous-time nonlinear system with unknown dynamics and external disturbances. The critic and action neural networks are designed to approximate an unknown long-term performance index and controller, respectively. The dead-zone event-triggered condition is developed to reduce communication and computational costs. Rigorous theoretical analysis is provided to show that the closed-loop system can be stabilized. The weight errors and the filtered tracking error are all uniformly ultimately bounded. Finally, to demonstrate the developed controller, the simulation results are provided using an autonomous underwater vehicle model.

Reinforcement learning for robust adaptive control of partially unknown nonlinear systems subject to unmatched uncertainties

This paper proposes a novel robust adaptive control strategy for partially unknown continuous-time nonlinear systems subject to unmatched uncertainties. Initially, the robust nonlinear control problem is converted into a nonlinear optimal control problem by constructing an appropriate value function for the auxiliary system. After that, within the framework of reinforcement learning, an identifier-critic architecture is developed. The presented architecture uses two neural networks: the identifier neural network (INN) which aims at estimating the unknown internal dynamics and the critic neural network (CNN) which tends to derive the approximate solution of the Hamilton-Jacobi-Bellman equation arising in the obtained optimal control problem. The INN is updated by using both the back-propagation algorithm and the e-modification technique. Meanwhile, the CNN is updated via the modified gradient descent method, which uses historical and current state data simultaneously. Based on the classic Lyapunov technique, all the signals in the closed-loop auxiliary system are proved to be uniformly ultimately bounded. Moreover, the original system is kept asymptotically stable under the obtained approximate optimal control. Finally, two illustrative examples, including the F-16 aircraft plant, are provided to demonstrate the effectiveness of the developed method.

Online reinforcement learning for a class of partially unknown continuous‐time nonlinear systems via value iteration

SummaryIn this paper, a modified value iteration–based approximate dynamic programming method is proposed for a class of affine nonlinear continuous‐time systems, whose dynamics are partially unknown. The value iteration algorithm is established in an online fashion, and the convergence proof is given. To attenuate the effect caused by the unascertained characteristics of the system dynamics, the integral reinforcement learning scheme is also used. In the proposed approximate dynamic programming method, it is emphasized that the single‐network structure is utilized to estimate the value functions and the control policies. That is, the iteration process is implemented on the actor/critic structure, in which case only the critic NN is required to be identified. Then, the least‐squares scheme is derived for the NN weights updating. Finally, a linear system and a nonlinear system are tested to evaluate the performance of the proposed online value iteration algorithm. Both of the examples show the feasibility and effectiveness of the proposed algorithms.

Reinforcement learning for adaptive optimal control of unknown continuous-time nonlinear systems with input constraints

In this paper, an adaptive reinforcement learning-based solution is developed for the infinite-horizon optimal control problem of constrained-input continuous-time nonlinear systems in the presence of nonlinearities with unknown structures. Two different types of neural networks (NNs) are employed to approximate the Hamilton–Jacobi–Bellman equation. That is, an recurrent NN is constructed to identify the unknown dynamical system, and two feedforward NNs are used as the actor and the critic to approximate the optimal control and the optimal cost, respectively. Based on this framework, the action NN and the critic NN are tuned simultaneously, without the requirement for the knowledge of system drift dynamics. Moreover, by using Lyapunov’s direct method, the weights of the action NN and the critic NN are guaranteed to be uniformly ultimately bounded, while keeping the closed-loop system stable. To demonstrate the effectiveness of the present approach, simulation results are illustrated.

Identification and control of continuous-time nonlinear systems via dynamic neural networks

In this paper, we present an algorithm for the online identification and adaptive control of a class of continuous-time nonlinear systems via dynamic neural networks. The plant considered is an unknown multi-input/multi-output continuous-time higher order nonlinear system. The control scheme includes two parts: a dynamic neural network is employed to perform system identification and a controller based on the proposed dynamic neural network is developed to track a reference trajectory. Stability analysis for the identification and the tracking errors is performed by means of Lyapunov stability criterion. Finally, we illustrate the effectiveness of these methods by computer simulations of the Duffing chaotic system and one-link rigid robot manipulator. The simulation results demonstrate that the model-based dynamic neural network control scheme is appropriate for control of unknown continuous-time nonlinear systems with output disturbance noise.

A regulator for a class of unknown continuous-time nonlinear systems

We introduce a control law for a class of unknown nonlinear continuous-time systems in which full state measurements are available. We show that as long as a certain feedback gain parameter is sufficiently large, the closed-loop system is stable. Furthermore, the magnitude of the control is bounded in the limit. As a corollary to the main result, we show that first-order nonlinear systems in the class we consider can be stabilized using linear proportional integral controllers, and we give explicit tuning rules for these controllers.