Continuous-time Nonlinear Systems Research Articles

Integral reinforcement learning-based dynamic event-triggered safety control for multiplayer Stackelberg–Nash games with time-varying state constraints

In this article, an integral reinforcement learning-based dynamic event-triggered safety control scheme is proposed to tackle the multiplayer Stackelberg–Nash games (MSNG) problem in continuous-time nonlinear systems with time-varying state constraints. Initially, a new barrier function (BF) is introduced by integrating traditional barrier functions with a novel smooth function to address the challenge of time-varying state constraints. Then, the constrained MSNG system is transformed into an unconstrained system through the application of the state transformation technique, which helps to characterize hierarchical decision problems as MSNG problems with the leader and followers. Meanwhile, the integral reinforcement learning (IRL) technique is also applied to ease the demand for precise system dynamics. Moreover, a new dynamic event-triggered control (DETC) mechanism is designed, resulting in coupled dynamic event-triggered Hamilton–Jacobi (HJ) equations. A single critic neural network (NN) is constructed to learn the optimal control laws for the leader and followers. By employing Lyapunov theory, the integral reinforcement learning-based dynamic event-triggered safety control method ensures that the system state and the critic NN weight errors are uniformly ultimately bounded (UUB). Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed algorithm.

Sliding-mode surface-based adaptive optimal nonzero-sum games for saturated nonlinear multi-player systems with identifier-critic networks

This paper addresses the sliding-mode surface-based adaptive optimal nonzero-sum games control problem for continuous-time nonlinear systems with input constraints. By constructing a value function associated with the sliding-mode surface, the original nonzero-sum games problem can be transformed into constructing multiple optimal control policies. Then, an adaptive dynamic programming method under identifier-critic networks is introduced to deal with the Hamilton–Jacobi equation, where the identifier networks are applied to approximate the unknown dynamics. A remarkable feature is that the critic updating laws are designed through experience replay and gradient descent methodologies, such that the persistence of excitation condition can be excluded. Based on the Lyapunov stability theory, all signals of the close-loop system are verified to be uniformly ultimately bounded. Finally, simulation results are given to illustrate the effectiveness of the proposed optimal nonzero-sum games scheme.

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

Convergence Analysis of Value Iteration Adaptive Dynamic Programming for Continuous-Time Nonlinear Systems.

This article is concerned with the convergence property and error bounds analysis of value iteration (VI) adaptive dynamic programming for continuous-time (CT) nonlinear systems. The size relationship between the total value function and the single integral step cost is described by assuming a contraction assumption. Then, the convergence property of VI is proved while the initial condition is an arbitrary positive semidefinite function. Moreover, the accumulated effects of approximation errors generated in each iteration are taken into consideration while using approximators to implement the algorithm. Based on the contraction assumption, the error bounds condition is proposed, which ensures the approximated iterative results converge to a neighborhood of the optimum, and the relation between the optimal solution and approximated iterative results is also derived. To make the contraction assumption more concrete, an estimation way is proposed to derive a conservative value of the assumption. Finally, three simulation cases are given to validate the theoretical results.

Robust stability of moving horizon estimation for continuous-time systems

Abstract We consider a moving horizon estimation (MHE) scheme involving a discounted least squares objective for general nonlinear continuous-time systems. Provided that the system is detectable (incrementally integral input/output-to-state stable, i-iIOSS), we show that there exists a sufficiently long estimation horizon that guarantees robust global exponential stability of the estimation error in a time-discounted L 2-to-L ∞ sense. In addition, we show that i-iIOSS Lyapunov functions can be efficiently constructed by verifying certain linear matrix inequality conditions. In combination, we propose a flexible Lyapunov-based MHE framework in continuous time, which particularly offers more tuning possibilities than its discrete-time analog, and provide sufficient conditions for stability that can be easily verified in practice. Our results are illustrated by a numerical example.

Continual online learning-based optimal tracking control of nonlinear strict-feedback systems: application to unmanned aerial vehicles

A novel optimal trajectory tracking scheme is introduced for nonlinear continuous-time systems in strict feedback form with uncertain dynamics by using neural networks (NNs). The method employs an actor-critic-based NN backstepping technique for minimizing a discounted value function along with an identifier to approximate unknown system dynamics that are expressed in augmented form. Novel online weight update laws for the actor and critic NNs are derived by using both the NN identifier and Hamilton-Jacobi-Bellman residual error. A new continual lifelong learning technique utilizing the Fisher Information Matrix via Hamilton-Jacobi-Bellman residual error is introduced to obtain the significance of weights in an online mode to overcome the issue of catastrophic forgetting for NNs, and closed-loop stability is analyzed and demonstrated. The effectiveness of the proposed method is shown in simulation by contrasting the proposed with a recent method from the literature on an underactuated unmanned aerial vehicle, covering both its translational and attitude dynamics.

Nearly optimal stabilization of unknown continuous-time nonlinear systems: A new parallel control approach

This paper develops a novel online nearly optimal control (ONOC) method for unknown continuous-time (CT) nonaffine nonlinear systems without recovering unknown systems. First, a dynamic control law is proposed for CT nonaffine nonlinear systems using parallel control. To achieve the proposed dynamic control law, an affine augmented system (AAS) is constructed according to the original system, and an augmented performance index (API) is constructed on the basis of the original performance index (OPI). Then, the stability relationship between the original system and the AAS is provided, and it is proven that, by selecting a suitable parameter in the API, optimal control of the AAS with the API is equivalent to near-optimal control of the original system with the OPI. Subsequently, based on the proposed dynamic control law, we extend integral reinforcement learning (IRL) to completely unknown CT nonaffine systems, and it is further proved that closed-loop signals are uniformly ultimately bounded (UUB) without the assumption that the input dynamics are bounded. Furthermore, the OPI can be set to an arbitrary positive-definite form, and the UUB bound for the state vector can be predetermined. Lastly, simulations are offered to exhibit the correctness of the developed ONOC method. Source code of this paper is available at: https://github.com/lujingweihh/Adaptive-dynamic-programming-algorithms/tree/main/model_free_integral_reinforcement_learning.

Hierarchical Sliding-Mode Surface-Based Adaptive Actor-Critic Optimal Control for Switched Nonlinear Systems With Unknown Perturbation.

This article studies the hierarchical sliding-mode surface (HSMS)-based adaptive optimal control problem for a class of switched continuous-time (CT) nonlinear systems with unknown perturbation under an actor-critic (AC) neural networks (NNs) architecture. First, a novel perturbation observer with a nested parameter adaptive law is designed to estimate the unknown perturbation. Then, by constructing an especial cost function related to HSMS, the original control issue is further converted into the problem of finding a series of optimal control policies. The solution to the HJB equation is identified by the HSMS-based AC NNs, where the actor and critic updating laws are developed to implement the reinforcement learning (RL) strategy simultaneously. The critic update law is designed via the gradient descent approach and the principle of standardization, such that the persistence of excitation (PE) condition is no longer needed. Based on the Lyapunov stability theory, all the signals of the closed-loop switched nonlinear systems are strictly proved to be bounded in the sense of uniformly ultimate boundedness (UUB). Finally, the simulation results are presented to verify the validity of the proposed adaptive optimal control scheme.

Output information-based intermittent optimal control for continuous-time nonlinear systems with unmatched uncertainties via adaptive dynamic programming

Intermittent control stands as a valuable strategy for resource conservation and cost reduction across diverse systems. Nonetheless, prevailing research is intractable to address the challenges posed by robust optimal intermittent control of nonlinear input-affine systems with unmatched uncertainties. This paper aims to fill this gap. Initially, we introduce an enhanced finite-time intermittent control approach to ensure stability within nonlinear dynamic systems harboring bounded errors. A neural networks (NNs) state observer is constructed to estimate system information. Subsequently, an optimal intermittent controller that operates within a finite time span, guaranteeing system stability by employing the Hamilton–Jacobi–Bellman (HJB) methodology. Furthermore, we devise an output information-based event-triggered intermittent (ETI) approach rooted in the robust adaptive dynamic programming (ADP) algorithm, furnishing an optimal intermittent control law. In this process, a critic NNs is introduced to estimate the cost function and optimal intermittent controller. Simulation results show that our proposed method is superior to existing intermittent control strategies.

Optimal Robust Control of Nonlinear Systems with Unknown Dynamics via NN Learning with Relaxed Excitation.

This paper presents an adaptive learning structure based on neural networks (NNs) to solve the optimal robust control problem for nonlinear continuous-time systems with unknown dynamics and disturbances. First, a system identifier is introduced to approximate the unknown system matrices and disturbances with the help of NNs and parameter estimation techniques. To obtain the optimal solution of the optimal robust control problem, a critic learning control structure is proposed to compute the approximate controller. Unlike existing identifier-critic NNs learning control methods, novel adaptive tuning laws based on Kreisselmeier's regressor extension and mixing technique are designed to estimate the unknown parameters of the two NNs under relaxed persistence of excitation conditions. Furthermore, theoretical analysis is also given to prove the significant relaxation of the proposed convergence conditions. Finally, effectiveness of the proposed learning approach is demonstrated via a simulation study.

Distributed Lebesgue Approximation Model for Distributed Continuous-Time Nonlinear Systems.

Approximation models play a crucial role in model-based methods, as they enhance both accuracy and computational efficiency. This article studies distributed and asynchronous discretized models to approach continuous-time nonlinear systems. The considered continuous-time system consists of some distributed but physically coupled nonlinear subsystems that exchange information. We propose two Lebesgue approximation models (LAMs): 1) the unconditionally triggered LAM (CT-LAM) and 2) the CT-LAM. In both approaches, a specific LAM approximates an individual subsystem. The iteration of each LAM is triggered by either itself or its neighbors. The collection of different LAMs executing asynchronously together form the approximation of the overall distributed continuous-time system. The aperiodic nature of LAMs allows for a reduction in the number of iterations in the approximation process, particularly when the system has slow dynamics. The difference between the unconditionally and CT-LAMs is that the latter checks an "importance" condition, further reducing the computational effort in individual LAMs. Furthermore, the proposed LAMs are analyzed by constructing a distributed event-triggered system which is proved to have the same state trajectories as the LAMs with linear interpolation. Through this specific event-triggered system, we derive conditions on the quantization sizes in LAMs to ensure asymptotic stability of the LAMs, boundedness of the state errors, and prevention of Zeno behavior. Finally, simulations are carried out on a quarter-car suspension system to show the advantage and efficiency of the proposed approaches.

Robust Optimal Parallel Tracking Control Based on Adaptive Dynamic Programming.

This article focuses on a novel robust optimal parallel tracking control method for continuous-time (CT) nonlinear systems subject to uncertainties. First, the designed virtual controller facilitates the transformation of the original nonlinear system into an affine system with an augmented state vector, which promotes the introduction of the optimal parallel tracking control problem. Then, this article generates fresh insight into counteracting the effects of uncertainty by developing a novel parallel control system that invokes the formulated virtual control law and an auxiliary variable obtained from the relationship between the solutions of the optimal control problems for the uncertain system and the nominal one. Next, critic neural networks (NNs) approximate the Hamilton-Jacobi-Bellman (HJB) equations' solution to implement the proposed robust optimal control method via adaptive dynamic programming (ADP). Finally, simulation experiments demonstrate the proposed method's remarkable effectiveness.

Event-Triggered Robust Adaptive Dynamic Programming for Multiplayer Stackelberg-Nash Games of Uncertain Nonlinear Systems.

In this article, an event-triggered robust adaptive dynamic programming (ETRADP) algorithm is developed to solve a class of multiplayer Stackelberg-Nash games (MSNGs) for uncertain nonlinear continuous-time systems. Considering the different roles of players in the MSNG, the hierarchical decision-making process is described as the designed value functions for the leader and all followers, which assist to transform the robust control problem of the uncertain nonlinear system into an optimal regulation problem of the nominal system. Then, an online policy iteration algorithm is formulated to solve the derived coupled Hamilton-Jacobi equation. Meanwhile, an event-triggered mechanism is designed to alleviate computational and communication burdens. Moreover, critic neural networks (NNs) are constructed to obtain the event-triggered approximate optimal control polices for all players, which constitute the Stackelberg-Nash equilibrium of the MSNG. By using Lyapunov's direct method, the stability of the closed-loop uncertain nonlinear system is guaranteed under the ETRADP-based control scheme in the sense of uniform ultimate boundedness. Finally, a numerical simulation is provided to demonstrate the effectiveness of the present ETRADP-based control scheme.

Almost Sure Stability of Dual Switching Continuous-time Nonlinear System With Deterministic and Stochastic Subsystems

Almost Sure Stability of Dual Switching Continuous-time Nonlinear System With Deterministic and Stochastic Subsystems

Robust model predictive control for continuous nonlinear systems with the quasi-infinite adaptive horizon algorithm

The control input derived from model predictive control (MPC) scheme depends on the receding horizon whose length needs to be determined. Design length is also a trade-off between computational complexity and optimization metrics. To this end, a framework for adaptive selection of the horizon length has been developed for discrete systems in previous works. In this paper, this framework is extended to continuous nonlinear systems. A new algorithm is presented to solve the problem of how to adaptively select the horizon length. Then, according to the traditional method and the robust MPC scheme based on tubes, the recursive feasibility and robustness of the algorithm (the adaptive horizon nonlinear model predictive control, called AH-NMPC) for quasi-infinite time domain adaptive horizontal continuous-time nonlinear systems are proved. Finally, the NMPC controller is used to perform path-tracking experiments on an autonomous vehicle to verify the good effects of the algorithm (AH-NMPC).

Semi-global asymptotic state agreement of nonlinear multi-agent systems with communication delays under directed switching topologies

This paper investigates the semi-global state agreement problem for continuous-time nonlinear multi-agent systems with communication delays under direct and switching topologies. A systematic solution to the semi-global state agreement is presented and based on which a sufficient and necessary condition on the control laws is obtained, as long as the direct and switching communication topologies satisfy a mild joint connectivity assumption. Illustrative simulations are provided to validate the efficacy of the proposed theoretical results.

Periodic Event-Triggered MPC for Continuous-Time Nonlinear Systems With Bounded Disturbances

This paper is concerned with periodic event-triggering laws in robust model predictive control (MPC) for continuous-time constrained nonlinear systems. The online optimal control problem solved at triggering times is introduced. A periodic static event-triggering condition, in which a fixed sampling time interval plays an important role in avoiding Zeno behavior, is presented to alleviate continuous checking of event detections. Then a periodic dynamic event-triggering condition is investigated to further enlarge the minimal inter-execution time. Single-mode MPC with a prediction horizon larger than the control one is considered. Sufficient conditions of recursive feasibility for the online optimal control problem are derived. In order to relax the sufficient condition of stability, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Ultimately Boundedness</i> properties are utilized in stability analysis. Finally, numerical simulation is provided to demonstrate the effectiveness of the proposed methods.

Distributed event-triggered fuzzy control for nonlinear interconnected systems

This paper deals with the asynchronous distributed event-triggered control of continuous-time nonlinear interconnected systems. The nonlinear dynamics of the subsystems are represented by Takagi–Sugeno fuzzy models via the sector nonlinearity approach. Moreover, the nonlinear interconnections among the subsystems are assumed to be known and sector-bounded. In the event-triggered control setup, where the state information is asynchronously available to the local controllers only in specific time instants, it is necessary to deal with the asynchronous premise variables because they introduce extra difficulties to derive suitable co-design conditions. To deal with this issue, we propose a new triggering strategy such that the local event-triggering mechanisms (ETMs) counteract the effects of asynchronous premise variables. With this new distributed ETM, a co-design condition is proposed, and the existence of a strictly positive minimum inter-event time is proved to exclude Zeno behaviour. Moreover, a multi-objective optimization procedure is introduced to enlarge the estimate of the domain of attraction of the closed-loop equilibrium and minimize the number of transmissions provided by the ETMs. Finally, the approach is validated through the synchronization of interconnected oscillators.

On the state estimation for nonlinear continuous-time fuzzy systems

On the state estimation for nonlinear continuous-time fuzzy systems

Periodic Event-Triggered Robust Distributed Model Predictive Control for Multiagent Systems With Input and Communication Delays

This paper investigates the problem of event-triggered distributed model predictive control (DMPC) for continuous-time nonlinear multi-agent systems (MASs) subject to bounded disturbances, input and communication delays simultaneously. The compensation schemes for input and communication delays are proposed, respectively. A new optimal control problem (OCP) to achieve consensus among all agents is formulated under the proposed delay compensation schemes. To alleviate the communication burden and sensing cost, a novel input delay-related periodic event-triggering condition is proposed based on the state error to determine when to calculate the new control input. Sufficient condition guaranteeing the feasibility of the OCP is achieved with the proposed event-triggered control scheme. In stability analysis, a novel time-varying Lyapunov function is constructed, and the input-to-state practical stability (ISpS) of the MASs is derived by using the quasi-Lipschitz continuous property of the Lyapunov function. Numerical results related to vehicle platooning demonstrate the effectiveness of the proposed event-triggered DMPC.