Unknown Drift Dynamics Research Articles

Optimal Asymptotic Tracking Control for Nonzero-Sum Differential Game Systems with Unknown Drift Dynamics via Integral Reinforcement Learning

This paper employs an integral reinforcement learning (IRL) method to investigate the optimal tracking control problem (OTCP) for nonlinear nonzero-sum (NZS) differential game systems with unknown drift dynamics. Unlike existing methods, which can only bound the tracking error, the proposed approach ensures that the tracking error asymptotically converges to zero. This study begins by constructing an augmented system using the tracking error and reference signal, transforming the original OTCP into solving the coupled Hamilton–Jacobi (HJ) equation of the augmented system. Because the HJ equation contains unknown drift dynamics and cannot be directly solved, the IRL method is utilized to convert the HJ equation into an equivalent equation without unknown drift dynamics. To solve this equation, a critic neural network (NN) is employed to approximate the complex value function based on the tracking error and reference information data. For the unknown NN weights, the least squares (LS) method is used to design an estimation law, and the convergence of the weight estimation error is subsequently proven. The approximate solution of optimal control converges to the Nash equilibrium, and the tracking error asymptotically converges to zero in the closed system. Finally, we validate the effectiveness of the proposed method in this paper based on MATLAB using the ode45 method and least squares method to execute Algorithm 2.

An analytical adaptive optimal control approach without solving HJB equation for nonlinear systems with input constraints

AbstractThis paper presents an analytical method to solve the optimal control problem for affine nonlinear systems with unknown drift dynamics. A new non‐quadratic cost function over an infinite horizon is presented that considers input constraints and includes the cost of the feed‐forward component of the control law. The mean value theorem for vector‐valued functions has been used to derive an integral form of this theorem. Based on this theorem, a rigorous proof is provided demonstrating that the cost function can be converted into another form. In the presence of input constraints, this converted form enables extracting the optimal control solution without solving the HJB equation. Additionally, unknown nonlinearity effects in drift dynamics are compensated in the control input. This is accomplished by estimating the unknown drift dynamics via an adaptive neural network (NN) approach. It is proven that the states and weights of NN are uniformly ultimately bounded based on a Lyapunov technique. The necessary and sufficient conditions are provided that ensure the optimality of the infinite horizon optimal control problem with a discount factor. As a result, it is demonstrated that the proposed approach satisfies the optimality criteria. To evaluate the effectiveness of the proposed approach, simulation examples are provided.

Hierarchical Reinforcement Learning-based Supervisory Control of Unknown Nonlinear Systems

A supervisory control approach using hierarchical reinforcement learning (HRL) is developed to approximate the solution to optimal regulation problems for a control-affine, continuous-time nonlinear system with unknown drift dynamics. This result contains two objectives. The first objective is to approximate the optimal control policy that minimizes the infinite horizon cost function of each approximate dynamic programming (ADP) sub-controller. The second objective is to design a switching rule, by comparing the approximated optimal value functions of the ADP sub-controllers, to ensure that switching between subsystems yields a lower cost than using one subsystem. An integral concurrent learning-based parameter identifier approximates the unknown drift dynamics. Uniformly ultimately bounded regulation of the system states to a neighborhood of the origin, and convergence of the approximate control policy to a neighborhood of the optimal control policy, are proven using a Lyapunov-based stability and dwell-time analysis.

Data‐Driven Optimal Tracking Control of Nonlinear Affine Systems Based on Differential Dynamic Programming

In this paper, a data‐driven differential dynamic programming (DDP) algorithm is presented for the optimal tracking problems of nonlinear affine systems with unknown drift dynamics. Optimal tracking dynamics are established between the system states and expected trajectory. By using the DDP method and test data, a second‐order data‐driven framework of the Hamilton–Jacobi–Bellman (HJB) equation is constructed, in which system approximation and value function approximation are used. By using this framework, a data‐driven iteration algorithm is proposed to achieve the optimal tracking controller. A simulation example is provided to verify the effectiveness of the proposed approach.

Real-Time Modular Deep Neural Network-Based Adaptive Control of Nonlinear Systems

A real-time deep neural network (DNN) adaptive control architecture is developed for uncertain control-affine nonlinear systems to track a time-varying desired trajectory. A Lyapunov-based analysis is used to develop adaptation laws for the output-layer weights and develop constraints for inner-layer weight adaptation laws. Unlike existing works in neural network and DNN-based control, the developed method establishes a framework to simultaneously update the weights of multiple layers for a DNN of arbitrary depth in real-time. The real-time controller and weight update laws enable the system to track a time-varying trajectory while compensating for unknown drift dynamics and parametric DNN uncertainties. A nonsmooth Lyapunov-based analysis is used to guarantee semi-global asymptotic tracking. Comparative numerical simulation results are included to demonstrate the efficacy of the developed method.

Online event-triggered adaptive critic design for multi-player zero-sum games of partially unknown nonlinear systems with input constraints

This paper focuses on the design of online event-triggered optimal control strategy for multi-player zero-sum games (MP-ZSGs) with control constraints when the system model is partially unknown. Non-quadratic functions are utilized to construct the cost functions under the condition of control constraints. The proposed algorithm is designed based on the framework of identifier-critic networks. The unknown drift dynamics model is reconstructed by an identifier neural network (INN) using the input and output data. The near-optimal event-based controls and time-based disturbances are designed by training a critic neural network (CNN). With the aid of the designed event-triggered mechanism (ETM), the needless computing and communication actions of the system signals have been reduced so as to save computing/communication resources. Meanwhile, to remove the persistence of excitation (PE) condition, the historical and current data are utilized to construct a modified tuning law of CNN. Theoretically, the uniform ultimate boundedness (UUB) properties of the system states and the critic weights errors are proved by Lyapunov approach. Moreover, the Zeno behavior is proved to be excluded under the designed triggering condition. Finally, the convergence and performance of the online method are verified by simulating a representative example.

Lyapunov-Based Real-Time and Iterative Adjustment of Deep Neural Networks

A real-time Deep Neural Network (DNN) adaptive control architecture is developed for general uncertain nonlinear dynamical systems to track a desired time-varying trajectory. A Lyapunov-based method is leveraged to develop adaptation laws for the output-layer weights of a DNN model in real-time while a data-driven supervised learning algorithm is used to update the inner-layer weights of the DNN. Specifically, the output-layer weights of the DNN are estimated using an unsupervised learning algorithm to provide responsiveness and guaranteed tracking performance with real-time feedback. The inner-layer weights of the DNN are trained with collected data sets to increase performance, and the adaptation laws are updated once a sufficient amount of data is collected. Building on the results in (Joshi and Chowdhary, 2019) and (Joshi et al., 2020), which focus on deep model reference adaptive control for linear systems with known drift dynamics and control effectiveness matrices, this letter considers general control-affine uncertain nonlinear systems. The real-time controller and adaptation laws enable the system to track a desired time-varying trajectory while compensating for the unknown drift dynamics and parameter uncertainties in the control effectiveness. A nonsmooth Lyapunov-based analysis is used to prove semi-global asymptotic tracking of the desired trajectory. Numerical simulation examples are included to validate the results, and the Levenberg-Marquardt algorithm is used to train the weights of the DNN.

Reinforcement Learning-Based Nearly Optimal Control for Constrained-Input Partially Unknown Systems Using Differentiator

In this article, a synchronous reinforcement-learning-based algorithm is developed for input-constrained partially unknown systems. The proposed control also alleviates the need for an initial stabilizing control. A first-order robust exact differentiator is employed to approximate unknown drift dynamics. Critic, actor, and disturbance neural networks (NNs) are established to approximate the value function, the control policy, and the disturbance policy, respectively. The Hamilton-Jacobi-Isaacs equation is solved by applying the value function approximation technique. The stability of the closed-loop system can be ensured. The state and weight errors of the three NNs are all uniformly ultimately bounded. Finally, the simulation results are provided to verify the effectiveness of the proposed method.

Data-based reinforcement learning approximate optimal control for an uncertain nonlinear system with control effectiveness faults

An infinite horizon approximate optimal control problem is developed for a system with unknown drift parameters and control effectiveness faults. A data-based filtered parameter estimator with a novel dynamic gain structure is developed to simultaneously estimate the unknown drift dynamics and control effectiveness fault. A local state-following approximate dynamic programming method is used to approximate the unknown optimal value function for an uncertain system. Using a relaxed persistence of excitation condition, a Lyapunov-based stability analysis shows exponential convergence to a residual error for the parameter estimation and uniformly ultimately bounded convergence for the closed-loop system. Simulation results are presented which demonstrate the effectiveness of the developed method.

Event‐triggered distributed H ∞ control of physically interconnected mobile Euler–Lagrange systems with slipping, skidding and dead zone

This study addresses an event-triggered distributed ℋ ∞ control method by extending traditional zero-sum differential games for physically interconnected non-holonomic mobile mechanical multi-agent systems with external disturbance and slipping, skidding and dead-zone disturbances. Initially, a problem of physically interconnected kinematic and dynamic control is transformed into an equivalent problem of event-triggered distributed ℋ ∞ control. Subsequently, the traditional two-player zerosum differential game is extended to a three-player zero-sum differential game, where a new player is included to approximate the worst dead-zone disturbance. To find player policies, an event-triggering condition and an event-triggered control law are proposed via neural networks (NNs). Although an NN weight-tuning law is designed on the basis of adaptive dynamic programming techniques, it can relax identification procedures for unknown drift dynamics and persistent excitation conditions. It also guarantees that the closed system is stable and the cost function converges to the bounded ℒ 2 -gain optimal value, while the Zeno behaviour is excluded. Finally, the effectiveness of the proposed method is verified by an application to a dead-zone torque multi-mobile robot system through numerical simulations.

Data-Based Reinforcement Learning for Nonzero-Sum Games With Unknown Drift Dynamics.

This paper is concerned about the nonlinear optimization problem of nonzero-sum (NZS) games with unknown drift dynamics. The data-based integral reinforcement learning (IRL) method is proposed to approximate the Nash equilibrium of NZS games iteratively. Furthermore, we prove that the data-based IRL method is equivalent to the model-based policy iteration algorithm, which guarantees the convergence of the proposed method. For the implementation purpose, a single-critic neural network structure for the NZS games is given. To enhance the application capability of the data-based IRL method, we design the updating laws of critic weights based on the offline and online iterative learning methods, respectively. Note that the experience replay technique is introduced in the online iterative learning, which can improve the convergence rate of critic weights during the learning process. The uniform ultimate boundedness of the critic weights are guaranteed using the Lyapunov method. Finally, the numerical results demonstrate the effectiveness of the data-based IRL algorithm for nonlinear NZS games with unknown drift dynamics.

Event-Triggered Optimal Control for Partially Unknown Constrained-Input Systems via Adaptive Dynamic Programming

Event-triggered control has been an effective tool in dealing with problems with finite communication and computation resources. In this paper, we design an event-triggered control for nonlinear constrained-input continuous-time systems based on the optimal policy. Constraints on controls are handled using a bounded function. To learn the optimal solution with partially unknown dynamics, an online adaptive dynamic programming algorithm is proposed. The identifier network, the critic network, and the actor network are employed to approximate the unknown drift dynamics, the optimal value, and the optimal policy, respectively. The identifier is tuned based on online data, which further trains the critic and actor at triggering instants. A concurrent learning technique repeatedly uses past data to train the critic. Stability of the closed-loop system, and convergence of neural networks to the optimal solutions are proved by Lyapunov analysis. In the end, the algorithm is applied to the overhead crane system to observe the performance. The event-triggered optimal controller with constraints stabilizes the system and consumes much less sampling times.

Model-Based Reinforcement Learning for Infinite-Horizon Approximate Optimal Tracking.

This brief paper provides an approximate online adaptive solution to the infinite-horizon optimal tracking problem for control-affine continuous-time nonlinear systems with unknown drift dynamics. To relax the persistence of excitation condition, model-based reinforcement learning is implemented using a concurrent-learning-based system identifier to simulate experience by evaluating the Bellman error over unexplored areas of the state space. Tracking of the desired trajectory and convergence of the developed policy to a neighborhood of the optimal policy are established via Lyapunov-based stability analysis. Simulation results demonstrate the effectiveness of the developed technique.