Optimal Tracking Control Problem Research Articles

Safe reinforcement learning-based control using deep deterministic policy gradient algorithm and slime mould algorithm with experimental tower crane system validation

This paper presents a novel optimal control approach resulting from the combination between the safe Reinforcement Learning (RL) framework represented by a Deep Deterministic Policy Gradient (DDPG) algorithm and a Slime Mould Algorithm (SMA) as a representative nature-inspired optimization algorithm. The main drawbacks of the traditional DDPG-based safe RL optimal control approach are the possible instability of the control system caused by randomly generated initial values of the controller parameters and the lack of state safety guarantees in the first iterations of the learning process due to (i) and (ii): (i) the safety constraints are considered only in the DDPG-based training process of the controller, which is usually implemented as a neural network (NN); (ii) the initial values of the weights and the biases of the NN-based controller are initialized with randomly generated values. The proposed approach mitigates these drawbacks by initializing the parameters of the NN-based controller using SMA. The fitness function of the SMA-based initialization process is designed to incorporate state safety constraints into the search process, resulting in an initial NN-based controller with embedded state safety constraints. The proposed approach is compared to the classical one using real-time experimental results and performance indices popular for optimal reference tracking control problems and based on a state safety score.

Reinforcement learning-based optimal tracking control for uncertain multi-agent systems with uncertain topological networks

Recent decades, extensive applications examplified in intelligent connected vehicles (ICVs) and unmanned aerial vehicles (UAVs) have emerged with the rapidly development of multi-agent systems (MASs). Inspired by these applications, the optimal tracking control problem for uncertain MASs under uncertain topological networks is addressed based on the theory of observer design and reinforcement learning (RL). Thus, an adaptive extended observer based on concurrent learning (CL) technique is designed to simultaneously estimate system states and unknown parameters, where unknown parameters estimated convergence is guaranteed in a relaxed persistence of excitation condition. Moreover, a Luenberger observer is designed to estimate the state of the leader under uncertain topological networks, which acts as the information compensation of the leader. Via the proposed observers, an optimal tracking control algorithm is devised leveraging actor-critic (AC)-neural network (NN), which does not require the state derivative information. Lastly, a numerical simulation is performed to demonstrate the validity of the scheme in question.

Machine learning model‐based optimal tracking control of nonlinear affine systems with safety constraints

AbstractThis work focuses on the development of a machine learning (ML) model‐based framework for safe optimal tracking control of a class of nonlinear control‐affine systems to ensure simultaneous closed‐loop stability and safety. Specifically, a novel multilayer feedforward neural network (FNN) with a control‐affine architecture is designed to model nonlinear dynamic systems. Subsequently, a model‐based reinforcement learning (RL) framework is presented, utilizing a novel cost function with Control Lyapunov‐Barrier Function (CLBF) properties, to learn both the control policy and the optimal value function for an infinite‐horizon optimal tracking control problem for nonlinear systems with safety constraints. The efficacy of the proposed methodology is demonstrated through simulations of a one‐link robot manipulator and a chemical process example.

Output Feedback Adaptive Optimal Control of Multiple Unmanned Marine Vehicles with Unknown External Disturbance

This paper presents an optimal output-feedback tracking control problem for multiple unmanned marine vehicles (UMVs) to track a desired trajectory. To guarantee the control objective in an optimal manner, adaptive dynamic programming (ADP) with optimal compensation terms is adopted. A neural velocity observer is designed based on a neural network (NN) to estimate the unmeasured system states and the unknown system dynamics. Furthermore, a disturbance observer (DO) is proposed to go against the effect of the unknown external disturbance of the sea environment. It is proved that the proposed controller can guarantee that all signals in the closed-loop system are bounded. Simulation results are given to demonstrate the effectiveness of the proposed control algorithm.

Data-driven optimal cooperative tracking control for heterogeneous multi-agent systems

This paper presents a novel hierarchical control scheme for solving the data-driven optimal cooperative tracking control problem of heterogeneous multi-agent systems. Considering that followers cannot communicate with the leader, a prescribed-time fully distributed observer is devised to estimate the leader’s state for each follower. Then, the data-driven decentralized controller is designed to ensure that the follower’s output can track the leader’s one. Compared with the existing results, the advantages of the designed distributed observer are that the prescribed convergence time is completely predetermined by the designer, and the design of the observer gain is independent of the global topology information. Besides, the advantages of the designed decentralized controller are that neither the follower’s system model nor a known initial stabilizing control policy is required. Finally, simulation results exemplify the advantage of the proposed method.

Deep Koopman-operator-based model predictive control for free-floating space robots with disturbance observer

This paper investigates the optimal tracking control problem of free-floating space robots in the presence of non-ideal factors, such as model uncertainties and external disturbances. To address this issue, we first leverage the deep Koopman operator, facilitated with a deep neural network, to establish an offline formulation of a global linearization model for a micro-nano free-floating space robot. Based on the estimated linearization model, we then employ the model predictive control method for online optimization control, achieving a significantly reduced computational burden. Additionally, a decay factor is integrated into the model predictive control optimization objective function to balance the precision of joint angles tracking with the suppression of base satellite attitude disturbance. To further address the modeling inaccuracies and external disturbances, we incorporate a disturbance observer based on a radial basis function neural network for online compensation within the model predictive control framework. This augmentation enhances tracking precision and robustness. Several groups of simulation results are carried out to demonstrate the effectiveness of the proposed method, showing its capability of energy consumption, disturbance rejection and enhanced robustness. These highlight the potential of the proposed method to improve the control performance of free-floating space robots, even in the absence of a precise dynamic model.

Optimal Asymptotic Tracking Control for Nonzero-Sum Differential Game Systems with Unknown Drift Dynamics via Integral Reinforcement Learning

This paper employs an integral reinforcement learning (IRL) method to investigate the optimal tracking control problem (OTCP) for nonlinear nonzero-sum (NZS) differential game systems with unknown drift dynamics. Unlike existing methods, which can only bound the tracking error, the proposed approach ensures that the tracking error asymptotically converges to zero. This study begins by constructing an augmented system using the tracking error and reference signal, transforming the original OTCP into solving the coupled Hamilton–Jacobi (HJ) equation of the augmented system. Because the HJ equation contains unknown drift dynamics and cannot be directly solved, the IRL method is utilized to convert the HJ equation into an equivalent equation without unknown drift dynamics. To solve this equation, a critic neural network (NN) is employed to approximate the complex value function based on the tracking error and reference information data. For the unknown NN weights, the least squares (LS) method is used to design an estimation law, and the convergence of the weight estimation error is subsequently proven. The approximate solution of optimal control converges to the Nash equilibrium, and the tracking error asymptotically converges to zero in the closed system. Finally, we validate the effectiveness of the proposed method in this paper based on MATLAB using the ode45 method and least squares method to execute Algorithm 2.

Intelligent-Critic-Based Tracking Control of Discrete-Time Input-Affine Systems and Approximation Error Analysis With Application Verification.

In recent years, the application of function approximators, such as neural networks and polynomials, has ushered in a new stage of development in solving optimal control problems. However, considering the existence of approximation errors, the stability of the controlled system cannot be guaranteed. Therefore, in view of the prevalence of approximation errors, we investigate optimal tracking control problems for discrete-time systems. First, a novel value function is introduced into the intelligent critic framework. Second, an implicit method is utilized to demonstrate the boundedness of the iterative value functions with approximation errors. An explicit method is applied to prove the stability of the system with approximation errors. Furthermore, an evolving policy is designed to iteratively tackle the optimal tracking control problem and demonstrate the stability of the system. Finally, the effectiveness of the developed method is verified through numerical as well as practical examples.

A variable-barrier-function-based prescribed finite-time bounded-[formula omitted] reinforcement learning optimal tracking control strategy without dependence on initial condition

The prescribed finite-time bounded-H∞ optimal tracking control (OTC) problem is investigated for a class of nonlinear systems with unknown initial tracking condition and external disturbances in this paper. A novel variable barrier function is designed to develop a new approach of prescribed performance control (PPC) which is irrelevant to the initial condition of the constrained variable in the system. Meanwhile, the reinforcement learning (RL) method with the actor–critic architecture is used to optimize the control input of the system in order to obtain the least energy consumption. By means of a newly proposed prescribed finite-time performance function (PFTPF), a prescribed finite-time optimal tracking controller is obtained regardless of initial tracking condition of the system. At the same time, the bounded-H∞ control problem is considered for the external disturbances in the system. The designed controller not only ensures the boundedness of all signals in the closed-loop system, but also has a finite-time control performance and an H∞ anti-interference performance. Finally, the effectiveness and the superiority of this proposed method are verified by the simulations.

Observer‐based optimal fault‐tolerant tracking control for input‐constrained interconnected nonlinear systems with mismatched disturbances

SummaryThis paper investigates an observer‐based optimal fault‐tolerant tracking control problem for interconnected nonlinear systems with input constraints and mismatched disturbances via adaptive dynamic programming (ADP). Firstly, an augmented system consisting of tracking error and the reference trajectory is constructed, and then the original optimal tracking control problem is transformed into the optimal regulation control problem of the augmented system. An integral sliding‐mode‐based optimal fault‐tolerant control scheme is developed to eliminate the effect of actuator faults and guarantee the optimal control performance. Furthermore, a neural network‐based observer is designed to identify the completely unknown system dynamics based on the input‐output data, thereby relaxing the restriction on system dynamics. Subsequently, the modified Hamilton‐Jacobi‐Bellman equations are solved by using the ADP algorithm under a critic network framework. According to the Lyapunov approach, all signals in the closed‐loop augmented system are uniformly ultimately bounded. Finally, the effectiveness of the developed control scheme is demonstrated via simulation results.

Integral reinforcement learning-based event-triggered optimal tracking control for modular robot manipulators via non-zero-sum game

Abstract Under an event-triggered mechanism, a non-zero-sum (NZS) game optimal tracking control method for modular robot manipulator (MRM) systems with input constraints is proposed using the adaptive dynamic programming (ADP) method based on integral reinforcement learning (IRL). First, a dynamic model of the MRM system is developed based on joint torque feedback technology, consisting of an n-joint subsystem related to interconnected dynamic coupling (IDC). Second, we design a robust compensation controller to handle the known model term and an optimal compensation controller to deal with the uncertainty term caused by the IDC and friction, respectively. In addition, a nonlinear disturbance observer is established to dispose of the negative effects caused by the uncertain sensor output disturbance. Third, based on differential game theory, we transform the optimal tracking control problem of the MRM system into an n-player NZS game problem. Then, the IRL-based ADP method is adopted, which relaxes the need for system partial unknown dynamic information, and only a critic neural network is used to solve the coupled Hamilton–Jacobi equation, so as to obtain the optimal control policy. Then, using Lyapunov theory, the tracking error of the MRM system is demonstrated to be uniformly ultimately bounded. Finally, the effectiveness and superiority of the proposed algorithm are verified through experiments.

Advanced Optimal Tracking Control With Stability Guarantee via Novel Value Learning Formulation.

In this article, to solve the optimal tracking control problem (OTCP) for discrete-time (DT) nonlinear systems, general value iteration (GVI) scheme and online value iteration (VI) algorithms with novel value function are discussed. First, the disadvantage of the traditional value function for the OTCP is presented and the novel value function is introduced. Second, we analyze the monotonicity and convergence of GVI and establish the admissibility condition of GVI to evaluate the admissibility of the current iterative control. Note that a novel approach is introduced to analyze the admissibility. Third, based on the attraction domain, improved control policies with online VI can be obtained by judging the location of the current tracking error and reference point. Finally, the stability of the online VI-based control system is guaranteed. Besides, we provide two simulation examples to show the performance of the proposed methods.

Integral reinforcement learning‐based optimal tracking control for uncertain nonlinear systems under input constraint and specified performance constraints

AbstractThis article addresses the optimal tracking control problem with prescribed performance for uncertain nonlinear systems subject to input constraint and unknown disturbances. First, a fixed‐time monotonic convergence function is introduced to restrain tracking error, and a nonlinear mapping technique is employed to transform the constrained error into an unconstrained variable, then the fixed‐time output tracking issue is boiled down to the boundedness problem of the transformed variable. With the aid of a nonquadratic cost function, the input constraint is encoded into the optimization problem. To solve the unknown disturbances, an auxiliary system and an auxiliary disturbance policy are constructed, and the optimal control problem is formulated as a two‐player zero‐sum game. Moreover, a Hamilton–Jacobi–Isaacs (HJI) equation associated with this nonquadratic zero‐sum game is established to give the optimal control and the worst‐case disturbance policy solution. Subsequently, to avoid using knowledge of the system dynamics, three neural network approximators, namely, actor, critic, and disturbance, which are tuned online and simultaneously for approximating the solution of HJI, are constructed based on the integral reinforcement learning algorithm. Theoretical analysis shows that the reconstructed error system states and the weight estimation errors are semi‐globally uniformly ultimately bounded. Finally, the simulation study further tests the availability of the proposed control strategy.

Quadratic Tracking Control of Linear Stochastic Systems with Unknown Dynamics Using Average Off-Policy Q-Learning Method

This article investigates the optimal tracking control problem for data-based stochastic discrete-time linear systems. An average off-policy Q-learning algorithm is proposed to solve the optimal control problem with random disturbances. Compared with the existing off-policy reinforcement learning (RL) algorithm, the proposed average off-policy Q-learning algorithm avoids the assumption of an initial stability control. First, a pole placement strategy is used to design an initial stable control for systems with unknown dynamics. Second, the initial stable control is used to design a data-based average off-policy Q-learning algorithm. Then, this algorithm is used to solve the stochastic linear quadratic tracking (LQT) problem, and a convergence proof of the algorithm is provided. Finally, numerical examples show that this algorithm outperforms other algorithms in a simulation.

Robust optimal tracking control of multiple autonomous underwater vehicles subject to uncertain disturbances

AbstractThis paper considers the problem of robust optimal tracking control of multiple autonomous underwater Vehicles (AUVs) subject to uncertain external disturbances. First, the Takagi‐Sugeno (T‐S) fuzzy based technique is utilized to convert the high‐order nonlinear multi‐AUV system into a series of linearized subsystems. Second, a novel fully distributed sliding mode control (FDSMC) strategy is proposed to attenuate the disturbances. Meanwhile, the leader‐following consensus and the nearly optimization of the energy‐cost function for the multi‐AUV system can be achieved simultaneously through the designed optimal nominal control protocol. Moreover, the proposed control strategy has more mild constraints on the communication topologies. Finally, the effectiveness of the proposed FDSMC strategy is verified by numerical simulation studies.

Adaptive optimized backstepping tracking control for full‐state constrained nonlinear strict‐feedback systems without using barrier Lyapunov function method

AbstractIn this article, the problem of adaptive optimal tracking control is studied for nonlinear strict‐feedback systems. While not directly measurable, the states of these systems are subject to both time‐varying and asymmetric constraints. Bypassing the conventional barrier Lyapunov function method, the constrained system is transformed into its unconstrained counterpart, thereby obviating the need for feasibility conditions. A specially designed reinforcement learning (RL) algorithm, featuring an observer‐critic‐actor architecture, is deployed in an adaptive optimal control scheme to ensure the stabilization of the converted unconstrained system. Within this architecture, the observer estimates the unmeasurable system states, the critic evaluates the control performance, and the actor executes the control actions. Furthermore, enhancements to the RL algorithm lead to relaxed conditions of persistent excitation, and the design methodology for the observer overcomes the restrictions imposed by the Hurwitz equation. The Lyapunov stability theorem is applied for two primary purposes: to ascertain the boundedness of all signals within the closed‐loop system, and to ensure the accuracy of the output signal in tracking the desired reference trajectory. Finally, numerical and practical simulations are provided to corroborate the effectiveness of the proposed control strategy.

Adaptive optimal tracking control of networked linear systems under two-channel stochastic dropouts

This work investigates the adaptive optimal tracking control problem for networked discrete-time linear systems by directly using the data transmitted via communication networks. It is shown that the concerned problem can be addressed by solving two sub-problems: an adaptive optimal control one and an adaptive output regulation one. Two novel online reinforcement learning based policy iteration and value iteration algorithms are developed, which constitute an integrated framework to learn the optimal feedback control gain and the solutions to regulator equations by directly using the data transmitted via communication networks. Furthermore, it is shown that the tracking error of the closed-loop control system is mean square asymptotically stable even in the case of network packet stochastic dropouts. Simulation results demonstrate the efficacy of the proposed approaches.

Asymmetric Constrained Optimal Tracking Control With Critic Learning of Nonlinear Multiplayer Zero-Sum Games.

By utilizing a neural-network-based adaptive critic mechanism, the optimal tracking control problem is investigated for nonlinear continuous-time (CT) multiplayer zero-sum games (ZSGs) with asymmetric constraints. Initially, we build an augmented system with the tracking error system and the reference system. Moreover, a novel nonquadratic function is introduced to address asymmetric constraints. Then, we derive the tracking Hamilton-Jacobi-Isaacs (HJI) equation of the constrained nonlinear multiplayer ZSG. However, it is extremely hard to get the analytical solution to the HJI equation. Hence, an adaptive critic mechanism based on neural networks is established to estimate the optimal cost function, so as to obtain the near-optimal control policy set and the near worst disturbance policy set. In the process of neural critic learning, we only utilize one critic neural network and develop a new weight updating rule. After that, by using the Lyapunov approach, the uniform ultimate boundedness stability of the tracking error in the augmented system and the weight estimation error of the critic network is verified. Finally, two simulation examples are provided to demonstrate the efficacy of the established mechanism.

Adjustable iterative Q-learning for advanced neural tracking control with stability guarantee

In this article, an accelerated Q-learning algorithm with evolving control is established to solve the optimal tracking control problem. First, an accelerated Q-learning scheme is constructed with an advanced Q-function. By utilizing the advanced Q-function, calculating of the feedforward control input can be avoided and the terminal tracking error can be eliminated. Then, by introducing the relaxation factor, the convergence rate of the iterative Q-function sequence is accelerated significantly, which is a potential way to diminish the computational burden. Furthermore, the convergence, positive definiteness, and stability conditions of the accelerated Q-learning algorithm are analyzed with some preconditions of the relaxation factor. Thus, the developed algorithm can achieve evolving control. Finally, the fantastic performance of the developed algorithm with critic network implementation is verified through two simulation examples.

Multilayer adaptive critic design with digital twin for data-driven optimal tracking control and industrial applications

In this paper, an optimal trajectory tracking control problem for general nonlinear systems is investigated. An adaptive critic control method with the digital twin (DT) theory is developed. Divergent from the existing tracking control methods, the advantages of adaptive dynamic programming (ADP) and the theory of DT are combined in this paper, and the novel multilayer artificial system structure is constructed. The action-critic structure is employed by each artificial system to obtain an approximate optimal control policy. The model network (MN) is built by using the actual input and output data sets of the controlled system, which means the dependence on the dynamics of the system is overcame. Then, the weights of the trained action network (AN) and MN are passed to the real system to realize the optimal tracking control. The feasibility of the algorithm is proved by theoretical analysis. Finally, the algorithm is applied to a simple nonlinear torsional pendulum system and an industrial wastewater treatment system (WWTS), and the effectiveness of the algorithm is verified. The algorithm effectively realizes the tracking control of nonlinear systems.