Optimal Tracking Control Problem Research Articles

Robust optimal tracking control of multiple autonomous underwater vehicles subject to uncertain disturbances

AbstractThis paper considers the problem of robust optimal tracking control of multiple autonomous underwater Vehicles (AUVs) subject to uncertain external disturbances. First, the Takagi‐Sugeno (T‐S) fuzzy based technique is utilized to convert the high‐order nonlinear multi‐AUV system into a series of linearized subsystems. Second, a novel fully distributed sliding mode control (FDSMC) strategy is proposed to attenuate the disturbances. Meanwhile, the leader‐following consensus and the nearly optimization of the energy‐cost function for the multi‐AUV system can be achieved simultaneously through the designed optimal nominal control protocol. Moreover, the proposed control strategy has more mild constraints on the communication topologies. Finally, the effectiveness of the proposed FDSMC strategy is verified by numerical simulation studies.

Adaptive optimized backstepping tracking control for full‐state constrained nonlinear strict‐feedback systems without using barrier Lyapunov function method

AbstractIn this article, the problem of adaptive optimal tracking control is studied for nonlinear strict‐feedback systems. While not directly measurable, the states of these systems are subject to both time‐varying and asymmetric constraints. Bypassing the conventional barrier Lyapunov function method, the constrained system is transformed into its unconstrained counterpart, thereby obviating the need for feasibility conditions. A specially designed reinforcement learning (RL) algorithm, featuring an observer‐critic‐actor architecture, is deployed in an adaptive optimal control scheme to ensure the stabilization of the converted unconstrained system. Within this architecture, the observer estimates the unmeasurable system states, the critic evaluates the control performance, and the actor executes the control actions. Furthermore, enhancements to the RL algorithm lead to relaxed conditions of persistent excitation, and the design methodology for the observer overcomes the restrictions imposed by the Hurwitz equation. The Lyapunov stability theorem is applied for two primary purposes: to ascertain the boundedness of all signals within the closed‐loop system, and to ensure the accuracy of the output signal in tracking the desired reference trajectory. Finally, numerical and practical simulations are provided to corroborate the effectiveness of the proposed control strategy.

Asymmetric Constrained Optimal Tracking Control With Critic Learning of Nonlinear Multiplayer Zero-Sum Games.

By utilizing a neural-network-based adaptive critic mechanism, the optimal tracking control problem is investigated for nonlinear continuous-time (CT) multiplayer zero-sum games (ZSGs) with asymmetric constraints. Initially, we build an augmented system with the tracking error system and the reference system. Moreover, a novel nonquadratic function is introduced to address asymmetric constraints. Then, we derive the tracking Hamilton-Jacobi-Isaacs (HJI) equation of the constrained nonlinear multiplayer ZSG. However, it is extremely hard to get the analytical solution to the HJI equation. Hence, an adaptive critic mechanism based on neural networks is established to estimate the optimal cost function, so as to obtain the near-optimal control policy set and the near worst disturbance policy set. In the process of neural critic learning, we only utilize one critic neural network and develop a new weight updating rule. After that, by using the Lyapunov approach, the uniform ultimate boundedness stability of the tracking error in the augmented system and the weight estimation error of the critic network is verified. Finally, two simulation examples are provided to demonstrate the efficacy of the established mechanism.

Adjustable iterative Q-learning for advanced neural tracking control with stability guarantee

In this article, an accelerated Q-learning algorithm with evolving control is established to solve the optimal tracking control problem. First, an accelerated Q-learning scheme is constructed with an advanced Q-function. By utilizing the advanced Q-function, calculating of the feedforward control input can be avoided and the terminal tracking error can be eliminated. Then, by introducing the relaxation factor, the convergence rate of the iterative Q-function sequence is accelerated significantly, which is a potential way to diminish the computational burden. Furthermore, the convergence, positive definiteness, and stability conditions of the accelerated Q-learning algorithm are analyzed with some preconditions of the relaxation factor. Thus, the developed algorithm can achieve evolving control. Finally, the fantastic performance of the developed algorithm with critic network implementation is verified through two simulation examples.

Multilayer adaptive critic design with digital twin for data-driven optimal tracking control and industrial applications

In this paper, an optimal trajectory tracking control problem for general nonlinear systems is investigated. An adaptive critic control method with the digital twin (DT) theory is developed. Divergent from the existing tracking control methods, the advantages of adaptive dynamic programming (ADP) and the theory of DT are combined in this paper, and the novel multilayer artificial system structure is constructed. The action-critic structure is employed by each artificial system to obtain an approximate optimal control policy. The model network (MN) is built by using the actual input and output data sets of the controlled system, which means the dependence on the dynamics of the system is overcame. Then, the weights of the trained action network (AN) and MN are passed to the real system to realize the optimal tracking control. The feasibility of the algorithm is proved by theoretical analysis. Finally, the algorithm is applied to a simple nonlinear torsional pendulum system and an industrial wastewater treatment system (WWTS), and the effectiveness of the algorithm is verified. The algorithm effectively realizes the tracking control of nonlinear systems.

H∞$$ {H}_{\infty } $$ optimal output tracking control for Markov jump systems: A reinforcement learning‐based approach

AbstractIn this paper, the optimal output tracking control problem for Markov jump systems is investigated, where the two cases with known or completely unknown transition probabilities are both considered. Based on game theory and performance, quadratic cost is considered, where a discount parameter is introduced into the quadratic cost in order to track unstable systems and eliminate the assumption that the noise energy is bounded. The game coupled algebraic Riccati equation and the corresponding controller are presented by dynamic programming. The stochastic stability of the tracking error system is further investigated. Moreover, iterative and reinforcement learning‐based algorithms are proposed for solving the optimal tracking controller with known or completely unknown transition probabilities, respectively. Finally, some numerical simulations on a DC motor are performed to validate the effectiveness of the proposed results.

Time-varying gain extended state observer-based adaptive optimal control for disturbed unmanned helicopter

In this paper, the robust adaptive optimal tracking control problem is addressed for the disturbed unmanned helicopter based on the time-varying gain extended state observer (TVGESO) and adaptive dynamic programming (ADP) methods. Firstly, a novel TVGESO is developed to tackle the unknown disturbance, which can overcome the drawback of initial peaking phenomenon in the traditional linear ESO method. Meanwhile, compared with the nonlinear ESO, the proposed TVGESO possesses easier and rigorous stability analysis process. Subsequently, the optimal tracking control issue for the original unmanned helicopter system is transformed into an optimization stabilization problem. By means of the ADP and neural network techniques, the feedforward controller and optimal feedback controller are skillfully designed. Compared with the conventional backstepping approach, the designed anti-disturbance optimal controller can make the unmanned helicopter accomplish the tracking task with less energy. Finally, simulation comparisons demonstrate the validity of the developed control scheme.

Model-Free Optimal Tracking Control of Nonlinear Input-Affine Discrete-Time Systems via an Iterative Deterministic Q-Learning Algorithm.

In this article, a novel model-free dynamic inversion-based Q-learning (DIQL) algorithm is proposed to solve the optimal tracking control (OTC) problem of unknown nonlinear input-affine discrete-time (DT) systems. Compared with the existing DIQL algorithm and the discount factor-based Q-learning (DFQL) algorithm, the proposed algorithm can eliminate the tracking error while ensuring that it is model-free and off-policy. First, a new deterministic Q-learning iterative scheme is presented, and based on this scheme, a model-based off-policy DIQL algorithm is designed. The advantage of this new scheme is that it can avoid the training of unusual data and improve data utilization, thereby saving computing resources. Simultaneously, the convergence and stability of the designed algorithm are analyzed, and the proof that adding probing noise into the behavior policy does not affect the convergence is presented. Then, by introducing neural networks (NNs), the model-free version of the designed algorithm is further proposed so that the OTC problem can be solved without any knowledge about the system dynamics. Finally, three simulation examples are given to demonstrate the effectiveness of the proposed algorithm.

Intelligent-Critic-Based Tracking Control of Discrete-Time Input-Affine Systems and Approximation Error Analysis With Application Verification.

In recent years, the application of function approximators, such as neural networks and polynomials, has ushered in a new stage of development in solving optimal control problems. However, considering the existence of approximation errors, the stability of the controlled system cannot be guaranteed. Therefore, in view of the prevalence of approximation errors, we investigate optimal tracking control problems for discrete-time systems. First, a novel value function is introduced into the intelligent critic framework. Second, an implicit method is utilized to demonstrate the boundedness of the iterative value functions with approximation errors. An explicit method is applied to prove the stability of the system with approximation errors. Furthermore, an evolving policy is designed to iteratively tackle the optimal tracking control problem and demonstrate the stability of the system. Finally, the effectiveness of the developed method is verified through numerical as well as practical examples.

Advanced Optimal Tracking Control With Stability Guarantee via Novel Value Learning Formulation.

In this article, to solve the optimal tracking control problem (OTCP) for discrete-time (DT) nonlinear systems, general value iteration (GVI) scheme and online value iteration (VI) algorithms with novel value function are discussed. First, the disadvantage of the traditional value function for the OTCP is presented and the novel value function is introduced. Second, we analyze the monotonicity and convergence of GVI and establish the admissibility condition of GVI to evaluate the admissibility of the current iterative control. Note that a novel approach is introduced to analyze the admissibility. Third, based on the attraction domain, improved control policies with online VI can be obtained by judging the location of the current tracking error and reference point. Finally, the stability of the online VI-based control system is guaranteed. Besides, we provide two simulation examples to show the performance of the proposed methods.

Robust Optimal Parallel Tracking Control Based on Adaptive Dynamic Programming.

This article focuses on a novel robust optimal parallel tracking control method for continuous-time (CT) nonlinear systems subject to uncertainties. First, the designed virtual controller facilitates the transformation of the original nonlinear system into an affine system with an augmented state vector, which promotes the introduction of the optimal parallel tracking control problem. Then, this article generates fresh insight into counteracting the effects of uncertainty by developing a novel parallel control system that invokes the formulated virtual control law and an auxiliary variable obtained from the relationship between the solutions of the optimal control problems for the uncertain system and the nominal one. Next, critic neural networks (NNs) approximate the Hamilton-Jacobi-Bellman (HJB) equations' solution to implement the proposed robust optimal control method via adaptive dynamic programming (ADP). Finally, simulation experiments demonstrate the proposed method's remarkable effectiveness.

Model-free intelligent critic design with error analysis for neural tracking control

The core of the optimal tracking control problem for nonlinear systems is how to ensure that the controlled system tracks the desired trajectory. The utility functions in previous studies have different properties which affect the final tracking effect of the intelligent critic algorithm. In this paper, we introduce a novel utility function and propose a Q-function based policy iteration algorithm to eliminate the final tracking error. In addition, neural networks are used as function approximator to approximate the performance index and control policy. Considering the impact of the approximation error on the tracking performance, an approximation error bound for each iteration of the novel Q-function is established. Under the given conditions, the approximation Q-function converges to the finite neighborhood of the optimal value. Moreover, it is proved that weight estimation errors of neural networks are uniformly ultimately bounded. Finally, the effectiveness of the algorithm is verified by the simulation example.

Prescribed performance event-triggered fuzzy optimal tracking control for strict-feedback nonlinear systems

The prescribed performance event-triggered optimal tracking control problem is considered for a class of uncertain strict-feedback nonlinear systems with external disturbances. First, the disturbance observers are constructed to estimate external disturbances. Then, the controller contains an adaptive fuzzy controller and an optimal compensation term, in which the unknown nonlinearities and cost function are approximated by the fuzzy logic systems. An adaptive fuzzy controller is established by the dynamic surface control (DSC) technique, which is addressed to handle “computation complexity” issue occurred in conventional backstepping approach. Based on adaptive dynamic programming (ADP) method, an optimal compensation term is developed by minimizing the cost function. Furthermore, communication load is reduced via embedding event-triggered mechanism. In addition, the tracking error can be restricted in the prescribed region with the aid of the prescribed performance control. Thus, the whole control scheme can not only ensure that the tracking error converges to a boundary but also achieve optimization, reduce the communication burden and avoid “computation complexity” issue. The boundedness of all signals in the closed-loop is proved. Finally, the simulation examples illustrate the validity of the designed control scheme.

Actor–critic learning based PID control for robotic manipulators

In this paper, we propose a reinforcement learning structure for auto-tuning PID gains by solving an optimal tracking control problem for robot manipulators. Capitalizing on the actor–critic framework implemented by neural networks, we achieve optimal tracking performance, estimating unknown system dynamics simultaneously. The critic network is used to approximate the cost function, which serves as an indicator of control performance. With feedback from the critic, the actor network learns time-varying PID gains over time to optimize control input, thereby steering the system toward optimal performance. Furthermore, we utilize Lyapunov’s direct method to demonstrate the stability of the closed-loop system. This approach provides an analytical procedure for a stable robot manipulator system to systematically adjust PID gains, bypassing the ad-hoc and painstaking process. The resultant actor–critic PID-like control exhibits stable adaptive and learning capabilities while maintaining a simple structure and inexpensive online computational demands. Numerical simulations underscore the effectiveness and advantages of the proposed actor–critic neural network PID control.

Optimal tracking control of an Autonomous Underwater Vehicle: A PMP approach

This paper addresses the optimal tracking control problem of an Autonomous Underwater Vehicle (AUV) using Pontryagin’s Minimum Principle (PMP). The formulation uses a choice of Hamiltonian function using the desired control objective with AUV dynamics acting as dynamic constraints. We first develop PMP based on AUV kinematics and then extend it to the dynamics to arrive at optimal thrusts and moments. Necessary conditions for optimality are derived for both the models using PMP, which results in optimal trajectories that simultaneously minimize the tracking error as well as the control cost, thereby arriving at energy optimality. It was observed that the adjoint variables (costates) are indeed the momenta in the inertial and body-fixed frames. At the kinematic level, this forms a stable solution. The developed methodology is applied to both 2D and 3D AUV model with disturbances due to inputs and ocean currents. Numerical simulations are carried out with the derived control laws for a given trajectory tracking target. Quantitative evaluation of the performance and comparison of the controller is done using Mean Square Error(MSE) and Total Variation(TV) measures. The proposed control laws are found to achieve the desired control objectives.

Reinforcement learning-based optimal adaptive fuzzy control for nonlinear multi-agent systems with prescribed performance

In this paper, the problem of optimal adaptive consensus tracking control for nonlinear multi-agent systems with prescribed performance is investigated. To address the issue of satisfying the initial value conditions in existing results, an improved performance function is employed as the prescribed performance boundary, effectively resolving this problem. Then, by employing the error transformation function, the constrained system is converted into an unconstrained one. Furthermore, fuzzy logic systems are employed to identify unknown system parts. By applying the dynamic surface technique, the problem of "differential explosion", which often occurs in backstepping, is solved. Moreover, a distributed optimal adaptive fuzzy control protocol based on the reinforcement learning actor-critic algorithm is proposed. Under the proposed control scheme, it is proved that all the signals within the closed-loop system are bounded, and the consensus tracking errors have remained within the predefined bounds. Finally, the numerical simulation results demonstrate the effectiveness of the proposed scheme.

Dynamic optimal control of flow front position in injection molding process: A control parameterization-based method

High dynamic and precise control of process variables within permissible limits is the key to high product quality in the injection molding industry. Among these variables, the injection flow rate of molten polymer is particularly significant. In this paper, an effective optimal open-loop and state feedback controller design method is designed to realize the optimal tracking control of the melt polymer flow front position flow of melt polymer fluid during injection molding. To this end, the dynamic model of the filling process is firstly established, taking into account the nonlinearity of hydraulic control system and the effect of molten polymer flow on the dynamics of the injection ram, and then the dynamic optimal tracking control problem of flow front position of molten polymer in injection molding machine is proposed. Then, an open-loop controller and a multi-level feedback control law are well designed, wherein the control signal and the control feedback kernels are parameterized by control parameterization. The controller design problems are transformed into sequential optimal parameter decision problems. The explicit expressions of the gradient information of the objective function and constraints on the parameters of the decision variables are solved by using the state sensitivity equation analysis method, and the gradient information is combined with the nonlinear programming algorithm to solve the optimization problems efficiently. Finally, the viability and effectiveness of the proposed design method are verified by numerical simulations.

Off-policy two-dimensional reinforcement learning for optimal tracking control of batch processes with network-induced dropout and disturbances

In this paper, a new off-policy two-dimensional (2D) reinforcement learning approach is proposed to deal with the optimal tracking control (OTC) issue of batch processes with network-induced dropout and disturbances. A dropout 2D augmented Smith predictor is first devised to estimate the present extended state utilizing past data of time and batch orientations. The dropout 2D value function and Q-function are further defined, and their relation is analyzed to meet the optimal performance. On this basis, the dropout 2D Bellman equation is derived according to the principle of the Q-function. For the sake of addressing the dropout 2D OTC problem of batch processes, two algorithms, i.e., the off-line 2D policy iteration algorithm and the off-policy 2D Q-learning algorithm, are presented. The latter method is developed by applying only the input and the estimated state, not the underlying information of the system. Meanwhile, the analysis with regard to the unbiasedness of solutions and convergence is separately given. The effectiveness of the provided methodologies is eventually validated through the application of a simulated case during the filling process.

Hierarchical Sliding-Mode Surface-Based Adaptive Critic Tracking Control for Nonlinear Multiplayer Zero-Sum Games via Generalized Fuzzy Hyperbolic Models

This paper investigates the hierarchical sliding-mode surface (HSMS)-based adaptive critic tracking control problem for nonlinear multiplayer zero-sum games (ZSGs). First, a generalized fuzzy hyperbolic models-based identifier is employed to approximate the unknown nonlinear functions. According to the derived data-driven model, a static control policy is proposed to transform the considered optimal tracking control problem into an optimal regulation control problem of an induced error system. Subsequently, based on sliding mode control technology and the concept of hierarchical design, a novel optimal feedback control policy is developed to regulate the tracking error by minimizing a performance index function related to the HSMS. The solution to the Hamilton-Jacobi-Isaacs equation of nonlinear multiplayer ZSGs is obtained via the HSMS-based critic neural network. Furthermore, the historical stored data is utilized to conquer the difficulty associated with the persistence of excitation conditions. Based on Lyapunov stability theory, it is strictly proven that the tracking error and the critic neural network weight are uniformly ultimately bounded. Finally, two examples are presented to demonstrate the effectiveness of the proposed control scheme.

Event-based performance guaranteed tracking control for constrained nonlinear system via adaptive dynamic programming method

An optimal tracking control problem for a class of nonlinear systems with guaranteed performance and asymmetric input constraints is discussed in this paper. The control policy is implemented by adaptive dynamic programming (ADP) algorithm under two event-based triggering mechanisms. It is often challenging to design an optimal control law due to the system deviation caused by asymmetric input constraints. First, a prescribed performance control technique is employed to guarantee the tracking errors within predetermined boundaries. Subsequently, considering the asymmetric input constraints, a discounted non-quadratic cost function is introduced. Moreover, in order to reduce controller updates, an event-triggered control law is developed for ADP algorithm. After that, to further simplify the complexity of controller design, this work is extended to a self-triggered case for relaxing the need for continuous signal monitoring by hardware devices. By employing the Lyapunov method, the uniform ultimate boundedness of all signals is proved to be guaranteed. Finally, a simulation example on a mass–spring–damper system subject to asymmetric input constraints is provided to validate the effectiveness of the proposed control scheme.