Optimal Tracking Control Problem Research Articles

Learning-based near-optimal tracking control for industrial processes with slow and fast modes

This paper devotes to solving the optimal tracking control (OTC) problem of singular perturbation systems in industrial processes under the framework of reinforcement learning (RL) technology. The encountered challenges include the different time scales in system operations and an unknown slow process. The immeasurability of slow process states especially increases the difficulty of finding the optimal tracking controller. To overcome these challenges, a novel off-policy ridge RL method is developed after decomposing the singular perturbed systems using the singular perturbation (SP) theory and replacing unmeasured states using important mathematical manipulations. Theoretical analysis of approximate equivalence of the sum of solutions of subproblems to the solution of the OTC problem is presented. Finally, a mixed separation thickening process (MSTP) and a numerical example are used to verify the effectiveness.

Input-output data based tracking control under DoS attacks

ABSTRACT This paper investigates the secure optimal tracking control problem for cyber-physical systems, in which the controller-actuator channel is jammed by denial-of-service attacks. Without leveraging model knowledge, learning-based tracking control algorithms are proposed by utilizing the measured input-output data. The impact of malicious attacks on tracking performance is discussed. More specifically, an augmented system consisted of the system model and the reference model is derived. Then the states of the augmented system are rewritten by utilizing the input, output, and reference trajectory, following which both the Bellman equation and algebraic Riccati equation are given. The conditions of existence and uniqueness of the solution to the algebraic Riccati equation are derived. Furthermore, both policy iteration and value iteration learning-based tracking control algorithms are provided for cyber-physical systems against denial-of-service attacks, and the convergence of the algorithms is also proved. Finally, a DC motor system and a numerical example are given to illustrate the effectiveness of the proposed control algorithms.

OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH INPUT CONSTRAINT BASED REINFORCEMENT LEARNING

This paper introduces an optimal tracking controller for robot manipulators with saturation torques. The robot model is presented as a strict-feedback nonlinear system. Firstly, the position tracking control problem is transformed into the optimal tracking control problem. Subsequently, the saturated optimal control law is designed. The optimal control law is determined through the solution of the Hamilton-Jacobi-Bellman (HJB) equation. We use a reinforcement learning algorithm with only one neural network (NN) to approximate the solution of the equation HJB. The technique of experience replay is used to relax a persistent citation condition. By Lyapunov analysis, the tracking and the approximation errors are uniformly ultimately bounded (UUB). Finally, the simulation on a robot manipulator with saturation torques is performed to verify the efficiency of the proposed controller.

Optimal Robust Tracking Control of Injection Velocity in an Injection Molding Machine

Injection molding is a critical component of modern industrial operations, and achieving fast and stable control of injection molding machines (IMMs) is essential for producing high-quality plastic products. This paper focuses on solving an optimal tracking control problem of the injection velocity that arises in a typical nonlinear IMM. To this end, an efficient optimal robust controller is proposed and designed. The nonlinear injection velocity servo system is first approximately linearized at iteration points using the first-order Taylor expansion approach. Then, at each time node in the optimization process, the relevant algebraic Riccati equation is introduced, and the solution is used to construct an optimal robust feedback controller. Furthermore, a rigorous Lyapunov theorem analysis is employed to demonstrate the global stability properties of the proposed feedback controller. The results from numerical simulations show that the proposed optimal robust control strategy can successfully and rapidly achieve the best tracking of the intended injection velocity trajectory within a given time.

Further Results on Optimal Tracking Control for Nonlinear Systems With Nonzero Equilibrium via Adaptive Dynamic Programming.

This article develops a novel cost function (performance index function) to overcome the obstacles in solving the optimal tracking control problem for a class of nonlinear systems with known system dynamics via adaptive dynamic programming (ADP) technique. For the traditional optimal control problems, the assumption that the controlled system has zero equilibrium is generally required to guarantee the finiteness of an infinite horizon cost function and a unique solution. In order to solve the optimal tracking control problem of nonlinear systems with nonzero equilibrium, a specific cost function related to tracking errors and their derivatives is designed in this article, in which the aforementioned assumption and related obstacles are removed and the controller design process is simplified. Finally, comparative simulations are conducted on an inverted pendulum system to illustrate the effectiveness and advantages of the proposed optimal tracking control strategy.

Event-triggered near-optimal tracking control based on adaptive dynamic programming for discrete-time systems

Frequent state monitoring and controller updates can enhance the precision of tracking control, while simultaneously overburdening the communication network transmission. In this paper, For the purpose of saving communication costs, we propose event-triggered control algorithms for the optimal tracking control problem. First, we reconstruct the discrete-time nonlinear system into a converted system. Then, the adaptive dynamic programming algorithm is employed to find the optimal controller off-line, and the event-triggered scheme is used to reduce the communication costs online. Novel triggering conditions with fewer assumptions are designed to implement the event-triggered scheme. Different from existing works, the event-triggered scheme can be introduced not only into the converted system but also into the actual system, which is more practical because the actual controller is what one can only access in practice. In addition, with the developed algorithms, the tracking error can be proved to be stable at the origin, in other words, the actual system can be guaranteed to track the desired trajectory. Algorithms developed in this paper are implemented by three neural networks, the model network, the action network and the critic network. Finally, examples are presented to verify the effectiveness and rationality of the algorithms.

OPTIMAL TRACKING CONTROL FOR ROBOT MANIPULATORS WITH ASYMMETRIC SATURATION TORQUES BASED ON REINFORCEMENT LEARNING

This paper introduces an optimal tracking controller for robot manipulators with asymmetrically saturated torques and partially - unknown dynamics based on a reinforcement learning method using a neural network. Firstly, the feedforward control inputs are designed based on the backstepping technique to convert the tracking control problem into the optimal tracking control problem. Secondly, a cost function of the system with asymmetrically saturated input is defined, and the constrained Hamilton-Jacobi-Bellman equation is built, which is solved by the online reinforcement learning algorithm using only a single neural network. Then, the asymmetric saturation optimal control rule is determined. Additionally, the concurrent learning technique is used to relax the demand for the persistence of excitation conditions. The built algorithm ensures that the closed-loop system is asymptotically stable, the approximation error is uniformly ultimately bounded (UUB), and the cost function converges to the near-optimal value. Finally, the effectiveness of the proposed algorithm is shown through comparative simulations.

Data‐based learning control for optimization of nonlinear systems

Data‐based learning control for optimization of nonlinear systems

Based on Q-Learning Optimal Tracking Control Schemes for Linear Itô Stochastic Systems With Markovian Jumps

In this brief, a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm is designed to solve the optimal tracking control problem (OTCP) for stochastic system with Markovian jump. The problem is formulated by stochastic augmented system composed of the dynamic system and the reference trajectory system and then using ideas from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning to provide a solution. Namely, a critic neural network (NN) is used to estimate the optimal cost function while an actor NN is used to approximate the optimal controller. Moreover, both the requirements of transition probability and system matrix are avoided via the designed <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$Q$ </tex-math></inline-formula> -learning algorithm. Finally, we apply the designed algorithm in a single-area power system to track continuous sinusoidal waveforms, and the simulation result is provided to show the effectiveness and applicability of the algorithm.

Global optimal tracking control for multi-input-multi-output systems with adjoining cell mapping and a subdivision strategy

In this paper, global optimal tracking control for multi-input-multi-output systems using adjoining cell mapping technology is studied. An adaptive criterion is first proposed to improve the existing adjoining cell mapping method for optimal control. The criterion can judge the availability of mappings from different pre-image cells to their adjoining image cells adaptively, and help to build mapping database with the minimal loss of useful mapping information. An accessible and efficient backward search strategy to obtain global optimal control solution is also advanced. Then, a subdivision strategy for adjoining cell mapping to solve optimal tracking control problems for multi-input-multi-output systems is introduced. Finally, three examples illustrate the performance of the adaptive criterion and subdivision strategy. The adjoining cell mapping method for optimal control is greatly improved and the proposed subdivision strategy offers an efficient numerical method with general applicability to solve global optimal tracking control problems for nonlinear multi-input-multi-output systems.

Online adaptive optimal tracking control for model-free nonlinear systems via a dynamic neural network

This paper presents an online adaptive approximate solution for the optimal tracking control problem of model-free nonlinear systems. Firstly, a dynamic neural network identifier with properly designed weights updating laws is developed to identify the unknown dynamics. Then an adaptive optimal tracking control policy consisting of two terms is proposed, i.e. a steady-state control term is established to ensure the desired tracking performance at the steady state, and an optimal control term is proposed to ensure the optimal tracking error dynamics optimally. The composite Lyapunov method is used to analyse the stability of the closed-loop system. Two simulation examples are presented to demonstrate the effectiveness of the proposed method.

Information Fusion Tracking Control for a Class of Discrete-Time Systems With Delays

In this article, the optimal tracking control problem for a class of discrete time-delay systems with disturbance is studied using the information fusion control methodology. First, the disturbance estimation is regarded as a minimization problem, and an information fusion disturbance observer (IFDO) is designed. Second, combining the soft constraint requirement of the output tracking error contained in the performance index function, a fusion filter is constructed to estimate the system co-state and its information weight under system state and input delays. Finally, the obtained co-state and its information weight, the soft constraint requirement of the control input contained in the performance function, as well as the system delay dynamics, are all transformed into the information equations about the control input, and then the information fusion delay controller (IFDC) is directly obtained by fusing these information equations. As a result, the negative effects caused by disturbance and time delays can be effectively compensated at the same time. The convergence of the tracking errors of the closed-loop system is guaranteed by the Lyapunov stability theory. Numerical simulation results have been provided to demonstrate the effectiveness and robustness of the proposed control algorithm.

Learning-based robust optimal tracking controller design for unmanned underwater vehicles with full-state and input constraints

In this article, the optimal tracking control problem for unmanned underwater vehicles (UUVs) with full-state and input constraints under the presence of external disturbances and internal dynamic uncertainties is addressed. To achieve preassigned state constraints on UUVs, the traditional UUVs model is transformed into an unconstrained one by using two different nonlinear mappings (NMs). Then the robust tracking control problem of traditional UUVs model under position/Euler angles and velocity constraints is transformed to an optimal control problem of the transformed system without any constraints. A learning-based optimal control method is designed to solve the optimal control problem of the transformed system by employing the optimized backstepping (OB) paradigm and reinforcement-learning (RL) technique, achieving uniformly ultimately boundedness (UUB) subject to optimal cost. To deal with lumped disturbances for the velocity control loop, a neural-network (NN) identifier is employed and incorporated into actor–critic architecture, attaining robust tracking performance. Due to the adopted nonquadratic cost function with respect to the control input, the optimal control solution is established in the form of a hyperbolic tangent function to handle the input constraints. Compared with traditional PID method and MPC approach, the proposed controller can improve tracking performance of UUV by 32.04% and 79.64%, respectively.

Neuro‐adaptive finite‐time optimal tracking control for permanent magnet synchronous motors with full‐state constraints

SummaryIn this article, the optimal tracking control problem is investigated for permanent magnet synchronous motors (PMSMs) with full‐state constraints. By constructing multiple barrier‐type performance index functions, a neuro‐adaptive finite‐time optimized control scheme is presented under identifier‐actor‐critic architecture, where the virtual control laws and the actual laws are designed to optimize corresponding subsystems. It is proven that all signals of the closed‐loop system are uniformly ultimately bounded under the proposed control strategy, and the position tracking error converges to a small neighborhood of the origin in finite time. Besides, the system states are constrained to the effective operation range all the time. Finally, the simulation results and comparisons are carried out to further demonstrate the effectiveness of the proposed optimal control approach.

Perceptual Interaction-Based Path Tracking Control of Autonomous Vehicles Under DoS Attacks: A Reinforcement Learning Approach

This paper considers the distributed optimal path tracking control problem of autonomous ground vehicles (AGVs) with matched uncertainties under denial-of-service (DoS) attacks. First, to solve the uncertain system parameters, an inverse reinforcement learning (RL) technique is PROPOSED to seek the optimal controller instead of directly solving the robust control problem, which can ensure that the designed controller is globally optimal with respect to the proposal performance function. An off-policy RL algorithm is further proposed to learn the optimal control solution online without requiring any knowledge of the vehicle dynamics. Then, to lower the communication load among vehicles, a perception-based optimal resilient controller and its corresponding perception-based static deadband condition are developed, respectively. Moreover, the perception-based dynamic deadband control mechanism is further exploited to consider the secure path tracking control problem. The proposed algorithms verify that the RL-based optimal path tracking control can be solved, and the occurrence of the Zeno phenomenon can be avoided. Finally, a numerical simulation is demonstrated to verify the effectiveness of the proposed approach.

Optimal Drug Dosage Control Strategy of Immune Systems Using Reinforcement Learning

In this article, a reinforcement learning-based drug dosage control strategy is developed for immune systems with input constraints and dynamic uncertainties to sustain the number of tumor and immune cells in an acceptable level. First of all, the state of the immune system and the desired number of tumor and immune cells are constructed into an augmented state to derive an augmented immune system. By designing a discounted non-quadratic performance index function, the robust tracking control problem of immune systems with uncertainties is transformed into an optimal tracking control problem of nominal immune systems and the drug dosage can be limited within the specified range. Hereafter, a reinforcement learning algorithm and a critic-only structure are adopted to acquire the approximate optimal drug dosage control strategy. Furthermore, theoretical proof reveals that the proposed reinforcement learning-based drug dosage control strategy ensures the number of tumor and immune cells reaches the preset level under limited drug dosages and model uncertainties. Finally, simulation study verifies the availability of the developed drug dosage control strategy in different growth models of tumor cell.

Error Dynamics Based Dual Heuristic Dynamic Programming for Self-Learning Flight Control

A data-driven nonlinear control approach, called error dynamics-based dual heuristic dynamic programming (ED-DHP), is proposed for air vehicle attitude control. To solve the optimal tracking control problem, the augmented system is defined by the derived error dynamics and reference trajectory so that the actor neural network can learn the feedforward and feedback control terms at the same time. During the online self-learning process, the actor neural network learns the control policy by minimizing the augmented system’s value function. The input dynamics identified by the recursive least square (RLS) and output of the critic neural network are used to update the actor neural network. In addition, the total uncertainty term of the error dynamics is also identified by RLS, which can compensate for the uncertainty caused by inaccurate modeling, parameter perturbation, and so on. The outputs of ED-DHP include the rough trim surface, feedforward and feedback terms from the actor neural network, and the compensation. Based on this control scheme, the complete knowledge of system dynamics and the reference trajectory dynamics are not needed, and offline learning is unnecessary. To verify the self-learning ability of ED-DHP, two numerical experiments are carried out based on the established morphing air vehicle model. One is sinusoidal signal tracking at a fixed operating point, and the other is guidance command tracking with a morphing process at variable operating points. The simulation results demonstrate the good performance of ED-DHP for online self-learning attitude control and validate the robustness of the proposed scheme

Adaptive constrained output feedback optimal consensus tracking for uncertain nonlinear multi-agent systems and its application

This paper addresses the output feedback optimal consensus tracking control problem for a class of nonlinear multi-agent systems (MASs) with unknown internal dynamics and input saturation. Firstly, an adaptive observer is designed to reconstruct the unmeasurable states using neural networks (NNs) based on the augmented dynamic model. Subsequently, a novel distributed feedforward controller is proposed using the backstepping method, where an auxiliary state is introduced to deal with the input saturation problem. Unlike the traditional recursive design technique, the procedures we take can reduce the consensus tracking control problem into the optimal regulation issue, which is then solved by the adaptive dynamic programming (ADP) method. Therefore, the designed consensus control protocol consists of a distributed feedforward controller and a distributed optimal feedback controller. Moreover, the stability of the MASs is guaranteed by the Lyapunov theory. Numerical simulation on multi-missile guidance problem demonstrates the effectiveness of the proposed control scheme.

Dynamic event-triggered optimal tracking control for constrained nonlinear stochastic systems

This paper investigates the optimal tracking control problem (OTCP) for nonlinear stochastic systems with input constraints under the dynamic event-triggered mechanism (DETM). Firstly, the OTCP is converted into the stabilizing optimization control problem by constructing a novel stochastic augmented system. The discounted performance index with nonquadratic utility function is formulated such that the input constraint can be encoded into the optimization problem. Then the adaptive dynamic programming (ADP) method of the critic-only architecture is employed to approximate the solutions of the OTCP. Unlike the conventional ADP methods based on time-driven mechanism or static event-triggered mechanism (SETM), the proposed adaptive control scheme integrates the DETM to further lighten the computing and communication loads. Furthermore, the uniform ultimately boundedness (UUB) of the critic weights and the tracking error are analysed with the Lyapunov theory. Finally, the simulation results are provided to validate the effectiveness of the proposed approach.

Optimal Consensus Control Design for Multiagent Systems With Multiple Time Delay Using Adaptive Dynamic Programming.

In this article, a novel data-based adaptive dynamic programming (ADP) method is presented to solve the optimal consensus tracking control problem for discrete-time (DT) multiagent systems (MASs) with multiple time delays. Necessary and sufficient conditions of the corresponding equivalent time-delay system are provided on the basis of the causal transformations. Benefitting from the construction of tracking error dynamics, the optimal tracking problem can be transformed into settling the Nash-equilibrium in the graphical game, which can be completed by solving the coupled Hamilton-Jacobi (HJ) equations. An error estimator is introduced to construct the tracking error of the MASs only using the input and output (I/O) data. Therefore, the designed data-based ADP algorithm can minimize the cost functions and ensure the consensus of MASs without the knowledge of system dynamics. Finally, a numerical example is given to demonstrate the effectiveness of the proposed method.