Control Of Continuous-time Nonlinear Systems Research Articles

Q-Learning-Based Model Predictive Control for Nonlinear Continuous-Time Systems

In this paper, a Q-learning-based model predictive control using the Lyapunov technique (Q-LMPC) is proposed for the control of a class of continuous nonlinear systems with complicated dynamics, wh...

A Semi-Algebraic Optimization Approach to Data-Driven Control of Continuous-Time Nonlinear Systems

This letter considers the problem of designing state feedback data-driven controllers for nonlinear continuous-time systems. Specifically, we consider a scenario where the unknown dynamics can be parametrized in terms of known basis functions and the available measurements are corrupted by unknown-but-bounded noise. The goal is to use this noisy experimental data to directly design a rational state-feedback control law guaranteed to stabilize all plants compatible with the available information. The main result of this letter shows that, by using Rantzer's Dual Lyapunov approach, combined with elements from convex analysis, the problem can be recast as an optimization over positive polynomials, which can be relaxed to a semi-definite program through the use of Sum-of-Squares and semi-algebraic optimization arguments. Three academic examples are considered to illustrate the effectiveness of the proposed method.

Optimal Feedback Control of Pedestrian Flow in Heterogeneous Corridors

Maintaining the orderliness and efficiency of pedestrian flow through an architectural area is critical for the evacuation process. Especially, clogs and jams are easily triggered in width-changing areas. In this article, we consider pedestrian movement in heterogeneous corridors and design an optimal feedback control to regulate pedestrian flow. Flow characteristics are first studied based on microscopic social-force simulations. A Gaussian process describes the relationship between flow variables with the observation data. The macroscopic model for flow in heterogeneous corridors is developed. To avoid jams, discharges among these corridors are balanced with the narrowest corridor as the primary concern. At the equilibrium, a continuous-time nonlinear control system is formulated, and the adaptive dynamic programming learns the optimal feedback controller. Policy iteration (PI) and neural networks are combined together, and the convergence of neural-network-based PI is demonstrated by analyzing its equivalence to the Gauss–Newton method. Batch normalization is introduced to stabilize the learning process. Simulated experiments demonstrate that the control design can effectively regulate pedestrian flow for both macroscopic and microscopic models. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The development of video-processing techniques provides a powerful tool to detect human behavior in real time. In crowd events, the pedestrian movement must be regulated; otherwise, it is easy to fall into the faster-is-slower effect. It is especially important for evacuation routes with different widths. In this article, the optimal feedback control is studied to regulate pedestrian flow in heterogeneous corridors. It takes flow densities as state and produces commands that are composed of entrance influx and free-flow velocities. These commands can be executed with the support of speakers, displays, or the recently developed interactive robots. To avoid congestion, discharges of different corridors are balanced, and the system is optimally stabilized at equilibrium. Based on our work, engineers are able to design pedestrian flow control and achieve optimal evacuation in arbitrary heterogeneous corridors.

Dual Heuristic Programming for Optimal Control of Continuous-Time Nonlinear Systems Using Single Echo State Network.

This article presents an improved online adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of continuous-time nonlinear systems with infinite horizon cost. The Hamilton-Jacobi-Bellman (HJB) equation is iteratively approximated by a novel critic-only structure which is constructed using the single echo state network (ESN). Inspired by the dual heuristic programming (DHP) technique, ESN is designed to approximate the costate function, then to derive the optimal controller. As the ESN is characterized by the echo state property (ESP), it is proved that the ESN can successfully approximate the solution to the HJB equation. Besides, to eliminate the requirement for the initial admissible control, a new weight tuning law is designed by adding an alternative condition. The stability of the closed-loop optimal control system and the convergence of the out weights of the ESN are guaranteed by using the Lyapunov theorem in the sense of uniformly ultimately bounded (UUB). Two simulation examples, including linear system and nonlinear system, are given to illustrate the availability and effectiveness of the proposed approach by comparing it with the polynomial neural-network scheme.

Event-triggered receding horizon control via actor-critic design

In this paper, we propose a novel event-triggered near-optimal control for nonlinear continuous-time systems. The receding horizon principle is utilized to improve the system robustness and obtain better dynamic control performance. In the proposed structure, we first decompose the infinite horizon optimal control into a series of finite horizon optimal problems. Then a learning strategy is adopted, in which an actor network is employed to approximate the cost function and an critic network is used to learn the optimal control law in each finite horizon. Furthermore, in order to reduce the computational cost and transmission cost, an event-triggered strategy is applied. We design an adaptive trigger condition, so that the signal transmissions and controller updates are conducted in an aperiodic way. Detailed stability analysis shows that the nonlinear system with the developed event-triggered optimal control policy is asymptotically stable. Simulation results on a single-link robot arm with different noise types have demonstrated the effectiveness of the proposed method.

An identifier-actor-optimizer policy learning architecture for optimal control of continuous-time nonlinear systems

An intelligent solution method is proposed to achieve real-time optimal control for continuous-time nonlinear systems using a novel identifier-actor-optimizer (IAO) policy learning architecture. In this IAO-based policy learning approach, a dynamical identifier is developed to approximate the unknown part of system dynamics using deep neural networks (DNNs). Then, an indirect-method-based optimizer is proposed to generate high-quality optimal actions for system control considering both the constraints and performance index. Furthermore, a DNN-based actor is developed to approximate the obtained optimal actions and return good initial guesses to the optimizer. In this way, the traditional optimal control methods and state-of-the-art DNN techniques are combined in the IAO-based optimal policy learning method. Compared to the reinforcement learning algorithms with actor-critic architectures that suffer hard reward design and low computational efficiency, the IAO-based optimal policy learning algorithm enjoys fewer user-defined parameters, higher learning speeds, and steadier convergence properties in solving complex continuous-time optimal control problems (OCPs). Simulation results of three space flight control missions are given to substantiate the effectiveness of this IAO-based policy learning strategy and to illustrate the performance of the developed DNN-based optimal control method for continuous-time OCPs.

Data-driven dynamic multi-objective optimal control: A Hamiltonian-inequality driven satisficing reinforcement learning approach

Abstract This paper presents an iterative data-driven algorithm for solving dynamic multi-objective (MO) optimal control problems arising in control of nonlinear continuous-time systems with multiple objectives. It is first shown that the Hamiltonian function corresponding to each objective can serve as a comparison function to compare the performance of admissible policies. Relaxed Hamilton-Jacobi-bellman (HJB) equations in terms of HJB inequalities are then solved in a dynamic constrained MO framework to find Pareto-optimal solutions. Relation to satisficing (good enough) decision-making framework is shown. A Sum-of-Square (SOS)-based iterative algorithm is developed to solve the formulated MO optimization with HJB inequalities. To obviate the requirement of complete knowledge of the system dynamics, a data-driven satisficing reinforcement learning approach is proposed to solve the SOS optimization problem in real-time using only the information of the system trajectories measured during a time interval without having full knowledge of the system dynamics. Finally, a simulation example is provided to show the effectiveness of the proposed algorithm.

A Digital Receding-Horizon Learning Controller for Nonlinear Continuous-time Systems

Abstract The integration of reinforcement learning (RL) and model predictive control (MPC) is promising for solving nonlinear optimization problems in an efficient manner. In this paper, a digital receding horizon learning controller is proposed for continuous-time nonlinear systems with control constraints. The main idea is to develop a digital design for RL with actor-critic design (ACD) in the framework of MPC, to realize near-optimal control of continuous-time nonlinear systems. Different from classic RL for continuous-time systems, the actor adopted is learned in discrete-time steps, while the critic evaluates the learned control policy continuously in the time domain. Moreover, we use soft barrier functions to deal with control constraints and the robustness of the actor-critic network is proven. A simulation example is considered to show the effectiveness of the proposed approach.

Approximate Dynamic Programming with Gaussian Processes for Optimal Control of Continuous-Time Nonlinear Systems

In this paper, a new algorithm for realization of approximate dynamic programming (ADP) with Gaussian processes (GPs) for continuous-time (CT) nonlinear input-affine systems is proposed to infinite horizon optimal control problems. The convergence for the ADP algorithm is proven based on the assumption of an exact approximation, where both the cost function and the control input converge to their optimal values, that is, the solution to the Hamilton-Jacobi-Bellman (HJB) equation. The approximation errors, however, are unavoidable in almost every case of applications. In order to tackle the problem, the proposed algorithm is derived with the proof of convergence, where the cost function and the control input, which are both approximated, converge to those of the ADP as the number of data points for GPs approaches infinity. A numerical simulation demonstrates the effectiveness of the proposed algorithm.

Structured Online Learning-based Control of Continuous-time Nonlinear Systems

Abstract Model-based reinforcement learning techniques accelerate the learning task by employing a transition model to make predictions. In this paper, a model-based learning approach is presented that iteratively computes the optimal value function based on the most recent update of the model. Assuming a structured continuous-time model of the system in terms of a set of bases, we formulate an infinite horizon optimal control problem addressing a given control objective. The structure of the system along with a value function parameterized in the quadratic form provides a flexibility in analytically calculating an update rule for the parameters. Hence, a matrix differential equation of the parameters is obtained, where the solution is used to characterize the optimal feedback control in terms of the bases, at any time step. Moreover, the quadratic form of the value function suggests a compact way of updating the parameters that considerably decreases the computational complexity. Considering the state-dependency of the differential equation, we exploit the obtained framework as an online learning-based algorithm. In the numerical results, the presented algorithm is implemented on four nonlinear benchmark examples, where the regulation problem is successfully solved while an identified model of the system is obtained with a bounded prediction error.

Symbolic control design of nonlinear systems with outputs

Formal methods have been recently used as the basis of a systematic framework to address control design of continuous or hybrid systems with specifications expressed in a logic form. However, results available in the literature assume full information of the state, or of its quantization. This information may not be available in relevant applications. In this paper, we consider the more realistic scenario where the controller cannot access the state of the plant but can only access a quantized measurement of its outputs where nonidealities of the sensing devices can be modeled. We focus on a control problem where the plant is described by a possibly unstable continuous-time nonlinear control system, the controller is dynamic, digital and quantized, and takes as input a (quantized) measurement of the output of the plant, and the specification is expressed in terms of regular languages. The solution to the control problem is based on formal methods. A finite-state system, also called symbolic model approximating the plant is first derived and then used to find the solution to the control problem. An illustrative example is provided and the symbolic control of a car-like robot is presented.

Event-triggered H∞ optimal control for continuous-time nonlinear systems using neurodynamic programming

In this paper, we propose a novel event-triggered neurodynamic programming (NDP) method to tackle the continuous-time nonlinear systems H∞ optimal control problem. First, the H∞ optimal control problem is converted to a zero-sum (ZS) two-player differential game problem. Then, we derive a triggering condition for the ZS two-player differential game problem with an event-triggered control input. The event-triggered controller is updated only at the triggering instants where the triggering condition is violated. So the event-triggered control manner can significantly reduce the update frequency of the controller. Therefore, event-triggered control has been an effective technique in resolving problems with limited communication and computation resources and is considered as an alternative to the traditional time-triggered control. Furthermore, in order to implement event-triggered control manner, we employ an actor-critic-disturbance neural networks (NNs) framework to approximate the optimal control input policy, the optimal value function and the disturbance policy respectively. The Lyapunov function method is applied to study the stability of the event-triggered closed-loop system and the uniformly ultimately boundedness (UUB) of the NN’s weights vector. Furthermore, the Zeno behavior exclusion is proved. Finally, two simulation examples are given to verify the effectiveness of the proposed algorithm.

Adaptive optimal tracking control for nonlinear continuous-time systems with time delay using value iteration algorithm

In this paper, an integral reinforcement learning-based value iteration algorithm is developed for solving the infinite horizon optimal tracking control problem of nonlinear continuous-time systems with time delay. The main idea is using the value iteration technique to obtain the iterative control law, which optimizes the iterative performance index function. In contrast to the existing value iteration algorithms, the proposed IRL-based value iteration algorithm takes the time delay into account. Second, the convergence analysis of the proposed algorithm is given for the nonlinear continuous-time systems with time delay. Moreover, the critic neural network is utilized to approximate the performance index function and compute the optimal control law for facilitating the implementation of the iterative algorithm. Finally, the simulation results are presented to illustrate the effectiveness of the developed method.

Event-triggered single-network ADP method for constrained optimal tracking control of continuous-time non-linear systems

This paper investigates the optimal tracking control problem (OTCP) for continuous-time non-linear systems with input constraints. A novel event-triggered single-network adaptive dynamic programming (ADP) method is proposed to obtain the solution of constrained OTCP. By constructing an augmented system and introducing a novel discounted non-quadratic cost function, an event-triggered constrained tracking Hamilton–Jacobi–Bellman equation is formulated. Then, only a critic neural network (NN) is employed to learn the optimal value function and further obtain the optimal tracking controller, which enables the architecture of ADP implementation to be simpler. And a novel NN weights updating law is constructed, by which the restriction of initial admissible control is removed. Based on the Lyapunov theory, the convergence of critic NN weights and the stability of closed-loop system are demonstrated. The derived optimal tracking controller is updated only at the event-triggered instants decided by the designed event-triggered condition. Therefore, the communication burden can be reduced effectively. Finally, two simulation examples are given to verify the effectiveness of proposed method.

Dynamic output-feedback control for nonlinear continuous-time systems based on parametric uncertain subsystem and T-S fuzzy model

Dynamic output-feedback control for nonlinear continuous-time systems based on parametric uncertain subsystem and T-S fuzzy model

TS-based sampled-data model predictive controller for continuous-time nonlinear systems

This paper proposes a new fuzzy model predictive control approach for continuous-time nonlinear systems in terms of linear matrix inequalities (LMIs). The proposed approach is based on the Takagi–Sugeno fuzzy modeling, a quadratic Lyapunov function, and a sampled-data parallel distributed compensation controller with constant sampling time. The goal is designing the sampled-data controller such that at each sampling time, the stability of the closed-loop system is guaranteed and an infinite horizon cost function is minimised. The main advantage of the proposed approach is to eliminate the approximations induced from discretizing the original system and cost function upper bound minimisation. Consequently, a lower bound of the cost function is obtained and the performance of the proposed model predictive controller is improved compared to the recently published papers in the same field of interest. In addition, the Euclidean norm constraint of the control input vector is derived in terms of LMIs. To illustrate the merits of the proposed approach, the proposed technique is applied to a continuous stirred tank reactor system.

Strong accessibility and integral manifolds of the continuous-time nonlinear control systems

The paper develops the algebraic formalism that is based on the language of ideals and modules associated with the analytic control system given by the ordinary differential equations. Using the Nagano theorem we show how the integral manifolds of Lie algebra of the considered system can be determined by the generators of some ideal of the ring of germs of analytic functions that is invariant with respect to germs of vector fields from the Lie algebra associated with the system. Additionally, basing on the language of ideals and modules the strong accessibility problem is studied. The singular points where the rank of (co)distributions associated to the system is different than at points from their neighbourhoods are considered. Using the germs of one-forms associated with the generators of the ideal one can define the integrable codistribution that in general is not analytic.

Bibo Stabilisation of Continuous–Time Takagi–Sugeno Systems under Persistent Perturbations and Input Saturation

Abstract This paper presents a novel approach to the design of fuzzy state feedback controllers for continuous-time non-linear systems with input saturation under persistent perturbations. It is assumed that all the states of the Takagi-Sugeno (TS) fuzzy model representing a non-linear system are measurable. Such controllers achieve bounded input bounded output (BIBO) stabilisation in closed loop based on the computation of inescapable ellipsoids. These ellipsoids are computed with linear matrix inequalities (LMIs) that guarantee stabilisation with input saturation and persistent perturbations. In particular, two kinds of inescapable ellipsoids are computed when solving a multiobjective optimization problem: the maximum volume inescapable ellipsoids contained inside the validity domain of the TS fuzzy model and the smallest inescapable ellipsoids which guarantee a minimum *-norm (upper bound of the 1-norm) of the perturbed system. For every initial point contained in the maximum volume ellipsoid, the closed loop will enter the minimum *-norm ellipsoid after a finite time, and it will remain inside afterwards. Consequently, the designed controllers have a large domain of validity and ensure a small value for the 1-norm of closed loop.

Dynamic output-feedback control for nonlinear continuous-time systems based on parametric uncertain subsystem and interval type-2 fuzzy model

This paper addresses the interval type-2 fuzzy robust dynamic output-feedback control problem for a class of nonlinear continuous-time systems with parametric uncertainties and immeasurable premise variables. First, the parametric uncertainties are assumed to be a subsystem based on the control input matrix and output matrix, and described as a linear fractional. Secondly, the nonlinear continuous-time systems are described by the interval type-2 fuzzy model. Thirdly, the new dynamic output feedback controller is designed based on the interval type-2 fuzzy model and the linear fractional (parametric uncertainties), the sufficient conditions for robust stabilization are given in the form of linear matrix inequalities (LMIs). Compared with previous work, the developed methods not only have abilities to handle the fuzzy system with immeasurable premise variables but also can deal with the parametric uncertainties effectively. The results are further extended to a mobile robot case and a chemical process case. Finally, two simulation examples are performed to show the effectiveness of the propose methods.

Self-learning robust optimal control for continuous-time nonlinear systems with mismatched disturbances

This paper presents a novel adaptive dynamic programming(ADP)-based self-learning robust optimal control scheme for input-affine continuous-time nonlinear systems with mismatched disturbances. First, the stabilizing feedback controller for original nonlinear systems is designed by modifying the optimal control law of the auxiliary system. It is also demonstrated that this feedback controller can optimize a specified value function. Then, within the framework of ADP, a single critic network is constructed to solve the Hamilton–Jacobi–Bellman equation associated with the auxiliary system optimal control law. To update the critic network weights, an indicator function and a concurrent learning technique are employed. By using the proposed update law for the critic network, the restrictive conditions including the initial admissible control and the persistence of excitation condition are relaxed. Moreover, the stability of the closed-loop auxiliary system is guaranteed in the sense that all the signals are uniformly ultimately bounded. Finally, the applicability of the developed control strategy is illustrated through simulations for an unstable nonlinear plant and a power system.