Control Of Continuous-time Nonlinear Systems Research Articles

Combining hybrid metaheuristic algorithms and reinforcement learning to improve the optimal control of nonlinear continuous-time systems with input constraints

This paper proposes an innovative method for achieving optimal tracking control in nonlinear continuous-time systems with input constraints. The method combines reinforcement learning and hybrid metaheuristics to enhance the controller’s performance. Specifically, a hybrid metaheuristic algorithm is employed to optimize the hyperparameters of a critic neural network, which serves as the system’s controller. The proposed approach is evaluated through extensive simulation studies on a nonlinear system with input constraints. Results demonstrate its superiority over traditional control techniques in terms of accuracy, timeliness, and stability. Notably, the approach effectively eliminates overshoot and steady-state error while providing precise and prompt tracking and showcasing remarkable robustness against model uncertainties. By synergistically integrating reinforcement learning and hybrid metaheuristics, this approach represents a significant advancement in enhancing the control performance of complex nonlinear systems. The simulation studies confirm superiority of the proposed approach over existing techniques, offering a promising solution for achieving optimal tracking control in nonlinear systems with input constraints. This approach holds potential for a wide range of applications, including robotics, aerospace, and manufacturing, where precise and prompt tracking control is critical.

Output information-based intermittent optimal control for continuous-time nonlinear systems with unmatched uncertainties via adaptive dynamic programming

Intermittent control stands as a valuable strategy for resource conservation and cost reduction across diverse systems. Nonetheless, prevailing research is intractable to address the challenges posed by robust optimal intermittent control of nonlinear input-affine systems with unmatched uncertainties. This paper aims to fill this gap. Initially, we introduce an enhanced finite-time intermittent control approach to ensure stability within nonlinear dynamic systems harboring bounded errors. A neural networks (NNs) state observer is constructed to estimate system information. Subsequently, an optimal intermittent controller that operates within a finite time span, guaranteeing system stability by employing the Hamilton–Jacobi–Bellman (HJB) methodology. Furthermore, we devise an output information-based event-triggered intermittent (ETI) approach rooted in the robust adaptive dynamic programming (ADP) algorithm, furnishing an optimal intermittent control law. In this process, a critic NNs is introduced to estimate the cost function and optimal intermittent controller. Simulation results show that our proposed method is superior to existing intermittent control strategies.

Linear-Like Policy Iteration Based Optimal Control for Continuous-Time Nonlinear Systems

We propose a novel strategy to construct optimal controllers for continuous-time nonlinear systems by means of linear-like techniques, provided that the optimal value function is differentiable and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">quadratic-like</i> . This assumption covers a wide range of cases and holds locally around an equilibrium under mild assumptions. The proposed strategy does not require solving the Hamilton-Jacobi-Bellman equation, that is a nonlinear partial differential equation, which is known to be hard or impossible to solve. Instead, the Hamilton-Jacobi-Bellman equation is replaced with an easy-solvable state-dependent Lyapunov matrix equation. We exploit a linear-like factorization of the underlying nonlinear system and a policy-iteration algorithm to yield a linear-like policy-iteration for nonlinear systems. The proposed control strategy solves optimal nonlinear control problems in an asymptotically exact, yet still linear-like manner. We prove optimality of the resulting solution and illustrate the results via four examples.

Data-Driven Dynamic Multiobjective Optimal Control: An Aspiration-Satisfying Reinforcement Learning Approach.

This article presents an iterative data-driven algorithm for solving dynamic multiobjective (MO) optimal control problems arising in control of nonlinear continuous-time systems. It is first shown that the Hamiltonian functional corresponding to each objective can be leveraged to compare the performance of admissible policies. Hamiltonian inequalities are then used for which their satisfaction guarantees satisfying the objectives' aspirations. Relaxed Hamilton-Jacobi-Bellman (HJB) equations in terms of HJB inequalities are then solved in a dynamic constrained MO framework to find Pareto optimal solutions. Relation to satisficing (good enough) decision-making framework is shown. A sum-of-square (SOS)-based iterative algorithm is developed to solve the formulated aspiration-satisfying MO optimization. To obviate the requirement of complete knowledge of the system dynamics, a data-driven satisficing reinforcement learning approach is proposed to solve the SOS optimization problem in real time using only the information of the system trajectories measured during a time interval without having full knowledge of the system dynamics. Finally, two simulation examples are utilized to verify the analytical results of the proposed algorithm.

Resource limited event-triggered model predictive control for continuous-time nonlinear systems based on first-order hold

This paper investigates a resources-limited situation in the event-triggered model predictive control (ETMPC) for continuous-time nonlinear system with first-order hold fashion. In consideration of limited bandwidth in data transmission through wireless network under actual operation, our strategy divides the prediction horizon, and applies linear interpolation instead of zero-order hold fashion to obtain a better system performance, so that the reduction of resources and the optimization of strategy can be guaranteed. Furthermore, in actual industry processes, quadratic cost function cannot be implemented in all operations, then general cost function is adopted in this paper. Based on the first-order hold method and general cost function, the feasibility of the ETMPC algorithm and the stability of dynamical systems are analyzed. At last, a practical example is given to show the advantages of our method.

Reinforcement Learning and Adaptive Optimal Control for Continuous-Time Nonlinear Systems: A Value Iteration Approach.

This article studies the adaptive optimal control problem for continuous-time nonlinear systems described by differential equations. A key strategy is to exploit the value iteration (VI) method proposed initially by Bellman in 1957 as a fundamental tool to solve dynamic programming problems. However, previous VI methods are all exclusively devoted to the Markov decision processes and discrete-time dynamical systems. In this article, we aim to fill up the gap by developing a new continuous-time VI method that will be applied to address the adaptive or nonadaptive optimal control problems for continuous-time systems described by differential equations. Like the traditional VI, the continuous-time VI algorithm retains the nice feature that there is no need to assume the knowledge of an initial admissible control policy. As a direct application of the proposed VI method, a new class of adaptive optimal controllers is obtained for nonlinear systems with totally unknown dynamics. A learning-based control algorithm is proposed to show how to learn robust optimal controllers directly from real-time data. Finally, two examples are given to illustrate the efficacy of the proposed methodology.

Optimal sliding mode preview repetitive control for continuous-time nonlinear systems

In this paper, the problem of sliding mode preview repetitive control (SMPRC) for continuous-time nonlinear systems is proposed. First, an augmented delay system is constructed, in which the output of the modified repetitive controller is taken as a part of the augmented state vector. The augmented delay system is then transformed into the controllable canonical form by using state and linear transformations. To introduce preview compensation, a virtual optimal preview control (PC) law is designed for the uncontrollable subsystem. Next, a sliding mode control (SMC) law is cooperated with the virtual optimal PC law to ensure the robustness of the overall system in the presence of uncertainties and external disturbances. Stability of the proposed controller is also proved. Finally, the numerical simulation results demonstrate the effectiveness of the proposed method. Abbreviations: LQR: linear quadratic regulator; MRC: modified repetitive control; PC:preview control; PRC: preview repetitive control; RC: repetitive control; SFC: statefeedback control; SMC: sliding mode control; SMPRC: sliding mode preview repetitivecontrol.

Robust Trajectory Tracking Control for Continuous-Time Nonlinear Systems with State Constraints and Uncertain Disturbances.

In this paper, a robust trajectory tracking control method with state constraints and uncertain disturbances on the ground of adaptive dynamic programming (ADP) is proposed for nonlinear systems. Firstly, the augmented system consists of the tracking error and the reference trajectory, and the tracking control problems with uncertain disturbances is described as the problem of robust control adjustment. In addition, considering the nominal system of the augmented system, the guaranteed cost tracking control problem is transformed into the optimal control problem by using the discount coefficient in the nominal system. A new safe Hamilton–Jacobi–Bellman (HJB) equation is proposed by combining the cost function with the control barrier function (CBF), so that the behavior of violating the safety regulations for the system states will be punished. In order to solve the new safe HJB equation, a critic neural network (NN) is used to approximate the solution of the safe HJB equation. According to the Lyapunov stability theory, in the case of state constraints and uncertain disturbances, the system states and the parameters of the critic neural network are guaranteed to be uniformly ultimately bounded (UUB). At the end of this paper, the feasibility of the proposed method is verified by a simulation example.

Robustly complete synthesis of sampled-data control for continuous-time nonlinear systems with reach-and-stay objectives

Formal methods are becoming favorable for control and verification of safety-critical systems because of the rigorous model-based computation. Relying on an over-approximated model of the original system behaviors, formal control synthesis algorithms are not often complete, which means that a controller cannot necessarily be synthesized even if there exists one. The main result of this paper shows that, for continuous-time nonlinear systems, a sample-and-hold control strategy for a reach-and-stay specification can be synthesized whenever such a strategy exists for the same system with its dynamics perturbed by small disturbances. Control synthesis is carried out by a fixed-point algorithm that adaptively partitions the system state space into a finite number of cells. In each iteration, the reachable set from each cell after one sampling time is over-approximated within a precision determined by the bound of the disturbances. To meet such a requirement, we integrate validated high-order Taylor expansion of the system solution over one sampling period into every fixed-point iteration and provide a criterion for choosing the Taylor order and the partition precision. Two nonlinear system examples are given to illustrate the effectiveness of the proposed method.

Self-triggering adaptive optimal control for nonlinear systems based on encoding mechanism

This paper deals with self-triggering adaptive optimal control for nonlinear continuous-time systems. We propose a novel self-triggering control structure concerning a special encoding mechanism, which combines the trigger time of control and sampling and reduces both the control time and the sampling time. Such a triggering structure ensures the existence of a maximum triggering time in self-triggering control. When the system expression is known, the encoding mechanism will lead to high quantitative accuracy at a limited channel transmission rate. Moreover, we also provide a new control algorithm and triggering conditions of the proposed structure. Specifically, this algorithm solves the optimal control strategy by using the cost function approximated by neural networks. Besides, the derived closed-loop system is proven to be asymptotically stable. Finally, two examples are provided to illustrate the effectiveness of the proposed control method.

Model-Based adaptive event-Triggered control of nonlinear continuous-Time systems

This paper presents a model-based adaptive event-triggered control scheme for a class of uncertain single-input and single-output nonlinear continuous-time systems. To this aim, the explicit design of an associated controller is proposed by constructing an adaptive model, exploiting the principle of feedback linearization and using neural networks to approximate an unknown smooth nonlinear function. Then, the stability of the resulting impulsive dynamic system is strictly proved by using the Lyapunov stability theory, and the event-trigger condition is designed to realize that the NN weights and feedback signals are updated only when the condition is violated. In addition, the lower bound of inter-event times is strictly proved to be positive, which effectively avoids Zeno behavior. Compared with the conventional adaptive control based on the time-triggered scheme, feedback transmissions and NN weight updating only occur at necessary instants, such that the waste of communication resources is effectively reduced. Finally, the effectiveness of the developed control algorithm is verified by a simulation example.

Fuzzy Event-Triggered Integral Sliding Mode Control of Nonlinear Continuous-Time Systems

This article investigates the design of an event-triggered-based integral sliding mode control (SMC) for a class of multiinput Takagi–Sugeno fuzzy model (TSFM)-based nonlinear systems with disturbance. Incorporating integral SMC scheme can be used along with the event-triggered control to deal with the simultaneous external disturbances and unnecessary waste of system resources. To further accommodate the characteristic of event-triggered integral SMC (ETISMC), two fuzzy integral switching surfaces are established. One is the triggered-state dependent switching surface which is proposed to obtain the ETISMC law. The other one is the conventional continuous-state dependent switching surface which is designed to ensure the robustness of the controlled system from the initial time instance. The proposed ETISMC is applicable for both single–input and multi–input, and despite the previous studies, a more reasonable ETISMC law is proposed with asynchronous premise variables. Based on Lyapunov function and adopting a decreasing time-varying triggering threshold in the event-triggered condition, the ultimately bounded stability of the closed-loop system is guaranteed. Sufficient conditions for the ultimate bounded of the trajectories of the system states are formulated in terms of new offline linear matrix inequalities. Moreover, the positive lower bound of the internal execution is ensured, which means that there is no Zeno phenomenon. To demonstrate the superiority of the proposed technique, the suggested ETISMC is applied to a two-degree freedom helicopter system and the simulation results verify the validity of the proposed control strategy.

A modular framework for distributed model predictive control of nonlinear continuous-time systems (GRAMPC-D)

The modular open-source framework GRAMPC-D for model predictive control of distributed systems is presented in this paper. The modular concept allows to solve optimal control problems in a centralized and distributed fashion using the same problem description. It is tailored to computational efficiency with the focus on embedded hardware. The distributed solution is based on the alternating direction method of multipliers and uses the concept of neighbor approximation to enhance convergence speed. The presented framework can be accessed through C++ and Python and also supports plug-and-play and data exchange between agents over a network.

A Robust Iterative Learning Control for Continuous-Time Nonlinear Systems With Disturbances

In this paper, we study the trajectory tracking problem using iterative learning control for continuous-time nonlinear systems with a generic fixed relative degree in the presence of disturbances. This class of controllers iteratively refine the control input relying on the tracking error of the previous trials and some properly tuned learning gains. Sufficient conditions on these gains guarantee the monotonic convergence of the iterative process. However, the choice of the gains is heuristically hand-tuned given an approximated system model and no information on the disturbances. Thus, in the cases of inaccurate knowledge of the model or iteration-varying measurement errors, external disturbances, and delays, the convergence condition is unlikely to be verified at every iteration. To overcome this issue, we propose a robust convergence condition, which ensures the applicability of the pure feedforward control even if other classical conditions are not fulfilled for some trials due to the presence of disturbances. Furthermore, we quantify the upper bound of the nonrepetitive disturbance that the iterative algorithm is able to handle. Finally, we validate the convergence condition simulating the dynamics of a two degrees of freedom underactuated arm with elastic joints, where one is active, and the other is passive, and a Franka Emika Panda manipulator.

Entropy for Practical Stabilization

For deterministic continuous time nonlinear control systems, $\varepsilon $-practical stabilization entropy, and practical stabilization entropy are introduced. Here the rate of attraction is specified by a $\mathcal{KL} $-function. Upper and lower bounds for the diverse entropies are proved, with special attention to exponential $\mathcal{KL}$-functions. The relation to feedbacks is discussed; the linear case and several nonlinear examples are analyzed in detail.

Approximate Optimal Tracking Control of Nondifferentiable Signals for a Class of Continuous-Time Nonlinear Systems.

In this article, for a class of continuous-time nonlinear nonaffine systems with unknown dynamics, a robust approximate optimal tracking controller (RAOTC) is proposed in the framework of adaptive dynamic programming (ADP). The distinguishing contribution of this article is that a new Lyapunov function is constructed, by using which the derivative information of tracking errors is not required in computing its time derivative along with the solution of the closed-loop system. Thus, the proposed method can make the system states follow nondifferentiable reference signals, which removes the common assumption that the reference signals have to be continuous for tracking control of continuous-time nonlinear systems in the literature. The theoretical analysis, simulation, and application results well illustrate the effectiveness and superiority of the proposed method.

Q-Learning-Based Model Predictive Control for Nonlinear Continuous-Time Systems

In this paper, a Q-learning-based model predictive control using the Lyapunov technique (Q-LMPC) is proposed for the control of a class of continuous nonlinear systems with complicated dynamics, wh...

A Semi-Algebraic Optimization Approach to Data-Driven Control of Continuous-Time Nonlinear Systems

This letter considers the problem of designing state feedback data-driven controllers for nonlinear continuous-time systems. Specifically, we consider a scenario where the unknown dynamics can be parametrized in terms of known basis functions and the available measurements are corrupted by unknown-but-bounded noise. The goal is to use this noisy experimental data to directly design a rational state-feedback control law guaranteed to stabilize all plants compatible with the available information. The main result of this letter shows that, by using Rantzer's Dual Lyapunov approach, combined with elements from convex analysis, the problem can be recast as an optimization over positive polynomials, which can be relaxed to a semi-definite program through the use of Sum-of-Squares and semi-algebraic optimization arguments. Three academic examples are considered to illustrate the effectiveness of the proposed method.

Optimal Feedback Control of Pedestrian Flow in Heterogeneous Corridors

Maintaining the orderliness and efficiency of pedestrian flow through an architectural area is critical for the evacuation process. Especially, clogs and jams are easily triggered in width-changing areas. In this article, we consider pedestrian movement in heterogeneous corridors and design an optimal feedback control to regulate pedestrian flow. Flow characteristics are first studied based on microscopic social-force simulations. A Gaussian process describes the relationship between flow variables with the observation data. The macroscopic model for flow in heterogeneous corridors is developed. To avoid jams, discharges among these corridors are balanced with the narrowest corridor as the primary concern. At the equilibrium, a continuous-time nonlinear control system is formulated, and the adaptive dynamic programming learns the optimal feedback controller. Policy iteration (PI) and neural networks are combined together, and the convergence of neural-network-based PI is demonstrated by analyzing its equivalence to the Gauss–Newton method. Batch normalization is introduced to stabilize the learning process. Simulated experiments demonstrate that the control design can effectively regulate pedestrian flow for both macroscopic and microscopic models. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Note to Practitioners</i> —The development of video-processing techniques provides a powerful tool to detect human behavior in real time. In crowd events, the pedestrian movement must be regulated; otherwise, it is easy to fall into the faster-is-slower effect. It is especially important for evacuation routes with different widths. In this article, the optimal feedback control is studied to regulate pedestrian flow in heterogeneous corridors. It takes flow densities as state and produces commands that are composed of entrance influx and free-flow velocities. These commands can be executed with the support of speakers, displays, or the recently developed interactive robots. To avoid congestion, discharges of different corridors are balanced, and the system is optimally stabilized at equilibrium. Based on our work, engineers are able to design pedestrian flow control and achieve optimal evacuation in arbitrary heterogeneous corridors.

Dual Heuristic Programming for Optimal Control of Continuous-Time Nonlinear Systems Using Single Echo State Network.

This article presents an improved online adaptive dynamic programming (ADP) algorithm to solve the optimal control problem of continuous-time nonlinear systems with infinite horizon cost. The Hamilton-Jacobi-Bellman (HJB) equation is iteratively approximated by a novel critic-only structure which is constructed using the single echo state network (ESN). Inspired by the dual heuristic programming (DHP) technique, ESN is designed to approximate the costate function, then to derive the optimal controller. As the ESN is characterized by the echo state property (ESP), it is proved that the ESN can successfully approximate the solution to the HJB equation. Besides, to eliminate the requirement for the initial admissible control, a new weight tuning law is designed by adding an alternative condition. The stability of the closed-loop optimal control system and the convergence of the out weights of the ESN are guaranteed by using the Lyapunov theorem in the sense of uniformly ultimately bounded (UUB). Two simulation examples, including linear system and nonlinear system, are given to illustrate the availability and effectiveness of the proposed approach by comparing it with the polynomial neural-network scheme.