Optimal Control Of Nonlinear Systems Research Articles

Physics‐informed reinforcement learning for optimal control of nonlinear systems

AbstractThis article proposes a model‐free framework to solve the optimal control problem with an infinite‐horizon performance function for nonlinear systems with input constraints. Specifically, two Physics‐Informed Neural Networks (PINNs) that incorporate the knowledge of the Lyapunov stability theorem and the convergence conditions of the policy iteration algorithm are utilized to approximate the value function and control policy, respectively. Then, a Reinforcement Learning (RL) algorithm that does not require any first‐principles or data‐driven models of nonlinear systems is developed to iteratively learn a nearly optimal control policy. Furthermore, we provide a rigorous theoretical analysis showing the conditions that ensure the stability of closed‐loop systems with the control policy learned by RL and guarantee the convergence of the iteration algorithm. Finally, the proposed Physics‐Informed Reinforcement Learning (PIRL) method is applied to a chemical process example to demonstrate its effectiveness.

Reinforcement Learning-Based Adaptive Optimal Control for Nonlinear Systems With Asymmetric Hysteresis.

This article investigates the adaptive optimal tracking problem for a class of nonlinear affine systems with asymmetric Prandtl-Ishlinskii (PI) hysteresis nonlinearities based on actor-critic (A-C) learning mechanisms. Considering the huge obstacles arising from the uncertainty of hysteresis nonlinearity in actuators, we develop a scheme for the conflict between the construction of Hamilton functions and hysteresis nonlinearity. The actuator hysteresis forces the input into a hysteresis delay, thus preventing the Hamilton function from getting the current moment's input instantly and thus making optimization impossible. In the first step, an inverse model is constructed to compensate for the hysteresis model with a shift factor. In the second step, we compensate for the control input by designing a feedback controller and incorporating the estimation and approximation errors into the Hamilton error. Optimal control, the other part of the actual control input, is obtained by taking partial derivatives of the Hamiltonian function after the nonlinearities have been circumvented. At the end, a simulation is given to validate the developed solution.

Distributed Stochastic Optimal Control of Nonlinear Systems Based on ADMM

Distributed Stochastic Optimal Control of Nonlinear Systems Based on ADMM

Distributed Optimal Control of Nonlinear System Based on Policy Gradient With External Disturbance

Distributed Optimal Control of Nonlinear System Based on Policy Gradient With External Disturbance

Multiobjective Integrated Optimal Control for Nonlinear Systems.

The multiobjective optimal control method optimizes the performance indexes of nonlinear systems to obtain setpoints, and designs a controller to track the setpoints. However, the stepwise optimal control method that independently analyzes the optimization process may obtain unfeasible and difficult to track setpoints, which will reduce the operation and control performance of the systems. To solve this problem, a multiobjective integrated optimal control (MIOC) strategy is proposed for nonlinear systems in this article. The main contributions of MIOC are threefold. First, in the framework of multiobjective model predictive control, an integrated control structure with a comprehensive cost function and a collaborative optimization algorithm is designed to achieve the coordinate optimal control. Second, for the time inconformity of setpoints and control laws caused by the characteristic of tracking control, the different prediction horizons are designed for the comprehensive cost function. Then, the collaborative optimization algorithm is proposed for the comprehensive cost function to achieve the integrated solution of setpoints and control laws to enhance the operation and control performance of nonlinear systems. Third, the stability and control performance analysis of MIOC is provided. Finally, the proposed MIOC method is applied for a nonlinear system to demonstrate its effectiveness.

Control Lyapunov‐barrier function‐based safe reinforcement learning for nonlinear optimal control

AbstractThis article develops a safe reinforcement learning (SRL) algorithm for optimal control of nonlinear systems with input constraints. First, we design a novel performance index function by taking advantage of control Lyapunov‐barrier functions (CLBF) with inherent safety and stability properties to ensure closed‐loop stability and safety during operation under the optimal control policy. Additionally, since it is challenging to represent the CLBF‐based value function as an explicit function of process states, neural networks (NNs) are used to approximate the value function using the process operational data that indicate safe and unsafe operations. Theoretical results on the stability, safety, and optimality of the SRL algorithm are developed, accounting for the approximation error of the NN‐based value function. Finally, the efficacy of the proposed safe optimal control scheme is shown using an application to a chemical process example.

Approximate time optimal control by deep neural networks trained with numerically obtained optimal trajectories

AbstractThis paper focuses on online time optimal control of nonlinear systems. This is achieved by approximating the results of time optimal control problems (TOCP) with deep neural networks (DNN) depending on the initial and terminal system state. In general, solving a TOCP for nonlinear systems is a computationally challenging task. Especially in the context of time optimal nonlinear model predictive control (TMPC) with hard real time constraints successful termination of a TOCP within sample times suitable for controlling mechanical systems cannot be guaranteed. Therefore, our approach is to train three DNNs with different aspects of numerical solutions of TOCPs with random initial and terminal state. These networks can then be used to approximate the TMPC by a one step model predictive control scheme with a significantly simpler structure and decreased calculation time. In order to verify this procedure by simulation, it is applied to a prove of concept example as well as the model of an industrial robot.

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Critic Learning-Based Safe Optimal Control for Nonlinear Systems with Asymmetric Input Constraints and Unmatched Disturbances.

In this paper, the safe optimal control method for continuous-time (CT) nonlinear safety-critical systems with asymmetric input constraints and unmatched disturbances based on the adaptive dynamic programming (ADP) is investigated. Initially, a new non-quadratic form function is implemented to effectively handle the asymmetric input constraints. Subsequently, the safe optimal control problem is transformed into a two-player zero-sum game (ZSG) problem to suppress the influence of unmatched disturbances, and a new Hamilton-Jacobi-Isaacs (HJI) equation is introduced by integrating the control barrier function (CBF) with the cost function to penalize unsafe behavior. Moreover, a damping factor is embedded in the CBF to balance safety and optimality. To obtain a safe optimal controller, only one critic neural network (CNN) is utilized to tackle the complex HJI equation, leading to a decreased computational load in contrast to the utilization of the conventional actor-critic network. Then, the system state and the parameters of the CNN are uniformly ultimately bounded (UUB) through the application of the Lyapunov stability method. Lastly, two examples are presented to confirm the efficacy of the presented approach.

Simulating Interaction Movements via Model Predictive Control

We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.

Approximate Optimal Control for Nonlinear Systems With Periodic Event-Triggered Mechanism.

This article investigates the approximate optimal control problem for nonlinear affine systems under the periodic event triggered control (PETC) strategy. In terms of optimal control, a theoretical comparison of continuous control, traditional event-based control (ETC), and PETC from the perspective of stability convergence, concluding that PETC does not significantly affect the convergence rate than ETC. It is the first time to present PETC for optimal control target of nonlinear systems. A critic network is introduced to approximate the optimal value function based on the idea of reinforcement learning (RL). It is proven that the discrete updating time series from PETC can also be utilized to determine the updating time of the learning network. In this way, the gradient-based weight estimation for continuous systems is developed in discrete form. Then, the uniformly ultimately bounded (UUB) condition of controlled systems is analyzed to ensure the stability of the designed method. Finally, two illustrative examples are given to show the effectiveness of the method.

Optimal Control of Nonlinear Systems With Unsymmetrical Input Constraints and its Application to the UAV Circumnavigation Problem

In this article, a novel design scheme is introduced to solve the optimal control problem for nonlinear systems with unsymmetrical and state-dependent input constraints. By introducing an initial stabilizing control policy as the baseline of the constructed optimal control policy, we remove the assumption in the current study for the adaptive optimal control, that is, the internal dynamics should hold zero when the state of the system is in the origin. Particularly, nonlinear control systems with partially unknown dynamics are investigated and the procedure to acquire the corresponding optimal control policy is presented. The stability for the closed-loop dynamics and the optimality of the obtained control policy are both proved. Besides, we apply the proposed control design framework to solve the optimal circumnavigation problem based on the accumulative Fisher information for a fixed-wing unmanned aerial vehicle (UAV). The control performance of our algorithm is compared with that of the existing circumnavigation control policy in a numerical simulation.

Relaxed Actor-Critic With Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems

This paper presents the Relaxed Continuous-Time Actor-critic (RCTAC) algorithm, a method for finding the nearly optimal policy for nonlinear continuous-time (CT) systems with known dynamics and infinite horizon, such as the path-tracking control of vehicles. RCTAC has several advantages over existing adaptive dynamic programming algorithms for CT systems. It does not require the ``admissibility" of the initialized policy or the input-affine nature of controlled systems for convergence. Instead, given any initial policy, RCTAC can converge to an admissible, and subsequently nearly optimal policy for a general nonlinear system with a saturated controller. RCTAC consists of two phases: a warm-up phase and a generalized policy iteration phase. The warm-up phase minimizes the square of the Hamiltonian to achieve admissibility, while the generalized policy iteration phase relaxes the update termination conditions for faster convergence. The convergence and optimality of the algorithm are proven through Lyapunov analysis, and its effectiveness is demonstrated through simulations and real-world path-tracking tasks.

Data‐based learning control for optimization of nonlinear systems

Data‐based learning control for optimization of nonlinear systems

Product process innovation model of fuzzy optimal control of nonlinear system with finite time horizon under granular differentiability concept

Product process innovation model of fuzzy optimal control of nonlinear system with finite time horizon under granular differentiability concept

Data-driven optimal control via linear transfer operators: A convex approach

This paper is concerned with the data-driven optimal control of nonlinear systems. We present a convex formulation of the optimal control problem with a discounted cost function. We consider optimal control problems with both positive and negative discount factors. The convex approach relies on lifting nonlinear system dynamics in the space of densities using the linear Perron–Frobenius operator. This lifting leads to an infinite-dimensional convex optimization formulation of the optimal control problem. The data-driven approximation of the optimization problem relies on the approximation of the Koopman operator and its dual: the Perron–Frobenius operator, using a polynomial basis function. We write the approximate finite-dimensional optimization problem as a polynomial optimization which is then solved efficiently using a sum-of-squares-based optimization framework. Simulation results demonstrate the efficacy of the developed data-driven optimal control framework.

Distributed optimal control of nonlinear systems using a second-order augmented Lagrangian method

In this paper, we propose a distributed second-order augmented Lagrangian method for distributed optimal control problems, which can be exploited for distributed model predictive control. We employ a primal-dual interior-point approach for the inner iteration of the augmented Lagrangian and distribute the corresponding computations using message passing over what is known as the clique tree of the problem. The algorithm converges to its centralized counterpart and it requires fewer communications between sub-systems as compared to algorithms such as the alternating direction method of multipliers. We illustrate the efficiency of the framework when applied to randomly generated interconnected sub-systems as well as to a vehicle platooning problem.

Excitation for Adaptive Optimal Control of Nonlinear Systems in Differential Games

This article focuses on the fulfillment of the persistent excitation (PE) condition for signals which result from transformations by means of polynomials. This is essential, e.g., for the convergence of adaptive dynamic programming algorithms due to commonly used polynomial function approximators. As theoretical statements are scarce regarding the nonlinear transformation of PE signals, we propose conditions on the system state such that its transformation by polynomials is PE. To validate our theoretical statements, we develop an exemplary excitation procedure based on our conditions using a feed-forward control approach and demonstrate the effectiveness of our method in a nonzero-sum differential game. In this setting, our approach outperforms commonly used probing noise in terms of convergence time and the degree of PE, shown by a numerical example.

Fuzzy optimal control of nonlinear systems by fuzzy generalized cell mapping method with Bellman’s principle

A novel method is proposed in this paper to obtain global solutions of fuzzy optimal control with fixed state terminal conditions and control bounds. The global solution implies that the optimal control solutions are valid for all the initial conditions in a region of the state space. The method makes use of Bellman’s principle of optimality and fuzzy generalized cell mapping method (FGCM). A discrete form of fuzzy master equation with a control dependent transition membership matrix is generated by using the FGCM. This allows to evaluate both the transient and the steady-state responses of the controlled system. The method, simply called FGCM with BP, is applied to three nonlinear systems leading to excellent control performances.

Optimal control approach for nonlinear chemical processes with uncertainty and application to a continuous stirred-tank reactor problem

Practical chemical process is usually a dynamic process including uncertainty. Stochastic constraints can be used to chemical process modeling, where constraints cannot be strictly satisfied or need not be fully satisfied. Thus, optimal control of nonlinear systems with stochastic constraints can be available to address practical nonlinear chemical process problems. This problem is hard to cope with due to the stochastic constraints. By introducing a novel smooth and differentiable approximation function, an approximation-based approach is proposed to address this issue, where the stochastic constraints are replaced by some deterministic ones. Following that, the stochastic constrained optimal control problem is converted into a deterministic parametric optimization problem. Convergence results show that the approximation function and the corresponding feasible set converge uniformly to that of the original problem. Then, the optimal solution of the deterministic parametric optimization problem is guaranteed to converge uniformly to the optimal solution of the original problem. Following that, a computation approach is proposed for solving the original problem. Numerical results, obtained from a nonlinear continuous stirred-tank reactor problem including stochastic constraints, show that the proposed approach is less conservative compared with the existing typical methods and can obtain a stable and robust performance when considering the small perturbations in initial system state.