Optimal Control Of Nonlinear Systems Research Articles

Knowledge-Data Driven Optimal Control for Nonlinear Systems and Its Application to Wastewater Treatment Process.

Optimal control is developed to guarantee nonlinear systems run in an optimum operating state. However, since the operation demands of systems are dynamically changeable, it is difficult for optimal control to obtain reliable optimal solutions to achieve satisfying operation performance. To overcome this problem, a knowledge-data driven optimal control (KDDOC) for nonlinear systems is designed in this article. First, an adaptive initialization strategy, using the knowledge from historical operation information of nonlinear systems, is employed to dynamically preset parameters of KDDOC. Then, the initial performance of KDDOC can be enhanced for nonlinear systems. Second, a knowledge guide-based global best selection mechanism is used to assist KDDOC in searching for the optimal solutions under different operation demands. Then, dynamic optimal solutions of KDDOC can be obtained to adapt to flexible changes in nonlinear systems. Third, a knowledge direct-based exploitation mechanism is presented to accelerate the solving process of KDDOC. Then, the demand response speed of KDDOC can be improved to ensure nonlinear systems with optimal operation performance in different states. Finally, the performance of KDDOC is validated on a simulation and a practical process. Several experimental results illustrate the effectiveness of the proposed optimal control for nonlinear systems.

Physics‐informed reinforcement learning for optimal control of nonlinear systems

AbstractThis article proposes a model‐free framework to solve the optimal control problem with an infinite‐horizon performance function for nonlinear systems with input constraints. Specifically, two Physics‐Informed Neural Networks (PINNs) that incorporate the knowledge of the Lyapunov stability theorem and the convergence conditions of the policy iteration algorithm are utilized to approximate the value function and control policy, respectively. Then, a Reinforcement Learning (RL) algorithm that does not require any first‐principles or data‐driven models of nonlinear systems is developed to iteratively learn a nearly optimal control policy. Furthermore, we provide a rigorous theoretical analysis showing the conditions that ensure the stability of closed‐loop systems with the control policy learned by RL and guarantee the convergence of the iteration algorithm. Finally, the proposed Physics‐Informed Reinforcement Learning (PIRL) method is applied to a chemical process example to demonstrate its effectiveness.

Reinforcement Learning-Based Adaptive Optimal Control for Nonlinear Systems With Asymmetric Hysteresis.

This article investigates the adaptive optimal tracking problem for a class of nonlinear affine systems with asymmetric Prandtl-Ishlinskii (PI) hysteresis nonlinearities based on actor-critic (A-C) learning mechanisms. Considering the huge obstacles arising from the uncertainty of hysteresis nonlinearity in actuators, we develop a scheme for the conflict between the construction of Hamilton functions and hysteresis nonlinearity. The actuator hysteresis forces the input into a hysteresis delay, thus preventing the Hamilton function from getting the current moment's input instantly and thus making optimization impossible. In the first step, an inverse model is constructed to compensate for the hysteresis model with a shift factor. In the second step, we compensate for the control input by designing a feedback controller and incorporating the estimation and approximation errors into the Hamilton error. Optimal control, the other part of the actual control input, is obtained by taking partial derivatives of the Hamiltonian function after the nonlinearities have been circumvented. At the end, a simulation is given to validate the developed solution.

Distributed Stochastic Optimal Control of Nonlinear Systems Based on ADMM

Distributed Stochastic Optimal Control of Nonlinear Systems Based on ADMM

Distributed Optimal Control of Nonlinear System Based on Policy Gradient With External Disturbance

Distributed Optimal Control of Nonlinear System Based on Policy Gradient With External Disturbance

Safe Transfer-Reinforcement-Learning-Based Optimal Control of Nonlinear Systems.

Traditional reinforcement learning (RL) methods for optimal control of nonlinear processes often face challenges, such as high computational demands, long training times, and difficulties in ensuring the safety of closed-loop systems during training. To address these issues, this work proposes a safe transfer RL (TRL) framework. The TRL algorithm leverages knowledge from pretrained source tasks to accelerate learning in a new, related target task, significantly reducing both learning time and computational resources required for optimizing control policies. To ensure safety during knowledge transfer and training, data collection and optimization of the control policy are performed within a control invariant set (CIS) throughout the learning process. Furthermore, we theoretically analyze the errors between the approximate and optimal control policies by accounting for the differences between source and target tasks. Finally, the proposed TRL method is applied to the case studies of chemical processes to demonstrate its effectiveness in solving the optimal control problem with improved computational efficiency and guaranteed safety.

Multiobjective Integrated Optimal Control for Nonlinear Systems.

The multiobjective optimal control method optimizes the performance indexes of nonlinear systems to obtain setpoints, and designs a controller to track the setpoints. However, the stepwise optimal control method that independently analyzes the optimization process may obtain unfeasible and difficult to track setpoints, which will reduce the operation and control performance of the systems. To solve this problem, a multiobjective integrated optimal control (MIOC) strategy is proposed for nonlinear systems in this article. The main contributions of MIOC are threefold. First, in the framework of multiobjective model predictive control, an integrated control structure with a comprehensive cost function and a collaborative optimization algorithm is designed to achieve the coordinate optimal control. Second, for the time inconformity of setpoints and control laws caused by the characteristic of tracking control, the different prediction horizons are designed for the comprehensive cost function. Then, the collaborative optimization algorithm is proposed for the comprehensive cost function to achieve the integrated solution of setpoints and control laws to enhance the operation and control performance of nonlinear systems. Third, the stability and control performance analysis of MIOC is provided. Finally, the proposed MIOC method is applied for a nonlinear system to demonstrate its effectiveness.

Control Lyapunov‐barrier function‐based safe reinforcement learning for nonlinear optimal control

AbstractThis article develops a safe reinforcement learning (SRL) algorithm for optimal control of nonlinear systems with input constraints. First, we design a novel performance index function by taking advantage of control Lyapunov‐barrier functions (CLBF) with inherent safety and stability properties to ensure closed‐loop stability and safety during operation under the optimal control policy. Additionally, since it is challenging to represent the CLBF‐based value function as an explicit function of process states, neural networks (NNs) are used to approximate the value function using the process operational data that indicate safe and unsafe operations. Theoretical results on the stability, safety, and optimality of the SRL algorithm are developed, accounting for the approximation error of the NN‐based value function. Finally, the efficacy of the proposed safe optimal control scheme is shown using an application to a chemical process example.

Approximate time optimal control by deep neural networks trained with numerically obtained optimal trajectories

AbstractThis paper focuses on online time optimal control of nonlinear systems. This is achieved by approximating the results of time optimal control problems (TOCP) with deep neural networks (DNN) depending on the initial and terminal system state. In general, solving a TOCP for nonlinear systems is a computationally challenging task. Especially in the context of time optimal nonlinear model predictive control (TMPC) with hard real time constraints successful termination of a TOCP within sample times suitable for controlling mechanical systems cannot be guaranteed. Therefore, our approach is to train three DNNs with different aspects of numerical solutions of TOCPs with random initial and terminal state. These networks can then be used to approximate the TMPC by a one step model predictive control scheme with a significantly simpler structure and decreased calculation time. In order to verify this procedure by simulation, it is applied to a prove of concept example as well as the model of an industrial robot.

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Optimal control of nonlinear system based on deterministic policy gradient with eligibility traces

Critic Learning-Based Safe Optimal Control for Nonlinear Systems with Asymmetric Input Constraints and Unmatched Disturbances.

In this paper, the safe optimal control method for continuous-time (CT) nonlinear safety-critical systems with asymmetric input constraints and unmatched disturbances based on the adaptive dynamic programming (ADP) is investigated. Initially, a new non-quadratic form function is implemented to effectively handle the asymmetric input constraints. Subsequently, the safe optimal control problem is transformed into a two-player zero-sum game (ZSG) problem to suppress the influence of unmatched disturbances, and a new Hamilton-Jacobi-Isaacs (HJI) equation is introduced by integrating the control barrier function (CBF) with the cost function to penalize unsafe behavior. Moreover, a damping factor is embedded in the CBF to balance safety and optimality. To obtain a safe optimal controller, only one critic neural network (CNN) is utilized to tackle the complex HJI equation, leading to a decreased computational load in contrast to the utilization of the conventional actor-critic network. Then, the system state and the parameters of the CNN are uniformly ultimately bounded (UUB) through the application of the Lyapunov stability method. Lastly, two examples are presented to confirm the efficacy of the presented approach.

Simulating Interaction Movements via Model Predictive Control

We present a Model Predictive Control (MPC) framework to simulate movement in interaction with computers, focusing on mid-air pointing as an example. Starting from understanding interaction from an Optimal Feedback Control (OFC) perspective, we assume that users aim at minimizing an internalized cost function, subject to the constraints imposed by the human body and the interactive system. Unlike previous approaches used in HCI, MPC can compute optimal controls for nonlinear systems. This allows to use state-of-the-art biomechanical models and handle nonlinearities that occur in almost any interactive system. Instead of torque actuation, our model employs second-order muscles acting directly at the joints. We compare three different cost functions and evaluate the simulation against user movements in a pointing study. Our results show that the combination of distance, control, and joint acceleration cost matches individual users’ movements best, and predicts movements with an accuracy that is within the between-user variance. To aid HCI researchers and designers in applying our approach for different users, interaction techniques, or tasks, we make our SimMPC framework, including CFAT, a tool to identify maximum voluntary torques in joint-actuated models, publicly available, and give step-by-step instructions.

Approximate Optimal Control for Nonlinear Systems With Periodic Event-Triggered Mechanism.

This article investigates the approximate optimal control problem for nonlinear affine systems under the periodic event triggered control (PETC) strategy. In terms of optimal control, a theoretical comparison of continuous control, traditional event-based control (ETC), and PETC from the perspective of stability convergence, concluding that PETC does not significantly affect the convergence rate than ETC. It is the first time to present PETC for optimal control target of nonlinear systems. A critic network is introduced to approximate the optimal value function based on the idea of reinforcement learning (RL). It is proven that the discrete updating time series from PETC can also be utilized to determine the updating time of the learning network. In this way, the gradient-based weight estimation for continuous systems is developed in discrete form. Then, the uniformly ultimately bounded (UUB) condition of controlled systems is analyzed to ensure the stability of the designed method. Finally, two illustrative examples are given to show the effectiveness of the method.

Optimal Control of Nonlinear Systems With Unsymmetrical Input Constraints and its Application to the UAV Circumnavigation Problem

In this article, a novel design scheme is introduced to solve the optimal control problem for nonlinear systems with unsymmetrical and state-dependent input constraints. By introducing an initial stabilizing control policy as the baseline of the constructed optimal control policy, we remove the assumption in the current study for the adaptive optimal control, that is, the internal dynamics should hold zero when the state of the system is in the origin. Particularly, nonlinear control systems with partially unknown dynamics are investigated and the procedure to acquire the corresponding optimal control policy is presented. The stability for the closed-loop dynamics and the optimality of the obtained control policy are both proved. Besides, we apply the proposed control design framework to solve the optimal circumnavigation problem based on the accumulative Fisher information for a fixed-wing unmanned aerial vehicle (UAV). The control performance of our algorithm is compared with that of the existing circumnavigation control policy in a numerical simulation.

Relaxed Actor-Critic With Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems

This paper presents the Relaxed Continuous-Time Actor-critic (RCTAC) algorithm, a method for finding the nearly optimal policy for nonlinear continuous-time (CT) systems with known dynamics and infinite horizon, such as the path-tracking control of vehicles. RCTAC has several advantages over existing adaptive dynamic programming algorithms for CT systems. It does not require the ``admissibility" of the initialized policy or the input-affine nature of controlled systems for convergence. Instead, given any initial policy, RCTAC can converge to an admissible, and subsequently nearly optimal policy for a general nonlinear system with a saturated controller. RCTAC consists of two phases: a warm-up phase and a generalized policy iteration phase. The warm-up phase minimizes the square of the Hamiltonian to achieve admissibility, while the generalized policy iteration phase relaxes the update termination conditions for faster convergence. The convergence and optimality of the algorithm are proven through Lyapunov analysis, and its effectiveness is demonstrated through simulations and real-world path-tracking tasks.

Data‐based learning control for optimization of nonlinear systems

Data‐based learning control for optimization of nonlinear systems

Product process innovation model of fuzzy optimal control of nonlinear system with finite time horizon under granular differentiability concept

Product process innovation model of fuzzy optimal control of nonlinear system with finite time horizon under granular differentiability concept

Data-driven optimal control via linear transfer operators: A convex approach

This paper is concerned with the data-driven optimal control of nonlinear systems. We present a convex formulation of the optimal control problem with a discounted cost function. We consider optimal control problems with both positive and negative discount factors. The convex approach relies on lifting nonlinear system dynamics in the space of densities using the linear Perron–Frobenius operator. This lifting leads to an infinite-dimensional convex optimization formulation of the optimal control problem. The data-driven approximation of the optimization problem relies on the approximation of the Koopman operator and its dual: the Perron–Frobenius operator, using a polynomial basis function. We write the approximate finite-dimensional optimization problem as a polynomial optimization which is then solved efficiently using a sum-of-squares-based optimization framework. Simulation results demonstrate the efficacy of the developed data-driven optimal control framework.

Distributed optimal control of nonlinear systems using a second-order augmented Lagrangian method

In this paper, we propose a distributed second-order augmented Lagrangian method for distributed optimal control problems, which can be exploited for distributed model predictive control. We employ a primal-dual interior-point approach for the inner iteration of the augmented Lagrangian and distribute the corresponding computations using message passing over what is known as the clique tree of the problem. The algorithm converges to its centralized counterpart and it requires fewer communications between sub-systems as compared to algorithms such as the alternating direction method of multipliers. We illustrate the efficiency of the framework when applied to randomly generated interconnected sub-systems as well as to a vehicle platooning problem.

Excitation for Adaptive Optimal Control of Nonlinear Systems in Differential Games

This article focuses on the fulfillment of the persistent excitation (PE) condition for signals which result from transformations by means of polynomials. This is essential, e.g., for the convergence of adaptive dynamic programming algorithms due to commonly used polynomial function approximators. As theoretical statements are scarce regarding the nonlinear transformation of PE signals, we propose conditions on the system state such that its transformation by polynomials is PE. To validate our theoretical statements, we develop an exemplary excitation procedure based on our conditions using a feed-forward control approach and demonstrate the effectiveness of our method in a nonzero-sum differential game. In this setting, our approach outperforms commonly used probing noise in terms of convergence time and the degree of PE, shown by a numerical example.