Robust safe reinforcement learning control of unknown continuous-time nonlinear systems with state constraints and disturbances

Abstract

Industrial processes must operate safely and optimally in practice, which typically entails solving a constrained optimal control (COC) problem. Finding a control policy for such a problem, however, is challenging especially when the system dynamics are unknown. In this paper, a novel safe reinforcement learning (RL) algorithm is proposed to deal with the COC problem for the continuous-time nonlinear system with unknown dynamics and disturbances. The presented method can ensure that the system operates within a predefined safe region in the presence of system model uncertainties and external disturbances, which goes beyond the results of typical RL methods. To handle the system state constraints, the problem is first transformed into an unconstrained one via the proposed data-driven slack function approach. Then an improved model-based RL method is devised to learn a near-optimal and safe control policy in real-time. Finally, a robust compensator and the composite neural-network updating rule are designed to eliminate the influence of disturbances and uncertainties. Theoretical analysis based on the Lyapunov approach is conducted to prove the closed-loop system stability and the algorithm convergence, which further guarantees the satisfaction of safety constraints. The effectiveness of the proposed RL algorithm is verified through simulations.