Tensor product model transformation-based reinforcement learning neural network controller with guaranteed stability

Abstract

Despite the success of neural network (NN) controllers, the stability of closed-loop NN control systems is still a major challenge. This paper proposes a constrained reinforcement learning (RL) algorithm to design optimal NN controllers for a class of nonlinear dynamical systems while guaranteeing the stability of the control systems. By representing a controlled system as a tensor product (TP) model, a sufficient stability condition for a closed-loop NN control system is derived, which is then employed as a training constraint for the NN controller throughout the training process to ensure the stability in each training step. By showing the existence of a Lyapunov function, which is also directly obtained during the training process, the stability can be theoretically confirmed. Simulation studies on various nonlinear dynamical systems illustrate that the NN controllers trained with the proposed constrained RL algorithm outperform linear–quadratic–regulator controllers and yield roughly the same performance index values as those trained with an unconstrained RL algorithm, indicating that the imposed constraint does not significantly affect the training performance. Most importantly, while simulation results show that all controllers provide closed-loop stability, only the NN controllers trained with the proposed constrained RL algorithm theoretically guarantee stability.