Stability-certified reinforcement learning control via spectral normalization

Abstract

In this study, two types of methods from different perspectives based on spectral normalization (SN) are described for ensuring the stability of a feedback system controlled by a neural network (NN). The first one is that the L2 gain of the feedback system is bounded less than 1 to satisfy a stability condition derived from the small-gain theorem. When explicitly including the stability condition, the first type of method may provide an insufficient performance on the NN controller due to its strict stability condition. To overcome this difficulty, the second type of method is proposed, ensuring local stability with a larger region of attraction. In this second type, the stability is ensured by solving linear matrix inequalities after training the NN controller. SN improves the feasibility of the a posteriori stability test by constructing tighter local sectors. Numerical experiments show that the second type of method provides sufficient performance compared with the first one and ensures sufficient stability compared with existing reinforcement learning algorithms.11Project page: https://sites.google.com/g.ecc.u-tokyo.ac.jp/stability-certified-rl-via-sn.