An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems via Barrier Certificate Generation

Zhengfeng Yang,Yidan Zhang,Xia Zeng,Zhenbing Zeng,Zhiming Liu,Wang Lin,Xiaochao Tang

doi:10.1007/978-3-030-81685-8_22

Abstract

AbstractIn this paper, we propose a safe reinforcement learning approach to synthesize deep neural network (DNN) controllers for nonlinear systems subject to safety constraints. The proposed approach employs an iterative scheme where alearnerand averifierinteract to synthesize safe DNN controllers. Thelearnertrains a DNN controller via deep reinforcement learning, and theverifiercertifies the learned controller through computing a maximal safe initial region and its corresponding barrier certificate, based on polynomial abstraction and bilinear matrix inequalities solving. Compared with the existing verification-in-the-loop synthesis methods, our iterative framework is a sequential synthesis scheme of controllers and barrier certificates, which can learn safe controllers with adaptive barrier certificates rather than user-defined ones. We implement the tool SRLBC and evaluate its performance over a set of benchmark examples. The experimental results demonstrate that our approach efficiently synthesizes safe DNN controllers even for a nonlinear system with dimension up to 12.

Highlights

The design and synthesis of controllers for dynamical systems is a fundamental problem in the field of control
– We propose a safe reinforcement learning via barrier certificate generation to synthesize deep neural network (DNN) controller, which can guarantee the unbounded-time safety of the closed-loop systems
We have developed a novel scheme for synthesizing safe controllers of nonlinear systems with control against safety constraints

Summary

Introduction

The design and synthesis of controllers for dynamical systems is a fundamental problem in the field of control. A majority of these works lack formal reasoning about the safety of such DNN-controlled dynamical systems from such learning process. To guarantee the safety property of synthesized DNN controllers, considerable works focus on the safety verification of DNN-controlled closed-loop systems, which is a really hard problem because it is tangled with highly nonlinear DNN expressions. Other than formally verifying synthesized DNN controllers, more recent works have been proposed to learn DNN controllers for dynamical systems with safety guarantees [8,39,40]. A verification-in-the-loop DNN controller training algorithm is presented in [8], which integrates RL framework with user-provided control barrier functions (CBFs) for reward function encoding, combined with SMT based formal CBF checking; a correctness-by-design method is proposed in [39] that first learns DNN controllers and barrier certificates simultaneously using supervised learning, and performs posterior formal verification of barrier certificates via SMT solvers

Objectives

Methods

Conclusion