Memory-Augmented Lyapunov-Based Safe Reinforcement Learning: End-to-End Safety Under Uncertainty

Abstract

Despite recent advances in safe reinforcement learning (RL), safety constraints are often violated at deployment; especially under extreme uncertainty in memory-based partially observable environments. To address these limitations, we propose a memory-augmented Lyapunov-based safe RL model. The primary contributions of our method include (i) an explicit memory module based on Transformers to process long time horizons of information feedback from the environment; (ii) a memory-augmented Lyapunov function to determine a safe set of policies, and (iii) an exploration module that identifies highly rewarding safe actions by characterizing the uncertainty in the environment. We evaluate the proposed model in reactive OpenAI Safety Gym and memory-based partially observable DMLab-30 environments. The results of these experiments show that the proposed method significantly outperforms state-of-the-art baselines. Specifically, our proposed method achieves the lowest constraint costs among the tested benchmarks, while delivering high returns. Moreover, we perform ablation studies that show significant contributions of the introduced Transformer-based encoder, memory-augmented Lyapunov functions, and the uncertainty-aware exploration module.