Safe Reinforcement Learning Benchmark Environments for Aerospace Control Systems

Abstract

Recent advancements in reinforcement learning techniques demonstrate an ability to make decisions in high dimen-sional state spaces and complex real-time strategy games. In contrast to supervised learning which features large data sets, there are relatively few existing environments for training rein-forcement learning agents. In addition, small differences in re-wards or action spaces can drastically change the difficulty and results of the training environments. Benchmarks seek to tackle both of these challenges by creating common environments, in the form of “Gyms” to train and compare reinforcement learning techniques, approaches, and algorithms. Many gyms, such as the classical control and Atari games environments, have become standard in new research on reinforcement learning. Researchers can easily compare and benchmark competing so-lutions across publications on these universal baselines enabling rapid innovation and collaboration. However, there are currently no standard set of environments for aerospace problems, and many of the gyms in the literature do not include safety con-straints or run time assurance systems that intervene when the reinforcement learning agent violates safety constraints. This manuscript describes the development of the Aerospace SafeRL Framework and accompanying Aerospace SafeRL Benchmarks that include interactive environments, safety constraints, soft-ware interfaces for run time assurance safety monitors with base implementations, and an initial set of baseline solutions. This initial set of scenarios introduces simple RL environments that expose the kinds of motion patterns, dynamics, and safety constraints encountered in air and space problems in 2D and 3D. This manuscript also describes standardized evaluation metrics for these environments to provide a consistent performance measurement with aerospace relevance. These benchmarks pro-vide a structured foundation for future reinforcement learning algorithms, run time assurance designs, and neural network verification techniques for the aerospace domain.