Censored Markov Decision Processes: A Framework for Safe Reinforcement Learning in Collaboration with External Systems

Abstract

The importance of safe reinforcement learning (safe RL) is widely recognized for enhancing real world systems. In this study, we construct the censored Markov decision process (CeMDP), a new Markov Decision Process (MDP) framework that describes the interaction of environment, learner and external systems, e.g., human intervention or pre-designed controller for emergency response. We also theoretically analyze the relation of CeMDP to existing frameworks such as the semi-Markov decision process, MDP with Option (OMDP) and standard MDP; the analysis clarifies that CeMDP is a special case of OMDP and can, with environment redefinition, be represented by MDP. This finding allows us to design planning and reinforcement learning algorithms for CeMDP. We confirm the validity of the theory and algorithms by numerical experiments.