Safe batch constrained deep reinforcement learning with generative adversarial network

Abstract

Batch-constrained reinforcement learning constrains the learned policy to be close to the behavior policy, which holds a tremendous promise for alleviating the distributional shift in offline reinforcement learning. Existing batch-constrained techniques rely on perturbation models to adjust the actions generated from the generative model to maximize the estimated value function. However, the perturbation model deviates from the distribution of the offline datasets and introduces a new distribution drift problem, which affects the performance of the learned policies. In addition, since offline reinforcement learning cannot learn by trial and error, the final policies are often prone to failure in reality or make unsafe decisions when trained with a noisy or small size dataset. To address the above issues, this paper employs constrained generative adversarial network to generate actions with given states. Specifically, we train the generator to maximize the estimated value and constrain the state-action pairs to follow the dataset distribution. The perturbation model is trained to maximize the probability of the perturbed actions belonging to the dataset and minimize the likelihood of taking dangerous actions. Moreover, we utilize safety critics to predict the risk of the actions under a state. Experimental results show that the proposed method is effective and can choose safe actions while maintaining a high performance in offline settings.