A dynamic penalty approach to state constraint handling in deep reinforcement learning

Abstract

Deep reinforcement learning (RL) has emerged as a promising approach to solving sequential optimization problems that involve high dimensional state/action space and stochastic uncertainties. Unfortunately, many such problems, especially those related to process control, involve state constraints (typically expressed as inequality constraints) that are difficult to handle. Most RL application studies have incorporated inequality constraints into the training by adding penalty terms for violating the constraints to the reward function. However, while training neural networks to learn the value (or Q) function, one can run into numerical difficulties caused by sharp changes in the value function (VF) at the constraint boundary. This problem can lead to an array of issues including getting stuck in local minima and slow convergence during training which ultimately manifest as poor closed-loop performance when training samples are limited. In this paper, we first examine the effect of the penalty function form on the neural network training performance in deep RL algorithms. To address the slow convergence, we propose a dynamic penalty (DP) approach where the penalty factor is gradually and systematically increased as the iteration episodes proceed during training. The agents trained by a Deep Q-Learning algorithm with the proposed approach were compared with agents trained with other constant penalty functions in a vehicle control problem and in a battery management control problem. Results show that the DP approach can help improve the accuracy of the VF approximation, leading to superior results in the constraint satisfaction and in the degree of violation.