Deep reinforcement learning with reference system to handle constraints for energy-efficient train control

Abstract

Train energy-efficient control involves complicated optimization processes subject to constraints such as speed, time, position and comfort requirements. Conventional optimization techniques are not apt at accumulating numerous solution instances into decision intelligence by learning for consecutively confronted new problems. Deep reinforcement learning (DRL), which can directly output control decisions based on current states, has shown great potentials for next-generation intelligent control . However, if the DRL is directly applied to energy-efficient train control, the received results are almost unsatisfactory. The reason lies in that the agent may get into confusion about how to trade off those constraints, and spend great computational time performing a large number of meaningless explorations. This article attempts to propose an approach of DRL with a reference system (DRL-RS) for proactive constraint handling, where the reference system deals with checking and correcting the agent’s learning progresses to avoid stepping farther and farther onto the erroneous road. The proposed approach is evaluated by the numerical experiments on train control in metro lines. The experimental results demonstrate that the DRL-RS can achieve faster learning convergence, compared with the directly applied DRL. Furthermore, it is possible to reduce more energy consumption than the commonly used genetic algorithm .