QueueLearner: A Knowledge-Combined Reinforcement Learning to Understand Queuing Evolution in Isolated Traffic Signal Control

Abstract

Reinforcement learning (RL) methods have been used in traffic signal control for decades. Traditional RL controller with model-free design treat traffic states as a Markov Process (MP) to approximate future benefit of control strategy and improve its policy with discretized action space. In such treatment, the statistical connection between traffic state and RL produced action might mismatch with theoretical understanding. The mismatching can lead to a invalid control when traffic under control differing from statistical assumption. To enhance inference processes of RL controller with traffic engineering knowledge, a knowledge-combined RL controller, QueueLearner, is proposed for the isolated intersection control problem. The estimated queue is first introduced to teach queuing evolution with realistic detection to a Deep Q-Network (DQN) controller. Furthermore, a changeable phase duration is employed to address the DQN limitation of discrete action. The proposed QueueLearner outperforms traffic engineering and RL methods in a diverse simulation experiment with the same data conditions.