A distributed deep reinforcement learning method for traffic light control

Abstract

This paper proposes a distributed deep reinforcement learning algorithm for the traffic light control problem, which consists of local learning and global consensus. Firstly, the reinforcement learning environment for the traffic light control problem is built by defining the three key elements of state, action, and reward. Then, the CNN-based deep Q-network is designed to process the quantized traffic state information to obtain the state-action values. After locally optimizing the deep Q-networks of multiple traffic light agents based on their experience samples, the consensus algorithm is subsequently applied to globally update these agents that are connected over a decentralized communication topology. In this way, the distributed learning agents learn from their neighbors’ experience to optimize the modeling process without actually sharing experience data samples. Lastly, homogeneous and heterogeneous traffic flow patterns on different intersections are simulated in SUMO to verify the superiority of the proposed distributed deep Q-networks, with the comparison to the fixed-time strategy, local learning and centralized learning algorithms. The simulation study demonstrates that the distributed learning algorithm without a central server shows comparable performance with centralized learning, which is much better than fixed-time strategy and the local learning method in both homogeneous and heterogeneous traffic scenarios.