Cooperative traffic signal control through a counterfactual multi-agent deep actor critic approach

Abstract

Signal control has been effective to alleviate urban traffic congestion. Massive related works about signal timing optimization have been proposed and led to many signal control methods and systems. In recent years, reinforcement learning (RL) algorithms have attracted the increasing attention of researchers in the area of signal control optimization, since they can learn the optimal timing policy themselves by analyzing changing patterns between the traffic condition and signal timing plans. Multi-intersection traffic signal control problems face more challenges than classical single intersection problems such as the dimensionality issue as the joint action space of multiple agents grows exponentially with the number of intersections. The existing state-of-art multi-agent reinforcement learning algorithms developed for other areas may not suit well with traffic signal control given the complex spatial–temporal nature of traffic flows. In this paper, we propose a novel multi-agent counterfactual actor–critic with scheduler (MACS) framework for multiple interactions. In this method, decentralized actors control the traffic signals and the centralized critic combines recurrent policies with feed-forward critics. Additionally, a scheduler module that exchanges information among agents helps the individual agent better understand the entire environment. The proposed method was evaluated using simulation experiments based on a real-world urban street network in Shenzhen, China. Results showed that the proposed method outperforms the classic model-based method and several existing RL-based methods according to different measures such as queue length, average travel time delay, and throughput.