A semi-decentralized feudal multi-agent learned-goal algorithm for multi-intersection traffic signal control

Abstract

Traffic signal control helps to reduce traffic congestion and thus has been studied for a few decades. Algorithms for multi-intersection traffic signal control that adopt hierarchical deep reinforcement learning have been shown to achieve state-of-the-art performance. For these methods, a two-level hierarchical structure is generally used, in which the low-level policies employ latent goals induced by the high-level policies. Existing research either designs the latent goals manually or acquires the latent goals from the environment. However, these latent goals are not optimal, which leads to the non-optimal low-level policies. To improve this, we propose a learned-goal soft actor–critic (LSAC) algorithm, by which the optimal latent goals are automatically learned and then are used in low-level policies. We then propose a semi-decentralized feudal multi-agent (SFM) framework, which can alleviate the problem that existing multi-agent framework faces, i.e., the fast growing state space with the increase of the number of agents. Combining the above two proposed methods, an SFM-LSAC algorithm is proposed in this paper for multi-intersection traffic signal control. Other techniques are also incorporated into the SFM-LSAC algorithm such as attention mechanism. Experimental results in three real-world traffic scenarios illustrate that the SFM-LSAC algorithm outperforms other four state-of-the-art multi-intersection traffic signal control algorithms, i.e., reduces the average intersection delay, the average queue length, the average travel time, meanwhile improves the average flow and average travel speed, thus could be a new promising algorithm for multi-intersection traffic signal control.