Cooperative multi-agent actor–critic control of traffic network flow based on edge computing

Abstract

Most of the existing traffic signal control strategies are hard to satisfy the real-time requirements of traffic big data analysis, knowledge reasoning and decision making for sophisticated traffic dynamics and heterogeneous intersection structures in the context of Internet of Vehicles (IoV). In this paper, we attempt to propose a cooperative multi-agent actor–critic (CMAC) deep reinforcement learning (DRL) approach with value decomposition based on edge computing architecture. The intuition behind CMAC is to decompose the global actor–critic learning tasks into several local actor–critic​ sub-problems with respect to each intersection. Each agent searches the local optimal decision by actor–critic network that takes the discrete state encoding about several consecutive frames of image-like traffic states as the inputs of the network. Among them, the green ratio output strategy considering multiple constraints is formulated in the output layer of the actor network, so that the continuous control of traffic signals using multi-agent DRL (MADRL) can be realized. Furthermore, a cooperative mechanism that considers contribution weight distributions of local agents to the global traffic pattern is proposed to coordinate multiple local agents to evolve toward global optimization. Especially, some parallel training tasks of CMAC with a large number of computing loads are deployed on the cloud side in the edge computing architecture to accelerate learning and reconstructing knowledge. The well-trained multi-agent model is downloaded from the cloud side into the edge side for real-time decision making of traffic network flow adaptive control. Simulation results with regard to a realistic traffic network demonstrate that the proposed CMAC approach under edge computing architecture outperforms the value-decomposition based multi-agent actor–critic (VMAC), independent multi-agent actor–critic (IMAC), and the fixed timing control (FTC) in terms of alleviating traffic congestion.