Multi-Agent Reinforcement Learning Based Distributed Transmission in Collaborative Cloud-Edge Systems

Abstract

Millimeter-wave (mmWave) cloud-edge collaboration has emerged as an effective solution for low-latency transmission by harnessing the potentials of cloud and edge nodes (eNodes) as well as abundant bandwidth. In a mmWave dense network, centralized processing and coordinated multi-points processing require large amounts of signaling exchange overhead, which in turn increase the latency. As such, distributed mechanism is demanded. In this paper, we investigate joint user scheduling and beam selection strategies for eNodes and formulate the problem as a constrained Markov decision process. The objective and constraints are long-term average network delay cost minimization and instantaneous quality of service (QoS) guarantee for each user equipment. To update the distributed strategies, we apply the multi-agent reinforcement learning technique and propose a hybrid learning framework by extending the actor-critic model, where centralized training and decentralized execution are implemented. We utilize the Lagrange technique to design the reward functions. Additionally, a novel actor network architecture and an exploration scheme are proposed. Simulation results validate the effectiveness of the proposed intelligent distributed algorithm with a high degree of scalability, and show superior performance in terms of long-term average network delay cost and QoS satisfaction rates compared with other methods.