Distributed Multi-Agent Deep Q-Learning for Load Balancing User Association in Dense Networks

Abstract

Distributed learning can lead to effective user association with low overhead, but faces significant challenges in incorporating load balancing at all base stations (BS) because of coupling constraints. In this paper, we propose a distributed multi-agent deep Q-learning model for user association to satisfy the load balancing constraint at every BS. In particular, we design a deep Q-network (DQN) with target Q-network and experience replay buffer at each user as an agent, and propose a multi-agent matching policy to control the number of users connected to each BS for load balancing. The policy enhances network throughput performance by implementing a novel updating rule for the preference list at each BS. The proposed multi-agent DQN model operates in a fully distributed manner where each agent only uses local information and requires no information exchange between agents. Simulation results demonstrate that our proposed algorithm outperforms a conventional distributed load balancing algorithm and approaches a centralized scheme performance, while exhibiting fast convergence and high adaptability to channel changes.