Multi-Agent Cooperative Alternating Q-Learning Caching in D2D-Enabled Cellular Networks

Abstract

Edge caching has become an effective solution to cope with the challenges brought by the massive content delivery in cellular networks. In device- to-device (D2D) enabled caching cellular networks with time-varying user terminal (UT) movement and content popularity, we model these dynamic networks as a stochastic game to design a cooperative caching placement strategy. We consider the long-term caching placement reward of all UTs. Each UT becomes a learning agent and the caching placement strategy corresponds to the actions taken by the UTs. In an effort to solve the stochastic game problem, we propose a multi- agent cooperative alternating Q-learning (CAQL) caching placement algorithm. We discuss the convergence and complexity of CAQL, which can converge to a stable caching policy with low space complexity. Simulation results show that the proposed algorithm can effectively reduce the backhaul load and the average content access delay in dynamic environment.