Cooperative cache update using multi-agent recurrent deep reinforcement learning for mobile edge networks

Abstract

Caching the most likely to be requested content at the base stations in a cooperative manner can facilitate direct content delivery without fetching content from the remote content server and thus alleviate the user-perceived latency, reduce the burden on backhaul and minimize the duplicated content transmissions. Content popularity plays a vital role, and it drives caching on edge. In the literature, earlier works considered the content popularity either known earlier or obtained on prediction. However, the content popularity is time-varying and unknown in reality, so the above assumption makes it less practical. Therefore, this paper considers the cooperative cache replacement problem in a realistic scenario where the edge nodes are unaware of the content popularity in mobile edge networks. To address this problem, the main contribution of this paper is to design an intelligent content update mechanism using multi-agent deep reinforcement learning in dynamic environments. With the goal of maximizing the saved delay with deadline and capacity constraints, we formulate the cache replacement problem as Integer linear programming problem. Considering the dynamic nature of the content popularity, high dimensional parameters, and for an intelligent caching decision, we model the problem as a partially observable Markov decision process and present an efficient deep reinforcement learning algorithm by embedding the long short-term memory network (LSTM) into a multi-agent deep deterministic policy gradient formalism. The LSTM inclusion reduces the instability produced by partial observability of the environment. Extensive simulation results demonstrate that the proposed cooperative caching mechanism significantly improves the performance in terms of reward, acceleration ratio and hit ratio compared with existing mechanisms.