Linear Approximation based Q-Learning for Edge Caching in Massive MIMO Networks

Abstract

To meet increasing demands in wireless multimedia communications, caching of important contents in advance is one of the key solutions. Optimal caching depends on content popularity in future which is unknown. In this paper, modeling content popularity as a finite state Markov chain, reinforcement Q-learning is employed to learn optimal content placement strategy to maximize the average success probability (ASP) in homogeneous Poisson point process (PPP) distributed caching network having massive MIMO base stations. To improve over Q-learning, a linear function approximation based Q-learning is proposed which shows that only a constant number of (three) parameters need updation irrespective of size of state and action sets, while Q-learning in this context requires the parameters update of size number of states times number of actions. Given a set of available placement strategies, simulations show that the approximate Q-learning converge, successfully learns and provides the same best content placement as Q-learning, which shows the successful applicability and scalability of the approximate Q-learning.