A Deep Reinforcement Learning Approach to Proactive Content Pushing and Recommendation for Mobile Users

Dong Liu,Chenyang Yang

doi:10.1109/access.2019.2925019

Dong Liu, Chenyang Yang

Open Access

https://doi.org/10.1109/access.2019.2925019

Copy DOI

Abstract

The gain from proactive caching at mobile devices highly relies on the accurate prediction of user demands and mobility, which, however, is hard to achieve due to the random user behavior. In this paper, we leverage personalized content recommendation to reduce the uncertainty of user demands in sending requests. We formulate a joint content pushing and recommendation problem that maximizes the net profit of a mobile network operator. To cope with the challenges in modeling and learning user behavior, we establish a reinforcement learning (RL) framework to resolve the problem. To circumvent the curse of dimensionality of reinforcement learning for the joint problem, that is, with very large action and state spaces, we decompose the original problem into two RL problems, where two agents with different goals operate together, and we limit the number of possible actions in each state of the pushing agent by harnessing the well-learned recommendation policy. To enable the generalization of action values from experienced states to the unexperienced states with function approximation, we find a proper way to represent the state and action of the pushing agent. Then, we resort to double deep-Q network with dueling architecture to solve the two problems. The simulation results show that the learned recommendation and pushing policies are able to converge and can increase the net profit significantly compared with baseline policies.

Highlights

Content caching at the wireless edge has been acknowledged as a promising way to support the explosively increasing traffic demands and improve user experience [1]–[4]
SIMULATION RESULTS we compare the performance of the learned policy by deep reinforcement learning (DRL) with baseline policies via simulation
WORKS In this paper, we considered joint content pushing and recommendation for mobile users to increase the net profit of mobile network operators (MNOs) without comprising user experience

Summary

Introduction

Content caching at the wireless edge has been acknowledged as a promising way to support the explosively increasing traffic demands and improve user experience [1]–[4]. By caching at base stations (BSs), the traffic load of backhaul and service latency of users can be reduced, and network throughput and energy efficiency can be improved dramatically [5]–[7]. By further proactively pushing the contents into the cache of mobile devices at favorable channel conditions [3], [8], [9], users can enjoy zero latency with low cost of the network if the requested contents are pushed. Due to the huge number of available contents at content providers but limited cache size at each mobile device, the benefit of proactive pushing highly relies on the prediction of user behavior in requesting contents. The efficiency of proactive caching, e.g., the cache-hit ratio, at mobile devices may be unsatisfactory [3]

Objectives

Results

Conclusion