Delay-Aware Content Delivery With Deep Reinforcement Learning in Internet of Vehicles

Abstract

The rapid development of the Internet of Vehicles (IoV) enables various vehicular applications, such as image-aided navigation and traffic information management. It is important to provide efficient content delivery services for these vehicular applications. Caching popular content at roadside units (RSUs) is a promising way to improve content delivery efficiency. However, due to RSUs with limited cache space, it is very challenging to develop an effective content delivery policy that satisfies the high quality of service (QoS) requirements for vehicular applications. In this paper, we investigate the user-centric content delivery problem with service delay constraints in the IoV, where the objective is to minimize the vehicle’s cost under usage-based pricing. The problem of finding an optimal content delivery policy is modeled as a finite-horizon Markov decision process (MDP). Since the cache state of each RSU, and the wireless channel qualities between the vehicle and RSUs, are usually unknown to the vehicle a priori, the vehicle must learn the optimal delivery policy by interacting with the environment. To solve this problem and optimize the vehicle’s cost, we propose a double deep Q network (DDQN)-based algorithm, which implements dynamic content delivery decisions. Furthermore, the double deep Q network can overcome the large-scale state space and reduce Q value over-estimation. Numerical results show that our policy achieves a near-optimal performance when compared to the optimal policy that knows precisely cache state and wireless channel state. We also compare the effects of different caching strategies and vehicle mobility on the performance of the algorithm.