A Transfer Deep Q-Learning Framework for Resource Competition in Virtual Mobile Networks With Energy-Harvesting Base Stations

Abstract

This article considers the problem of resource sharing in a virtual mobile network with energy-harvesting base stations (BSs), where several virtual mobile operators (VMOs) lease radio resources (e.g., radio channels and green BSs) from a mobile network owner (MNO). The VMOs want to provide subscribed users with the best performance while ensuring minimal leasing costs. We aim to find the optimal resource leasing scheme for a VMO in order to maximize utility within the uncertainties of harvested energy, request arrivals, and resource prices. We first formulate the problem as a Markov decision process, during which the VMOs compete with each other for the radio resources by dynamically announcing their resource requirements to the MNO. We, then, employ a deep Q-learning algorithm so the VMO can find the optimal solution by interacting with the environment. With this algorithm, artificial neural networks are used to approximate the Q-value function, which can work effectively with large state and action spaces. We further employ the idea of transfer learning, which exploits the strategies learned in historical periods to speed up the learning process of the target task. Finally, we present comprehensive simulations to evaluate the performance of the proposed scheme under various configurations.