Deep reinforcement learning based scheduling for minimizing age of information in wireless powered sensor networks

Abstract

For real-time monitoring system, the age of information (AoI) is usually used to quantify the freshness of information at a monitor about some stochastic processes observed by the source node. In this paper, we consider the wireless powered sensor networks (WPSNs) where multiple sensor nodes send update packets to the base station. Time is divided into slots of equal duration and at each slot either wireless energy transfer or packet transmission is conducted. We aim to minimize the long-term average weighted sum-AoI of different processes at the base station. Specifically, we first formulate the average weighted sum-AoI minimization problem as a multi-stage stochastic non-linear integer programming (NLP) subject to the energy causality constraints. Second, we design an algorithm which first applies Lyapunov optimization to decouple the multi-stage stochastic NLP into per-frame deterministic NLP problems. Then in each frame, our algorithm utilizes the model-free DRL to solve the per-frame NLP problem with very low computational complexity where one exploration policy is designed to obtain multiple one-hot candidate actions based on single real-number output of neural network. We demonstrate through simulations that, our proposed algorithm can achieve greatly smaller average weighted sum-AoI than the available DQN-based algorithm and also alleviate the problem that some source nodes may have large instantaneous AoIs.