Reinforcement Learning for Motion Policies in Mobile Relaying Networks

Abstract

We consider joint beamforming and relay motion control in mobile relay beamforming networks, operating in a spatio-temporally varying channel environment. A time slotted approach is adopted, where in each slot, the relays implement optimal beamforming and estimate their optimal positions for the next slot. We place the problem of relay motion control in a sequential decision-making framework. We employ Reinforcement Learning (RL) to guide the relay motion, with the goal of maximizing the cumulative Signal-to-Interference+Noise Ratio (SINR) at the destination. First, we present a model based RL approach, which predictively estimates the SINR and accordingly determines the relay motion, based on partial knowledge of the channel model along with channel measurements at the current relay positions. Second, we propose a model-free deep Q-learning approach, which does not rely on channel models. For the deep Q-learning approach, we propose two modified Multilayer Perceptron Neural Networks (MLPs) for approximating the value function Q. The first modification applies a Fourier feature mapping of the state before passing it through the MLP. The second modification constitutes a different neural network architecture that uses sinusoids as activations between layers. Both modifications enable the MLP to better learn the high frequency value function and have a profound effect on convergence speed and SINR performance. Finally, we conduct a comparative analysis of all the presented approaches and provide insights on advantages and drawbacks.