Reinforcement-Learning-Based Spatial Resource Identification for IoT D2D Communications

Abstract

The exponential growth of the Internet of Things (IoTs) has led to an increasing demand for intelligent IoT devices (IoTDs), requiring innovative network capacity expansion. Recently, several research has been conducted on the identification of hidden network resources for network capacity expansion. However, the spatial resource identification scheme through the omnidirectional antenna has limitations in terms of frequency efficiency compared to the scheme with the directional antenna. In this article, we propose a directional spatial-resource identification technique for device-to-device (D2D) communication. To find the optimal identification parameters, we design the objective function and apply a reinforcement learning. The training data used for reinforcement learning are collected in each report phase, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$Q$</tex-math></inline-formula> -learning is applied to find the optimal beam set. Furthermore, based on the obtained frequency information, we propose a contention-based D2D communication scheme. The proposed contention-based D2D communication scheme can efficiently solve the deafness problem occurring in a directional D2D communication. Finally, we perform a simulation using optimum network performance (OPNET) to measure the performance and evaluate the effectiveness of the proposed technique. The simulation results show that the proposed schemes realize a better performance than the existing schemes proposed in previous works in terms of energy efficiency, frequency efficiency, aggregate network throughput, and deafness duration.