Secrecy Rate Maximization in THz-Aided Heterogeneous Networks: A Deep Reinforcement Learning Approach

Abstract

Densely deployed femtocells in heterogeneous networks (HetNets) improve the quality-of-service (QoS) and quality-of-experience (QoE) for both next-generation networks and end users, respectively. However, users near the femtocell edge, i.e., femto edge users (FEUs), which are far away from the Femto base station (FBS) may receive a low signal-to-interference-plus-noise ratio (SINR). Moreover, due to the distributed and dynamic nature of HetNets, femtocells may not resist various eavesdropping attacks from different attackers. Thus, it is essential to ensure the wireless channel's secrecy through which FEUs communicate with FBS since they may face eavesdropping attacks and may have low SINR and achievable data rates. Aiming to enhance the secrecy rate of femtocells, in this article, we formulate a joint optimization problem of beamforming vectors, artificial noise (AN), and power control for terahertz (THz)-enabled femtocells. Addressing this non-convex optimization problem is challenging due to the system's complexity and dynamic nature of FEUs. Thus, we translate the optimization problem into a multi-agent reinforcement learning (MARL) problem using the Markov decision process (MDP). Then, to solve the MDP with continuous action space, two deep reinforcement learning (DRL) based schemes, i.e., <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Deterministic Policy Gradient-based Secrecy Maximization (DDPG-SM)</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Asynchronous-Advantage-Actor-Critic based Secrecy Maximization (A3C-SM)</i> are proposed to maximize the secrecy rate of femtocells. The secrecy rate performance of the proposed schemes is compared and analyzed with Advantage-Actor-Critic (A2C) and state-of-the-art Perfect Secrecy Rate Maximization (PSRM) schemes. Simulation results show that the proposed <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> schemes achieve 37.73% and 22.64% better average secrecy rate in comparison to the PSRM scheme. Also, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A3C-SM</i> scheme outperforms all other <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DDPG-SM</i> , A2C, and PSRM schemes in terms of average secrecy rate, received SINR, and energy-efficiency of femtocells.