Experience-driven learning-based intelligent hybrid beamforming for massive MIMO mmWave communications

Abstract

We have studied massive MIMO hybrid beamforming (HBF) for millimeter-wave (mmWave) communications, where the transceivers only have a few radio frequency chain (RFC) numbers compared to the number of antenna elements. We propose a hybrid beamforming design to improve the system’s spectral, hardware, and computational efficiencies, where finding the precoding and combining matrices are formulated as optimization problems with practical constraints. The series of analog phase shifters creates a unit modulus constraint, making this problem non-convex and subsequently incurring unaffordable computational complexity. Advanced deep reinforcement learning techniques effectively handle non-convex problems in many domains; therefore, we have transformed this non-convex hybrid beamforming optimization problem using a reinforcement learning framework. These frameworks are solved using advanced deep reinforcement learning techniques implemented with experience replay schemes to maximize the spectral and learning efficiencies in highly uncertain wireless environments. We developed a twin-delayed deep deterministic (TD3) policy gradient-based hybrid beamforming scheme to overcome Q-learning’s substantial overestimation. We assumed a complete channel state information (CSI) to design our beamformers and then challenged this assumption by proposing a deep reinforcement learning-based channel estimation method. We reduced hybrid beamforming complexity using soft target double deep Q-learning to exploit mmWave channel sparsity. This method allowed us to construct the analog precoder by selecting channel dominant paths. We have demonstrated that the proposed approaches improve the system’s spectral and learning efficiencies compared to prior studies. We have also demonstrated that deep reinforcement learning is a versatile technique that can unleash the power of massive MIMO hybrid beamforming in mmWave systems for next-generation wireless communication.