Impact of memory on beamforming optimization for DDPG-assisted RIS-multi-user system

Abstract

Recently, reconfigurable intelligent surfaces (RIS) have garnered considerable attention as indispensable components driving the evolution of future 6G wireless communication systems. This heightened interest is primarily attributed to notable advancements in programmable meta-material fabrication, which enable the creation of highly versatile and adaptable surfaces. These surfaces, often referred to as intelligent reflecting arrays, represent a significant departure from the conventional capabilities of massive multiple-input multiple-output (MIMO) systems, thereby catalyzing the emergence of intelligent radio environments characterized by enhanced flexibility and efficiency. Our study is dedicated to the comprehensive exploration of coordinated design strategies that encompass both the transmission beamforming matrix at the base station and the phase shift matrix at the RIS. Leveraging recent breakthroughs in deep reinforcement learning (DRL), our approach harnesses the power of a Long Short-Term Memory (LSTM) based DRL algorithm. This algorithm orchestrates the joint design process through iterative interactions with the environment, strategically guided by predefined rewards operating within a continuous framework of states and actions. While recent research has underscored the efficacy of LSTM-based architectures in bolstering the learning capacity of reinforcement learning (RL) algorithms and simplifying the search process, our investigation reveals a nuanced insight. Contrary to prevailing trends, we find that the integration of memory into the Deep Deterministic Policy Gradient (DDPG) with LSTM (DDPG-LSTM) algorithm yields unexpected consequences, negatively impacting the system's overall performance. Through preliminary simulation results, we empirically demonstrate the adverse effect of memory on DDPG performance when applied to RIS systems. This finding not only sheds light on the complexities of optimizing 6G wireless communication systems but also underscores the importance of careful algorithmic design and parameter tuning in achieving desired outcomes in emerging technologies.