Saving Energy and Spectrum in Enabling URLLC Services: A Scalable RL Solution

Abstract

Communication systems supporting cyber-physical production applications should satisfy stringent delay and reliability requirements. Diversity techniques and power control are the main approaches to reduce latency and enhance the reliability of wireless communications at the expense of redundant transmissions and excessive resource usage. Focusing on the application layer reliability key performance indicators (KPIs), we design a deep reinforcement learning orchestrator for power control and hybrid automatic repeat request retransmissions to optimize these KPIs. Furthermore, to address the scalability issue that emerges in the per-device orchestration problem, we develop a new branching soft actor-critic framework in which a separate branch represents the action space of each industrial device. Our orchestrator enables near real-time control and can be implemented in the edge cloud. We test our solution with a 3GPP-compliant and realistic simulator for factory automation scenarios. Compared to the state-of-the-art, our solution offers significant scalability gains in terms of computational time and memory requirements. Our extensive experiments show significant improvements in our target KPIs, over the state-of-the-art, especially for 5th percentile user availability. To achieve these targets, our framework requires substantially less total energy or spectrum, thanks to our scalable RL solution.