Decentralized Reinforcement Learning-Based Access Control for Energy Sustainable Underwater Acoustic Sub-Network of MWCN

Abstract

Due to energy limitations, inadequate frequency band, mobility of underwater nodes, the most challenging issue of the next generation MWCN is the access control of the underwater acoustic sub-network. In this chapter, we focus on an Energy Sustainable Underwater acoustic sub-network of MWCN with tidal energy harvesting. For simplicity, we use the term ESUN for the energy sustainable underwater sub-network of MWCN in the later sections. Specifically, an analytical model is first developed to analyze the network performance of ESUN, characterizing the stochastic nature of energy harvesting and traffic demands of ESUN nodes, and the salient features of acoustic communication channels. It is found that the spatial uncertainty resulting from underwater acoustic communication may cause severe fairness issue. As such, an optimization problem is formulated to maximize the network throughput under fairness constraints by tuning the random-access parameters of each node. Given the global network information, including the number of nodes, energy harvesting rates, communication distances, etc., the optimization problem can be efficiently solved using the Branch and Bound (BnB) method. Considering a realistic network where the full network information may not be available at the ESUN nodes, we further propose a multi-agent reinforcement learning approach for each node to autonomously adapt the random-access parameter based on the interactions with the dynamic network environment. Numerical results show that the proposed learning algorithm outperforms the existing solutions in terms of the network throughput and approaches the derived theoretical bound.