Deep Reinforcement Learning Based MAC Protocol for Underwater Acoustic Networks

Abstract

Long propagation delay that causes throughput degradation of underwater acoustic networks (UWANs) is a critical issue in the medium access control (MAC) design in UWANs. In this paper, we put forth a new deep reinforcement learning (DRL) algorithm called delayed-reward deep Q-network (DR-DQN) to solve this problem. In particular, we propose a new multiple access protocol, referred to as delayed-reward DRL multiple access (DR-DLMA). The system objective is to find an optimal channel access strategy that maximizes the network throughput, by making full use of the available time-slots resulted from the long propagation delay or not used by other nodes in the network. We also give the upper bound on the network throughput in various cases as our benchmark, and show that the DR-DLMA can achieve a near-optimal performance with all the different propagation delays. Furthermore, we evaluate the performance of the proposed protocol in heterogeneous and homogeneous networks, respectively. When coexisting with one FW-ALOHA node, the maximum gap between the network throughput achieved by the DR-DLMA and the optimal solution is within 10%. In homogeneous networks, we demonstrate that DR-DLMA outperforms traditional MAC protocols for UWANs (e.g., slotted FAMA and DOTS) in terms of the network throughput. Particularly, in the four-node network with a maximum propagation delay of 4, DR-DLMA yields 470% and 190% higher throughput on average than slotted FAMA and DOTS, respectively.