Energy-efficient underwater acoustic communication based on Dyna-Q with an adaptive action space

Abstract

The underwater acoustic (UWA) channel exhibits large spatial–temporal dynamics. This work focuses on adapting the transmission power to the channel condition to achieve energy efficiency in UWA communications. To tackle the unknown channel variation, a reinforcement learning (RL) algorithm called Dyna-Q is introduced. Consider the continuous variation of UWA channels. Instead of using a fixed action space in the Dyna-Q, an adaptive action space with an updated Q-value iteration is proposed in this work. The switching of the transmission power level occurs on a block-by-block basis during transmission, aiming to maximize the system’s long-term energy efficiency performance. Using the widely-used communication technique, Orthogonal Frequency-Division Multiplexing (OFDM), as an example, both simulations and experimental data processing results demonstrate that the proposed method outperforms two comparative methods, including the original Dyna-Q with fixed action spaces.