A general motion control architecture for an autonomous underwater vehicle with actuator faults and unknown disturbances through deep reinforcement learning

Abstract

This paper studies the application of deep reinforcement learning (RL) in the motion control tasks of an underactuated AUV with actuator faults and unknown disturbances. Firstly, a general state space and action space that can be applied to a variety of motion control tasks are customized. Furthermore, a universal reward function is designed to minimize the energy consumption of the AUV under the condition that the learned optimal control policy can achieve fast and high-precision control. Then, a simulation method with random reference values and simulated random disturbances is given to train the deep RL algorithm. In order to deploy the optimal control policy obtained from the simulation training directly to an actual AUV, we design five extended state observers (ESOs) to estimate the unknown disturbances in different directions, and take the estimated values as the disturbance states required to obtain the optimal control action. Combined with these ideas, a general AUV motion control architecture with simulation training and deployment process is developed. Finally, four different AUV motion control experiments are carried out, and the results confirm the generality and effectiveness of our architecture.