A general motion control framework for an autonomous underwater vehicle through deep reinforcement learning and disturbance observers

Abstract

This paper investigates the application of deep reinforcement learning (RL) in the motion control for an autonomous underwater vehicle (AUV), and proposes a novel general motion control framework which separates training and deployment. Firstly, the state space, action space, and reward function are customized under the condition of ensuring generality for various motion control tasks. Next, in order to efficiently learn the optimal motion control policy in the case that the AUV model is imprecise and there are unknown external disturbances, a virtual AUV model composed of the known and determined items of an actual AUV is put forward and a simulation training method is developed on this basis. Then, in the given deployment method, three independent extended state observers (ESOs) are designed to deal with the unknown items in different directions, and the final controller is obtained by compensating the estimated value of ESOs into the output of the optimal motion control policy obtained through simulation training. Finally, soft actor-critic is chosen as deep RL algorithm of the framework, and the generality and effectiveness of the proposed method are verified in four different AUV motion control tasks.