Dynamics-Aligned Transfer Reinforcement Learning For Autonomous Underwater Vehicle Control

Abstract

Autonomous Underwater Vehicles (AUVs) in shallow waters usually are subject to model uncertainties from loading and unloading objects, thruster malfunction due to thruster caught by waterweed, etc. Learning-based controllers are suitable for uncertainty attenuation control. But excessive uncertainties heavily affect the state transition in Markov Decision Process (MDP) or Partially Observable Markov Decision Process (POMDP). In addition, learning procedures on real AUV system requires a large number of training samples, which may damage systems. Therefore, a policy trained exclusively on a simulated model usually results in unsatisfactory performance on a real AUV system due to dynamics model mismatch (transition model mismatch). In this paper, we propose a transfer reinforcement learning approach that adapts a control policy to an underwater robot under dynamics model mismatch, for the purpose of uncertainty rejection. In particular, A policy trained on a source dynamics model is transferred to target dynamics models via dynamics alignment, by proposing a distance metric between two dynamics models. We have tested the proposed approach on a pose regulation task through numerical simulations of three scenarios, mainly including changes in thruster characteristics. The results have demonstrated the effectiveness of the proposed dynamics-aligned transfer reinforcement learning approach.