Markovian-Jump Reinforcement Learning for Autonomous Underwater Vehicles under Disturbances with Abrupt Changes

Abstract

This paper studies the position regulation problems of an Autonomous Underwater Vehicle (AUV) subject to external disturbances that may have abrupt variations due to some events, e.g., water flow hitting nearby underwater structures. The disturbing forces may frequently exceed the actuator capacities, necessitating a constrained optimization of control inputs over a future time horizon. However, the AUV dynamics and the parameters of the disturbance models are unknown. Estimating the Markovian processes of the disturbances is challenging since it is entangled with uncertainties from AUV dynamics. As opposed to a single-Markovian description, this paper formulates the disturbed AUV as an unknown Markovian-Jump Linear System (MJLS) by augmenting the AUV state with the unknown disturbance state. Based on an observer network and an embedded solver, this paper proposes a reinforcement learning approach, Disturbance-Attenuation-net (MDA–net), for attenuating Markovian-jump disturbances and stabilizing the disturbed AUV. MDA–net is trained based on the sensitivity analysis of the optimality conditions and is able to estimate the disturbance and its transition dynamics based on observations of AUV states and control inputs online. Extensive numerical simulations of position regulation problems and preliminary experiments in a tank testbed have shown that the proposed MDA–net outperforms the existing DOB–net and a classical approach, Robust Integral of Sign of Error (RISE).