Hybrid offline-online reinforcement learning for obstacle avoidance in autonomous underwater vehicles

Abstract

ABSTRACT This study presents a novel control framework for autonomous underwater vehicles (AUVs) that integrates offline and online reinforcement learning to enhance navigation accuracy and obstacle avoidance. Recognizing the limitations of online reinforcement learning due to high interaction demands and the challenges of offline learning from suboptimal data, we construct kinematic and dynamic models of AUVs as a foundation for our control strategies. Our controller employs a fusion strategy, utilizing offline datasets generated via the Model Predictive Path Integral (MPPI) method and exploration rewards based on kernel density estimation (KDE) to improve exploration of low-confidence areas. Extensive simulations demonstrate the effectiveness of our approach in complex scenarios, including navigation to multiple targets, power insufficiencies, water flow interference, and dynamic obstacles. The trained agents exhibited superior navigation accuracy and obstacle avoidance, underscoring the practicality of our combined learning methods for robust AUV performance in intricate environments.