Intelligent navigation method for multiple marine autonomous surface ships based on improved PPO algorithm

Abstract

To achieve the autonomous navigation of multiple Marine Autonomous Surface Ship (MASS), an intelligent planning and decision-making method for MASS based on Rapidly-exploring Random Trees star (RRT-star) and improved Proximal Policy Optimization (PPO) algorithm is proposed. The novelty of the study is: (1) enhancing the traditional PPO algorithm by the Generalized Advantage Estimation (GAE) and the Long Short-Term Memory (LSTM) network, which facilitate accelerated convergence of average reward, predict the state space, and enhance accuracy in estimating the advantage function. (2) proposing a complete reward function, which can not only guide MASS to navigate towards the waypoint, but also ensure that MASS complies with the COLREGs during collision avoidance. It is worth highlighting that the trained network model can be generalized to different scenarios. We attached the trained neural network model to different MASS and simulated it in different open and narrow waters. Even in emergency situations, the method can still make it deviate from COLREGs and make flexible collision avoidance decisions. The results show that this method can handle the multi-MASS encounter situation well and make MASS reach the waypoint safely.