Heading Control of a Ship Based on Deep Reinforcement Learning (RL)

Abstract

A deep reinforcement learning (RL) algorithm called Deep Q-Network(DQN) is used for the heading control of a ship in calm water and in waves. Ship’s state space is in continuous space and a set of five rudder angle actions are in discrete space. The optimal rudder action is selected based on maximum Q-value of the rudder actions. The ship positions and velocities serve as input and the Q-values associated with a set of rudder angles are the output of the DQN. Reward functions are designed such that the agent will try to reduce the Cross Track Error (CTE) and Heading Error (HE). The heading control of a KVLCC2 tanker in calm water and waves is investigated. The ship dynamics is represented using a 3DoF numerical model. The CTE and HE are calculated based on Line of Sight (LOS) Algorithm.