ABSTRACT One of the main challenges that faces the monitoring of maritime structures and ports is related to constantly changing weather and surrounding conditions. These conditions make manual actions very expensive, unreliable and imprecise. To overcome these problems, using Autonomous Surface Vehicles (ASV) is the best option. In this research, as part of a research project, a stereo vision system was implemented on an ASV to detect and track static and dynamic obstacles. In order to examine the proposed algorithm, two sets of experiments have been designed; first, the ASV moves toward stationary objects to localise all static and dynamic objects. Second, the ASV tracks a target boat, which equipped with an RTK-GPS and estimates relative and absolute positions. The accuracy of the tracking algorithm was evaluated in the digital elevation map, with the lowest RMSE of 1.575, and 0.6621 (m) using raw and EKF data, respectively.