Comprehensive comparative evaluation of background subtraction algorithms in open sea environments

Abstract

In autonomous-ship and maritime security surveillance operations involving electro-optical sensors, the first phase of foreground segmentation and change detection using background subtraction (BS) algorithms is crucial. However, it is also the most complex in terms of execution time. Despite the development of several BS algorithms, maritime foreground segmentation and change detection remain major challenges owing to the complex, unconstrained, and diverse nature of ocean scenarios. However, only a few studies have investigated the applications of BS algorithms in maritime environments, especially those involving boats in the open sea. This study compares BS methods involving use of a non-static electro-optical sensor in combination with visible-light and infrared cameras to identify the best method for use in open sea scenarios, especially from the viewpoint of avoiding piracy and armed robbery. Thirty-seven methods, ranging from simple temporal differencing to more sophisticated ones, were validated via extensive experiments and analyses using realistic maritime datasets and practical maritime applications. In addition, because most methods considered in this study were not previously evaluated at the pixel level on open sea datasets, this paper proposes an appropriate maritime BS benchmark, based on which the 37 methods were compared to compensate for their prior lack of detailed analyses. The experimental results indicate that BS algorithms of the multiple features category can better handle maritime challenges, thereby realizing higher accuracies when analyzing visible-light and thermal videos. The proposed evaluation, therefore, complements those reported previously. Consequently, the proposed study enables users to identify the most suitable BS algorithm for use in intelligent maritime transportation, maritime security surveillance systems, and autonomous ships in the open sea.