Fish Tracking and Segmentation From Stereo Videos on the Wild Sea Surface for Electronic Monitoring of Rail Fishing

Abstract

Electronic monitoring of fishery activities has drawn increasing attention. Deformable objects, noise from the wild sea surface, and dynamic background, however, make conventional tracking and segmentation methods unreliable. In this paper, we present an online 3D tracking and segmentation system for stereo video-based monitoring of rail fish catching on the wild sea surface. Based on the result of a pre-trained image object (fish) detector, a Kalman filtering-based tracking system overcomes the issues of low detection scores of deformed objects and of unreliable bounding boxes by rescoring multiple object proposals using spatial information in 3D. A clustering-and-scoring strategy is then applied on the depth map so that a plane classification method can effectively segment the objects from the dynamic background without any prior modeling. The object segmentation is further refined using fully connected conditional random fields based on color and geometric features. Using the segmentation results, we can measure the 3D lengths of objects and update the positions of bounding boxes to help tracking. Experimental results show that a reliable tracking and measurement performance under noisy and dynamic sea surface environment can be achieved.