Collaborative spatial-temporal video salient object detection with cross attention transformer

Abstract

Noticing the effectiveness of explicit motion encoding in optical flow, a variety of recent works employ flow-guided two-branch structures to handle the video salient object detection (VSOD) task. However, most of them ignore the semantic gap between the moving objects and the salient objects. Besides, the long-range dependency is not sufficiently investigated due to the local convolution operation. To tackle these problems, in this paper, we propose a novel collaborative spatio-temporal feature fusion method based on the cross-attention transformer for VSOD. A Siamese feature extractor is introduced to jointly optimize the motion and static feature extraction with reduced network parameters. The deep level set method is employed to segment the moving objects from optical flow, and thus the dependence on saliency groundtruth is greatly reduced. Moreover, a cross-attention transformer is proposed to jointly optimize and fuse the static and motion features, as well as investigate long-range dependency. Experimental results on six commonly used video salient object datasets demonstrate that our method achieves state-of-the-art performance among all VSOD algorithms.