Cross-modal co-feedback cellular automata for RGB-T saliency detection

Abstract

Saliency cellular automata (CA), a temporally evolving model to efficiently locate salient object, has achieved great progress in saliency detection task. However, most the previous CA models originate from RGB data and are thus limited in some extreme scenes. Inspired by the observation that thermal infrared data (T) can overcome the limitation of RGB data themselves in some cases and RGB-T saliency detection has gained more and more attention. In this paper, we contribute a novel RGB-T saliency detection approach via Cross-modal Co-feedback Cellular Automata (C3A). Before this, we firstly present a novel weighted background-based map (WBM) to give each superpixel(image patch) an initial saliency value. Then C3A is proposed to improve the quality of the WBM. Specifically, it firstly establishes two complementary cellular automata (CA) mechanisms dependent on RGB and thermal infrared data, which respectively refine the WBM based on two different perspectives. To bridge RGB-T modalities, an iterative cross-modal co-feedback framework is contributed to optimize constantly their results. In other words, we regard the result of one modality(RGB or T)-induced CA as important feedback to update and optimize another modality(T or RGB)-induced CA during the iteration. Two modalities constantly pull out the useful and confident data to the opposite side, and so two CAs’ results are constantly refined until a stable state is generated, we then integrate the results of two modalities-induced CAs into the saliency map. Finally, a novel boundary-guided pixel-wise refinement (BPR) technology is proposed to further overcome the influence of inaccurate superpixel segmentation to the C3A and refine the saliency map generated by our C3A. For fairness, the proposed method is compared with other state-of-the-art methods on three RGB-T datasets, experimental results show the superiority of our model.