Hierarchical Two-stage modal fusion for Triple-modality salient object detection

Abstract

Salient object detection (SOD) is essential for home service robot grasping technology. Recently, studies have shown that using triple-modality (RGB-Depth-Thermal infrared, RGB-D-T) information can significantly improve the detection effect of salient object detection. The location accuracy and completeness of the salient object play a crucial role in the subsequent grasping process. This seriously affects the robot’s judgment of the object position and grasping position. Therefore, the critical problem of triple-modality SOD technology in robot grasping is locating the salient object accurately and detecting the salient object completely. Consequently, we propose a triple-modality salient object detection method based on hierarchical two-stage modal fusion. In the first fusion stage, we use the triple-modal information rationally to locate the salient object accurately. Considering the properties of the different modal information, we use the depth information to supplement and improve the visible light and thermal infrared information through an accurate selection fusion module (ASFM). Then in the second stage, we use the feature correlation enhancement module (FCEM) to realize the correlation of the different modal salient features. FCEM can perfectly combine the different modal information and make the salient object more complete. Comparative and challenging experiments demonstrate that the proposed method outperforms 14 state-of-the-art methods on the VDT2048 dataset. The code is available at: https://github.com/VDT-2048/HTMF.