MFENet: Multitype fusion and enhancement network for detecting salient objects in RGB-T images

Abstract

Recent progress in salient object detection (SOD) has been fueled substantially by the development of convolutional neural networks. However, several SOD methods do not fully exploit information from different modalities, consequently performing only marginally better than methods using a single modality. Therefore, we propose a multitype fusion and enhancement network (MFENet), following three steps “Encoder- Pre-decoder- Decoder” for RGB-thermal (RGB-T) SOD by completely exploiting the advantages of the RGB and thermal modalities through feature integration and enhancement. To better fuse two modalities' features, we have designed the cross-modality fusion module (CMFM) in the encoder part. As shallow features describe details and deep features provide semantic information, a multiscale interactive refinement module is designed in the pre-decoder part to complement multilevel features. Additionally, to further sharpen salient objects, we have proposed a high-level, low-level module that takes inputs from adjacent layers for gradual translation into a saliency map in the decoder part. This module provides semantic information for shallower features and the boundaries of salient objects can be gradually sharpened with subtle details. Extensive experiments show the effectiveness and robustness of the proposed MFENet and its substantial improvement over state-of-the-art RGB-T SOD methods. The codes and results will be available at: https://github.com/wujunyi1412/MFENet_DSP.