Illumination-Guided RGBT Object Detection With Inter- and Intra-Modality Fusion

Abstract

Robust object detection is hindered by various illumination conditions in real-world applications. Common practice introduces thermal modality to augment the detection capability of RGB images in poor illumination conditions. However, a major challenge for such work is how to leverage the complementary information of RGB and thermal images effectively. In this paper, we tackle this by combining the strengths of illumination-guided inter- and intra-modality information in both modalities. Specifically, we propose an effective object detection network dubbed TINet to adaptively fuse the complementary features extracted from RGB and thermal images. Firstly, we design an illumination-guided feature weighting module to guide the network in leaning towards a reliable modality. Then, an inter-modality attention module is developed to amplify and complement differential features between these two modalities. We further design an intra-modality attention module using object heatmap prediction to enhance the foreground features of each modality. Finally, we fuse the inter- and intra-modality features for object detection with illumination-guided feature weights. Experimental results demonstrate the effectiveness of the proposed method on the FLIR-aligned dataset and the KAIST multispectral pedestrian detection dataset.