YOLO-MS: Multispectral Object Detection via Feature Interaction and Self-Attention Guided Fusion

Abstract

Object detection is essential for an autonomous driving sensing system. Since the light condition is changed in unconstrained scenarios, the detection accuracy based on visible images can be greatly degraded. Although the detection accuracy can be improved by fusing visible and infrared images, existing multispectral object detection (MOD) algorithms suffer from inadequate inter-modal interaction and a lack of global dependence in the fusion approach. Thus, we propose a multispectral object detection framework called YOLO-MS by designing a Feature Interaction and Self-Attention Fusion Network (FISAFN) as the backbone network. Within the FISAFN, correlations between two modalities are extracted by the Feature Interaction Module (FIM) for reconstructing the information components of each modality and enhancing capability of information exchange. To filter redundant features and enhance complementary features, long-range information dependence between two modalities are established by using a Self-Attention Feature Fusion Module (SAFFM). Thus, a better information richness of the fused features can be achieved. Experimental results on the FLIR-aligned dataset and the MFD dataset demonstrate that the proposed YOLO-MS performs favorably against state-of-the-art approaches, including feature-level fusion and pixel-level fusion. And further, the proposed YOLO-MS possesses good detection performance under diverse scene conditions.