A multispectral feature fusion network for robust pedestrian detection

Abstract

The multispectral information, including both visible information and infrared information, can describe the detection target in a comprehensive manner. The deep learning (DL)-based detectors that fuse the multispectral features can detect pedestrians robustly in various environments. Therefore, this paper puts forward a robust multispectral feature fusion network (MSFFN) for pedestrian detection, which fully integrates the features extracted from visible light and infrared channels. Specifically, multiscale semantic features were extracted by two core modules, namely, multiscale feature extraction of visible images (MFEV) and multiscale feature extraction of infrared images (MFEI), and fused by the improved YOLOv3 network for pedestrian recognition. Through experiments on the KAIST dataset, it is proved that the MSFFN model can detect pedestrians more accurately than both MFEV and MFEI over daytime and nighttime images on multiple scales. The experimental results on the KAIST multispectral data set in the last section showed that our proposed MFMFN model was superior to a number of state-of-the-art multispectral pedestrian detectors methods in accuracy and speed. The model was also found to strike a good balance between accuracy and speed, and perform excellently on small input images. The research results shed important new light on the design of self-driving vehicles.