Enhancing Deep-Learning Object Detection Performance Based on Fusion of Infrared and Visible Images in Advanced Driver Assistance Systems

Abstract

Image recognition technology plays an important role in advanced driver assistance systems (ADAS). The objective of this study is to explore the feasibility of using heterogeneous image fusion to improve the object detection performance of the ADAS. Among the many possible combinations of image types, the fusion of infrared (IR) and visible (VIS) images has great potential because of their complementary characteristics. Most studies on image fusion assume that the images involved align themselves perfectly, which is unrealistic. We address this alignment issue in this study, review various methods of image alignment and fusion, and propose an image-fusion approach that combines alignment and fusion methods for the ADAS application. Finally, we used deep learning networks to detect pedestrian and vehicle objects before and after the image fusion. The experimental results show that the fusion of IR and VIS images can improve the object detection performance of deep-learning networks. Compared with previous studies on fusion, the proposed approach ranks top if the detection accuracy improvement and execution speed are considered as a whole. This study also found that, to use image fusion to improve the object detection accuracy of deep learning networks, it is better to use fused images directly instead of unfused VIS images as the training samples.