Fusing SAR and EO imagery using CNN RGB-input channels, feature level, and decision level fusion

Abstract

Within the field of target recognition, significant attention is given to data fusion techniques to optimize decision making in systems of multiple sensors. The challenge of fusing synthetic aperture radar (SAR) and electrooptical (EO) imagery is of particular interest to the defense community due to those sensors’ prevalence in target recognition systems. In this paper, the performances of two network architectures (a simple CNN and a ResNet) are compared, each implemented with multiple fusion methods to classify SAR and EO imagery of military targets. The Synthetic and Measured Paired Labeled Experiment (SAMPLE) dataset is used, an expansion of the MSTAR dataset, using both original measured SAR data and synthetic EO data. The classification performance of both networks is compared using the data modalities individually, using feature level fusion, using decision level fusion, and using a novel fusion method based on the three RGB-input channels of the ResNet (or other CNN for color image processing). In the input channel fusion method proposed, SAR imagery is fed to one of the three input channels, and the grayscale EO data is passed to a second of the three input channels. Despite its simplicity and off-the-shelf implementation, the input channel fusion method provides strong results, indicating it is worthy of further study.