SK-MMFMNet: A multi-dimensional fusion network of remote sensing images and EEG signals for multi-scale marine target recognition

Abstract

Intelligent recognition of multi-scale marine targets remains pivotal in studying marine resources and transportation. Multi-scale marine target recognition faces challenges such as blurred image, noise interference, varied target sizes, and random target positions. However, these hardly affect the judgment of human brain which could adeptly capture multi-scale targets and disregard noise interference. Therefore, this study proposes an innovative approach to recognize multi-scale marine targets through taking full advantages of the texture, color and structural information provided by remote sensing images and the quick classification ability of human brains, called Selective Kernel & Multi-dimensional Multimodal Data Fusion Module Network (SK-MMFMNet), which fuses remote sensing images and electroencephalography (EEG) signals to improve the accuracy of classifying multi-scale marine targets. In this study, we construct a multi-scale marine target dataset, which includes both remote sensing images of islands, wind turbines, and ships and their corresponding EEG signals from subjects while viewing remote sensing images. Then, the proposed approach extends the Multimodal Transfer Module (MMTM) based on attention mechanism to a dual fusion module across channel and spatial dimensions to fusing MobileNetV3 and EEGNet. Also, we embed the Selective Kernel Module into MobileNetV3 for addressing multi-scale features. The average experimental results across the three multi-scale marine target sub-dataset show that SK-MMFMNet exhibited accuracy improvements of 2.88 %, 21.60 %, and 1.08 %, moreover, F1-Score increments of 24.60 %, 162.22 %, and 14.32 % compared to MobileNetV3, EEGNet, and MMTMNet (MMTM-based fusion network). Visual analysis via Grad-CAM demonstrates that benefiting from EEG signals and Selective Kernel Module, our proposed SK-MMFMNet adjusts the network attention to exactly focus on the multi-scale target area, and thus achieves the best performance. Meanwhile, T-SNE visualization also proves the effectiveness of the three fusion modules and EEG signals for feature extraction. This study offers a valuable and promising insight for intelligent recognition of multi-scale marine targets.