Sample Intercorrelation-Based Multidomain Fusion Network for Aquatic Human Activity Recognition Using Millimeter-Wave Radar

Abstract

To address the issue that existing multi-domain fusion methods do not consider data correlation within mini-batch data, the first attempt is made in this letter to propose a method based on sample inter-correlation learning multi-domain fusion network (SIMFNet), which aims to accommodate multivariate domain data and further enhance the aquatic human activity recognition performance. To fully utilize the radar multidimensional information, the three-branch convolution neural network (CNN) feature extractor is first employed to extract domain-specific features from the time-range map (TRM), time-Doppler map (TDM) and cadence velocity diagram (CVD). Then, the multi-domain features are fused and fed into the graph construction layer (GCL) to generate instance graphs. Next, a graph aggregation layer (GAL) is applied to aggregate node information from various-hop neighborhood domains. Finally, node-level classification is used to achieve aquatic human activity recognition. The experimental results evaluated on the built aquatic human activity recognition dataset demonstrate that the proposed SIMFNet has better generalization performance than the state-of-the-art multi-domain fusion methods.