Spatial-Temporal Feature Fusion Neural Network for EEG-Based Emotion Recognition

Abstract

The temporal and spatial information of electroencephalogram (EEG) are essential for the emotion recognition model to learn the discriminative features. Hence, we propose a novel hybrid spatial-temporal feature fusion neural network (STFFNN) to extract the discriminative features and integrate complementary information. The generated power topographic maps, which capture dependencies among the electrodes, are fed to convolutional neural network (CNN) for spatial feature learning. Furthermore, instance normalizations (INs) and batch normalizations (BNs) within the CNN are appropriately combined to alleviate the individual difference and preserve the domain-invariant information. Meanwhile, a feedforward network is adopted for temporal feature learning. Due to the high dimensionality of EEG features, we propose a grid-search-based configurational optimization method to robustly reduce the dimensionality. Finally, inspired by the multimodal fusion strategies that leverage the complementarity of data to obtain more robust predictions, we utilize a bidirectional long short-term memory (Bi-LSTM) network for temporal and spatial feature fusion. To validate the effectiveness of the proposed method, the tenfold cross-validation experiments and subject-dependent experiments are both conducted on the DEAP database. The experimental results demonstrate that the proposed method achieves outstanding performance in emotion recognition with arousal and valence level.