Multi-modal Physiological Signal Fusion for Emotion Classification: A Multi-Head Attention Approach

Abstract

In this essay, a model-level fusion technique of multi-modal physiological signals using Multi-Head Attention is studied. A framework that utilizes multi-model physiological signals for the task of emotion classification is proposed. First, the GCRNN model, which combines the Graph Convolutional Network (GCN) and the Long and Short Term Memory (LSTM), captures the unique features of electroencephalogram (EEG) signals. The spatial and temporal information that makes up impulses from the EEG can be captured precisely by such a technique. The CCRNN model, which combines the Convolutional Neural Network (CNN) integrated with the Channel-wise Attention and the LSTM, is used for peripheral physiological signals. The model can extract useful features from peripheral physiological signals and automatically learn to weigh the importance of various channels. Finally, Multi-head Attention is employed to fuse the output of the GCRNN and CCRNN methods. The Multi-head Attention can automatically learn the relevance and importance of different modal signals and weigh them accordingly. Emotion classification is implemented by adding a component of Softmax to map what the model produced to discrete emotion categories. The DEAP dataset was utilized in this study for experimental verification, and the results indicate that the method using multi-modal physiological signal fusion is substantially greater in precision than the technique using simply EEG signals. Additionally, the Multi-head Attention fusion method performs better than previous fusion techniques.