Expression EEG Multimodal Emotion Recognition Method Based on the Bidirectional LSTM and Attention Mechanism.

Yifeng Zhao,Deyun Chen,Kaijian Xia

doi:10.1155/2021/9967592

Yifeng Zhao, Deyun Chen + Show 1 more

Open Access

https://doi.org/10.1155/2021/9967592

Copy DOI

Abstract

Due to the complexity of human emotions, there are some similarities between different emotion features. The existing emotion recognition method has the problems of difficulty of character extraction and low accuracy, so the bidirectional LSTM and attention mechanism based on the expression EEG multimodal emotion recognition method are proposed. Firstly, facial expression features are extracted based on the bilinear convolution network (BCN), and EEG signals are transformed into three groups of frequency band image sequences, and BCN is used to fuse the image features to obtain the multimodal emotion features of expression EEG. Then, through the LSTM with the attention mechanism, important data is extracted in the process of timing modeling, which effectively avoids the randomness or blindness of sampling methods. Finally, a feature fusion network with a three-layer bidirectional LSTM structure is designed to fuse the expression and EEG features, which is helpful to improve the accuracy of emotion recognition. On the MAHNOB-HCI and DEAP datasets, the proposed method is tested based on the MATLAB simulation platform. Experimental results show that the attention mechanism can enhance the visual effect of the image, and compared with other methods, the proposed method can extract emotion features from expressions and EEG signals more effectively, and the accuracy of emotion recognition is higher.

Highlights

With the rapid development of computer and artificial intelligence technology, computer intelligence is paid more and more attention
For the intrinsic bioelectric signals such as EEG and ECG, there is no intuitive expression of emotional information such as voice signals, the emotion recognition system built by bioelectric signals is more reliable because the bioelectric signals are not easy to be modified by subjective manipulation
Facial expression and EEG features are extracted based on bilinear convolution network (BCN), and the attention mechanism is introduced into the LSTM network

Summary

Introduction

With the rapid development of computer and artificial intelligence technology, computer intelligence is paid more and more attention. Based on the above analysis, an expression electroencephalogram multimodal affective recognition method based on the bidirectional LSTM and attention mechanism is proposed to solve the problems of the single mode, which are difficulty of deeply extracting signal characteristics and low accuracy of affective recognition. The proposed strategy based on BCN extracts facial expression features, converts the EEG signal into three bands of image sequences, and uses BCN to fuse its image features to obtain more accurate expression EEG multimodal affective features (2) In order to avoid the randomness or blindness of sampling methods, the attention mechanism is introduced in the LSTM network, and a characteristic fusion network with a three-layer bidirectional LSTM structure is designed to combine emoticons and EEG characteristics to improve the accuracy of emotion recognition

Related Works

Feature Extraction Based on the Bilinear Convolution Network

Experimental Scheme and Result Discussion

Findings

Conclusion