Abstract
Emotion recognition plays an important role in intelligent human–computer interaction, but the related research still faces the problems of low accuracy and subject dependence. In this paper, an open-source software toolbox called MindLink-Eumpy is developed to recognize emotions by integrating electroencephalogram (EEG) and facial expression information. MindLink-Eumpy first applies a series of tools to automatically obtain physiological data from subjects and then analyzes the obtained facial expression data and EEG data, respectively, and finally fuses the two different signals at a decision level. In the detection of facial expressions, the algorithm used by MindLink-Eumpy is a multitask convolutional neural network (CNN) based on transfer learning technique. In the detection of EEG, MindLink-Eumpy provides two algorithms, including a subject-dependent model based on support vector machine (SVM) and a subject-independent model based on long short-term memory network (LSTM). In the decision-level fusion, weight enumerator and AdaBoost technique are applied to combine the predictions of SVM and CNN. We conducted two offline experiments on the Database for Emotion Analysis Using Physiological Signals (DEAP) dataset and the Multimodal Database for Affect Recognition and Implicit Tagging (MAHNOB-HCI) dataset, respectively, and conducted an online experiment on 15 healthy subjects. The results show that multimodal methods outperform single-modal methods in both offline and online experiments. In the subject-dependent condition, the multimodal method achieved an accuracy of 71.00% in the valence dimension and an accuracy of 72.14% in the arousal dimension. In the subject-independent condition, the LSTM-based method achieved an accuracy of 78.56% in the valence dimension and an accuracy of 77.22% in the arousal dimension. The feasibility and efficiency of MindLink-Eumpy for emotion recognition is thus demonstrated.
Highlights
Emotions are biological states associated with the nervous system (Damasio, 1998), and its changes are related to subjective feelings and objective behavioral responses
We proposed two methods, weight enumerator and Adaboost, to improve the emotion recognition accuracy
We proposed another approach for subject-independent emotion recognition to make it suitable for more users
Summary
Emotions are biological states associated with the nervous system (Damasio, 1998), and its changes are related to subjective feelings and objective behavioral responses. - Two approaches of facial expression and EEG for emotion recognition - Two decision-level fusion methods for fusion of sub-classifiers in different modalities to improve accuracy - Reads, processes, visualizes multimodal real-time data (facial images and EEG signals) and stores data into a folder system - Subject-independent emotion recognition approach based on LSTM in EEG modality. In addition to data processing methods, MindLink-Eumpy provides emotion classification methods, including the multitask CNN method in the facial image toolbox, the SVM and an LSTM in the EEG toolbox, and two methods in the fusion toolbox for decision-level fusion More details of these methods are shown in Methods for Emotion Recognition section. MindLink-Eumpy provides machine learning algorithms and deep learning techniques with promising performance based on both accuracy and algorithm complexity These accurate methods are based on multimodal emotion recognition, and subject independence is achieved with singlemodal EEG data. Our toolbox incorporates database, experimental paradigm, and software tools, which allow developers to extend model functionality and optimize usability in collaborative development
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have