Subject wise data augmentation based on balancing factor for quaternary emotion recognition through hybrid deep learning model

Abstract

An electroencephalogram (EEG) identifies neuronal activity as electrical currents produced by a group of specialized pyramidal cells within the brain due to synchronized activity. EEG signal contains essential information about brain activity and is often used to measure the enthusiasm of individual behavioural feelings and polarity. Recently, the researcher focused on classifying EEG emotions from multimodal DEAP data sets into binary (high and low) and ternary (low, medium and high) classes based on valence and arousal scales. However, for deep and intrinsic emotion recognition, multiclass classification is preferred. But due to the limited data sample and improper class distribution, multiclass classification is more challenging. This paper proposes a data augmentation-based hybrid deep learning model to classify EEG signals over four different emotional stages, i.e. happy (H), relaxed (R), anger (A) and sad (S). The hybrid model encapsulates a convolutional neural network (CNN) and bi-directional long short-term memory (Bi-LSTM) to recognize EEG samples’ emotions. Furthermore, the proposed model performs segmentation to enhance EEG data samples, and SMOTE (Synthetic minority oversampling technique) is committed to achieving alleviated skewness in EEG samples among four emotion classes. The experiments are conducted on all the data of 32 subjects before and after processing the data augmentation algorithm on the DEAP data. The average improvement in precision, recall, F1-score, accuracy, and specificity is 6.02%, 9.44%, 7.98%, 6.01%, and 2.66% respectively after data augmentation, and statistically varies with the balancing factor.