Emotion Recognition Framework Research Articles

A novel dual-modal emotion recognition algorithm with fusing hybrid features of audio signal and speech context

With regard to human–machine interaction, accurate emotion recognition is a challenging problem. In this paper, efforts were taken to explore the possibility to complete the feature abstraction and fusion by the homogeneous network component, and propose a dual-modal emotion recognition framework that is composed of a parallel convolution (Pconv) module and attention-based bidirectional long short-term memory (BLSTM) module. The Pconv module employs parallel methods to extract multidimensional social features and provides more effective representation capacity. Attention-based BLSTM module is utilized to strengthen key information extraction and maintain the relevance between information. Experiments conducted on the CH-SIMS dataset indicate that the recognition accuracy reaches 74.70% on audio data and 77.13% on text, while the accuracy of the dual-modal fusion model reaches 90.02%. Through experiments it proves the feasibility to process heterogeneous information within homogeneous network component, and demonstrates that attention-based BLSTM module would achieve best coordination with the feature fusion realized by Pconv module. This can give great flexibility for the modality expansion and architecture design.

EEG-based cross-subject emotion recognition using Fourier-Bessel series expansion based empirical wavelet transform and NCA feature selection method

Automated emotion recognition using brain electroencephalogram (EEG) signals is predominantly used for the accurate assessment of human actions as compared to facial expression or speech signals. Various signal processing methods have been used for extracting representative features from EEG signals for emotion recognition. However, the EEG signals are non-stationary and vary across the subjects as well as in different sessions of the same subject; hence it exhibits poor generalizability and low classification accuracy for an emotion classification of cross subjects. In this paper, EEG signals-based automated cross-subject emotion recognition framework is proposed using the Fourier-Bessel series expansion-based empirical wavelet transform (FBSE-EWT) method. This method is used to decompose the EEG signals from each channel into four sub-band signals. Manually ten channels are selected from the frontal lobe, from which entropy and energy features are extracted from each sub-band signal. The subject variability is reduced using an average moving filter method on each channel to obtain the smoothened feature vector of size 80. The three feature selection techniques, such as neighborhood component analysis (NCA), relief-F, and mRMR, are used to obtain an optimal feature vector. The machine learning models, such as artificial neural network (ANN), k-nearest neighborhood (k-NN) with two (fine and weighted) functions, and ensemble bagged tree classifiers are trained by the obtained feature vectors. The experiments are performed on two publicly accessible databases, named SJTU emotion EEG dataset (SEED) and dataset for emotion analysis using physiological signals (DEAP). The training and testing of the models have been performed using 10-fold cross-validation and leave-one-subject-out-cross-validation (LOSOCV). The proposed framework based on FBSE-EWT and NCA feature selection approach shows superior results for classifying human emotions compared to other state-of-art emotion classification models.

A novel spatio-temporal convolutional neural framework for multimodal emotion recognition

Proposing a practical method for high-performance emotion recognition could facilitate human–computer interaction. Among existing methods, deep learning techniques have improved the performance of emotion recognition systems. In this work, a new multimodal neural design is presented wherein audio and visual data are combined as the input to a hybrid network comprised of a bidirectional long short term memory (BiLSTM) network and two convolutional neural networks (CNNs). The spatial and temporal features extracted from video frames are fused with Mel-Frequency Cepstral Coefficients (MFCCs) and energy features extracted from audio signals and BiLSTM network outputs. Finally, a Softmax classifier is used to classify inputs into the set of target categories. The proposed model is evaluated on Surrey Audio–Visual Expressed Emotion (SAVEE), Ryerson Audio–Visual Database of Emotional Speech and Song (RAVDESS), and Ryerson Multimedia research Lab (RML) databases. Experimental results on these datasets prove the effectiveness of the proposed model where it achieves the accuracy of 99.75%, 94.99%, and 99.23% for the SAVEE, RAVDESS, and RML databases, respectively. Our experimental study reveals that the suggested method is more effective than existing algorithms in adapting to emotion recognition in these datasets.

A novel framework for facial emotion recognition with noisy and de noisy techniques applied in data pre-processing

A novel framework for facial emotion recognition with noisy and de noisy techniques applied in data pre-processing

Building a three-level multimodal emotion recognition framework

Multimodal emotion detection has been one of the main lines of research in the field of Affective Computing (AC) in recent years. Multimodal detectors aggregate information coming from different channels or modalities to determine what emotion users are expressing with a higher degree of accuracy. However, despite the benefits offered by this kind of detectors, their presence in real implementations is still scarce for various reasons. In this paper, we propose a technology-agnostic framework, HERA, to facilitate the creation of multimodal emotion detectors, offering a tool characterized by its modularity and the interface-based programming approach adopted in its development. HERA (Heterogeneous Emotional Results Aggregator) offers an architecture to integrate different emotion detection services and aggregate their heterogeneous results to produce a final result using a common format. This proposal constitutes a step forward in the development of multimodal detectors, providing an architecture to manage different detectors and fuse the results produced by them in a sensible way. We assessed the validity of the proposal by testing the system with several developers with no previous knowledge about affective technology and emotion detection. The assessment was performed applying the Computer System Usability Questionnaire and the Twelve Cognitive Dimensions Questionnaire, used by The Visual Studio Usability group at Microsoft, obtaining positive results and important feedback for future versions of the system.

Facial Emotion Recognition using Convolutional Neural Networks

Abstract: Humans use their facial expressions to communicate their emotions, which is a strong tool in communication. Facial expression identification is one of the most difficult and powerful challenges in social communication, as facial expressions are crucial in nonverbal communication. Facial Expression Recognition (FER) is just an important study topic in Artificial Intelligence, with numerous recent experiments employing Convolutional Neural Networks (CNNs). The emotions that have grown in the face image have a significant impact on judgments and debates on a variety of topics. Surprise, fear, disgust, anger, happiness, and sorrow are the six basic categories in which a person's emotional states can be categorized according to psychological theory. The automated identification of these emotions from facial photos can be useful in human-computer interaction and a variety of other situations. Deep neural networks, in particular, are capable of learning complicated characteristics and classifying the derived patterns. A deep learning-based framework for human emotion recognition is offered in this system. The proposed framework extracts feature with Gabor filters before classifying them with a Convolutional Neural Network (CNN). The suggested technique improves both the speed of CNN training and the recognition accuracy, according to the results of the experiments. Keywords: convolutional neural network (CNN), Gabor filter

FACIAL EMOTION RECOGNITION

Facial Emotion is the noticeable indication of the full of affective state, cognitive activity, intension, personality, and psychopathology of an individual and plays a communicative role in interpersonal relations. Automatic recognition of facial emotion can be a significant part of normal human-machine interfaces; it might likewise be utilized in conduct science and in clinical practice. An automatic Facial Emotion Recognition framework requirement to perform identification and area of countenances in a jumbled scene, facial component extraction, and look characterization. The facial emotion recognition framework is carried out utilizing the Deep Convolution Neural Network (DCNN). The CNN model of the undertaking depends on LeNet Architecture. Kaggle Facial Expression Dataset (FER-2013) with seven facial feelings named as happy, sad, surprise, fear, anger, disgust, and neutral is utilized in this venture. The framework accomplished 65% ± 5% accuracy.

LEMON: A Lightweight Facial Emotion Recognition System for Assistive Robotics Based on Dilated Residual Convolutional Neural Networks

The development of a Social Intelligence System based on artificial intelligence is one of the cutting edge technologies in Assistive Robotics. Such systems need to create an empathic interaction with the users; therefore, it os required to include an Emotion Recognition (ER) framework which has to run, in near real-time, together with several other intelligent services. Most of the low-cost commercial robots, however, although more accessible by users and healthcare facilities, have to balance costs and effectiveness, resulting in under-performing hardware in terms of memory and processing unit. This aspect makes the design of the systems challenging, requiring a trade-off between the accuracy and the complexity of the adopted models. This paper proposes a compact and robust service for Assistive Robotics, called Lightweight EMotion recognitiON (LEMON), which uses image processing, Computer Vision and Deep Learning (DL) algorithms to recognize facial expressions. Specifically, the proposed DL model is based on Residual Convolutional Neural Networks with the combination of Dilated and Standard Convolution Layers. The first remarkable result is the few numbers (i.e., 1.6 Million) of parameters characterizing our model. In addition, Dilated Convolutions expand receptive fields exponentially with preserving resolution, less computation and memory cost to recognize the distinction among facial expressions by capturing the displacement of the pixels. Finally, to reduce the dying ReLU problem and improve the stability of the model, we apply an Exponential Linear Unit (ELU) activation function in the initial layers of the model. We have performed training and evaluation (via one- and five-fold cross validation) of the model with five datasets available in the community and one mixed dataset created by taking samples from all of them. With respect to the other approaches, our model achieves comparable results with a significant reduction in terms of the number of parameters.

DEEMD-SPP: A Novel Framework for Emotion Recognition Based on EEG Signals.

Electroencephalography (EEG) is one of the most widely-used biosignal capturing technology for investigating brain activities, cognitive diseases, and affective disorders. To understand the underlying principles of brain activities and affective disorders using EEG data, one of the fundamental tasks is to accurately identify emotions from EEG signals, which has attracted huge attention in the field of affective computing. To improve the accuracy and effectiveness of emotion recognition based on EEG data, previous studies have successfully developed numerous feature extraction methods and classifiers. Among them, ensemble empirical mode decomposition (EEMD) is an efficient signal decomposition technique for extracting EEG features. It can alleviate the mode-mixing problem by adding white noise to the source signal. However, there remain some issues when applying this method to recognition tasks. As the added noise cannot be filtered completely, spurious modes are generated due to the residual noise. Therefore, it is crucial to perform intrinsic mode function (IMF) selection to find the most valuable IMF components that represent brain activities. Furthermore, the number of decomposed IMFs is various to different original signals, thus how to unify feature dimensions needs better solutions. To solve these issues, we propose a novel forecasting framework, named DEEMD-SPP, to identify emotions from EEG signals, based on the combination of denoising ensemble empirical mode decomposition (DEEMD) and Spatial Pyramid Pooling Network (SPP-Net). First, DEEMD is proposed to decompose the EEG signals, which effectively eliminates residual noise in the IMFs and selects the most valuable IMFs. Second, time-domain and frequency-domain features are extracted from the selected IMFs. Finally, SPP-net is employed as the classifier to recognize emotions, which can effectively transform various-sized feature maps into fixed-sized feature vectors through the pyramid pooling layer. The experimental results demonstrate that our proposed DEEMD-SPP framework can effectively reduce the effect of spike-in white noise, accurately extract EEG features, and significantly improve the performance of emotion recognition.

EmoSeC: Emotion recognition from scene context

Context provides additional information to determine the actual emotional state of a person as part of a scene. Existing works on emotion recognition in context focused only on the features extracted from the entire image and the target’s body. In this work, we propose a comprehensive multi-cue based emotion recognition framework that incorporates the context, using a hybrid architecture comprised of four separate deep convolutional neural networks and a novel feature fusion mechanism. Each deep network presented in our proposed approach effectively learns the emotion-related features from the facial, body pose, non-target subject and entire image, individually. Experiments on Emotic, an in-painted and a newly constructed EmoSec datasets show that our proposed emotion recognition framework is promising when compared to existing methods in terms of accurately classifying emotions. Comparison with the state-of-the-art deep networks and off-the-shelf fusion techniques demonstrates that our network showed an improved performance. Furthermore, the performance evaluation of our proposed approach on all three datasets confirms that the contextual information influences the emotional state of a human.

Spontaneous Speech Emotion Recognition Using Multiscale Deep Convolutional LSTM

Recently, emotion recognition in real sceneries such as in the wild has attracted extensive attention in affective computing, because existing spontaneous emotions in real sceneries are more challenging and difficult to identify than other emotions. Motivated by the diverse effects of different lengths of audio spectrograms on emotion identification, this paper proposes a multiscale deep convolutional long short-term memory (LSTM) framework for spontaneous speech emotion recognition. Initially, a deep convolutional neural network (CNN) model is used to learn deep segment-level features on the basis of the created image-like three channels of spectrograms. Then, a deep LSTM model is adopted on the basis of the learned segment-level CNN features to capture the temporal dependency among all divided segments in an utterance for utterance-level emotion recognition. Finally, different emotion recognition results, obtained by combining CNN with LSTM at multiple lengths of segment-level spectrograms, are integrated by using a score-level fusion strategy. Experimental results on two challenging spontaneous emotional datasets, i.e., the AFEW5.0 and BAUM-1s databases, demonstrate the promising performance of the proposed method, outperforming state-of-the-art methods.

A New Hybrid Approach for Efficient Emotion Recognition using Deep Learning

Facial emotion recognition has been very popular area for researchers in last few decades and it is found to be very challenging and complex task due to large intra-class changes. Existing frameworks for this type of problem depends mostly on techniques like Gabor filters, principle component analysis (PCA), and independent component analysis(ICA) followed by some classification techniques trained by given videos and images. Most of these frameworks works significantly well image database acquired in limited conditions but not perform well with the dynamic images having varying faces and images. In the past years, various researches have been introduced framework for facial emotion recognition using deep learning methods. Although they work well, but there is always some gap found in their research. In this research, we introduced hybrid approach based on RNN and CNN which are able to retrieve some important parts in the given database and able to achieve very good results on the given database like EMOTIC, FER-13 and FERG. We are also able to show that our hybrid framework is able to accomplish promising accuracies with these datasets.

Depression Assessment Method: An EEG Emotion Recognition Framework Based on Spatiotemporal Neural Network.

The main characteristic of depression is emotional dysfunction, manifested by increased levels of negative emotions and decreased levels of positive emotions. Therefore, accurate emotion recognition is an effective way to assess depression. Among the various signals used for emotion recognition, electroencephalogram (EEG) signal has attracted widespread attention due to its multiple advantages, such as rich spatiotemporal information in multi-channel EEG signals. First, we use filtering and Euclidean alignment for data preprocessing. In the feature extraction, we use short-time Fourier transform and Hilbert–Huang transform to extract time-frequency features, and convolutional neural networks to extract spatial features. Finally, bi-directional long short-term memory explored the timing relationship. Before performing the convolution operation, according to the unique topology of the EEG channel, the EEG features are converted into 3D tensors. This study has achieved good results on two emotion databases: SEED and Emotional BCI of 2020 WORLD ROBOT COMPETITION. We applied this method to the recognition of depression based on EEG and achieved a recognition rate of more than 70% under the five-fold cross-validation. In addition, the subject-independent protocol on SEED data has achieved a state-of-the-art recognition rate, which exceeds the existing research methods. We propose a novel EEG emotion recognition framework for depression detection, which provides a robust algorithm for real-time clinical depression detection based on EEG.

A Multimodal Framework for Large-Scale Emotion Recognition by Fusing Music and Electrodermal Activity Signals

Considerable attention has been paid to physiological signal-based emotion recognition in the field of affective computing. For reliability and user-friendly acquisition, electrodermal activity (EDA) has a great advantage in practical applications. However, EDA-based emotion recognition with large-scale subjects is still a tough problem. The traditional well-designed classifiers with hand-crafted features produce poorer results because of their limited representation abilities. And the deep learning models with auto feature extraction suffer the overfitting drop-off because of large-scale individual differences. Since music has a strong correlation with human emotion, static music can be involved as the external benchmark to constrain various dynamic EDA signals. In this article, we make an attempt by fusing the subject’s individual EDA features and the external evoked music features. And we propose an end-to-end multimodal framework, the one-dimensional residual temporal and channel attention network (RTCAN-1D). For EDA features, the channel-temporal attention mechanism for EDA-based emotion recognition is first involved in mine the temporal and channel-wise dynamic and steady features. The comparisons with single EDA-based SOTA models on DEAP and AMIGOS datasets prove the effectiveness of RTCAN-1D to mine EDA features. For music features, we simply process the music signal with the open-source toolkit openSMILE to obtain external feature vectors. We conducted systematic and extensive evaluations. The experiments on the current largest music emotion dataset PMEmo validate that the fusion of EDA and music is a reliable and efficient solution for large-scale emotion recognition.

A dual framework for implicit and explicit emotion recognition: An ensemble of language models and computational linguistics

One of the research domains in the field of sentiment analysis is automatic emotion recognition in texts which is a worthy topic in human-computer interaction. Text processing has always faced many challenges. The main one is the structural and semantic differences of sentences which have had a significant impact on the malfunction of auto-recognition systems. This problem becomes more prominent in short texts in which words and their concurrences are limited and insufficient. As a result of this, word frequency and TF-IDF weighing cannot well represent the relationship between words and the appropriate feature vector, leading to an undesirable accuracy of emotion recognition. Thus, different strategies should be applied to improve the feature vector and to formulate the features properly. The desired strategy should be able to identify the words that can distinguish between classes well and also to find the relationships between words and meaningful phrases using natural language processing concepts. In this paper, a combination of emotional models, categorical and hierarchical, are used for an emotional text recognition which could discover simultaneously explicit and implicit emotion in a short text. Our approach called DuFER, proposed a weighed method which improves the feature vector using language models and computational linguistics through applying a modified TF-IDF weighing to words as well as Maximum Likelihood Estimation weighing to expressions. Four implicit and explicit emotion datasets are used for the experiments. The results show that the accuracy of both implicit and explicit emotion recognition has increased and DuFER is actually the first successful dual framework in recognizing implicit and explicit emotions from text.

An integrated framework for emotion recognition using speech and static images with deep classifier fusion approach

An integrated framework for emotion recognition using speech and static images with deep classifier fusion approach

A multiturn complementary generative framework for conversational emotion recognition

Conversational emotion recognition (CER) is a significant task due to its application in human–computer interaction. Existing work treats CER as an utterance-level classification task without considering that empathic response also reflects contextual emotion understanding. Previous work has proven that accurate recognition of emotions in the dialogue history is helpful to generate high-fit responses. In this paper, we investigate whether this conclusion is a sufficient and necessary condition. Specifically, we define an auxiliary empathic multiturn dialogue generation (MDG) task to enhance emotion understanding. Correspondingly, we present a Sequence-to-Sequence oriented framework that combines CER and MDG in a multitask learning manner to verify the complementarity between the two tasks. First, we use alternate recurrent neural networks to encode the content of historical utterances and represent the states of multiparty emotions, which are used for emotion classification. Second, since most MDG methods ignore the emotional coherence of the dialogue context itself, we use affine transformation to fuse hidden states of content and emotions to initialize the decoder. Finally, at each step of generation, an attention mechanism is used to fuse information from the dialogue history to ensure emotional coherence. The CER results of our models outperform the state-of-the-art on three prevalent emotional dialogue data sets. Further analysis demonstrates the mutual promotion and empathy interpretability between MDG and CER. Furthermore, our framework is scalable for different coding strategies and multimodal fusion. To the best of our knowledge, this is the first work to explore CER from the perspective of empathy through multitask learning with dialogue generation.

Emotion recognition of EEG signals based on variational mode decomposition and weighted cascade forest.

Emotion recognition is of a great significance in intelligent medical treatment and intelligent transportation. With the development of human-computer interaction technology, emotion recognition based on Electroencephalogram (EEG) signals has been widely concerned by scholars. In this study, an EEG emotion recognition framework is proposed. Firstly, variational mode decomposition (VMD) is used to decompose the nonlinear and non-stationary EEG signals to obtain intrinsic mode functions (IMFs) at different frequencies. Then sliding window tactic is used to extract the characteristics of EEG signals under different frequency. Aiming at the issue of feature redundancy, a new variable selection method is proposed to improve the adaptive elastic net (AEN) by the minimum common redundancy maximum relevance criterion. Weighted cascade forest (CF) classifier is constructed for emotion recognition. The experimental results on the public dataset DEAP show that the valence classification accuracy of the proposed method reaches 80.94%, and the classification accuracy of arousal is 74.77%. Compared with some existing methods, it effectively improves the accuracy of EEG emotion recognition.

Consistency Regularization을 적용한 멀티모달 한국어 감정인식

Recently, the demand for artificial intelligence-based voice services, identifying and appropriately responding to user needs based on voice, is increasing. In particular, technology for recognizing emotions, which is non-verbal information of human voice, is receiving significant attention to improve the quality of voice services. Therefore, speech emotion recognition models based on deep learning is actively studied with rich English data, and a multi-modal emotion recognition framework with a speech recognition module has been proposed to utilize both voice and text information. However, the framework with speech recognition module has a disadvantage in an actual environment where ambient noise exists. The performance of the framework decreases along with the decrease of the speech recognition rate. In addition, it is challenging to apply deep learning-based models to Korean emotion recognition because, unlike English, emotion data is not abundant. To address the drawback of the framework, we propose a consistency regularization learning methodology that can reflect the difference between the content of speech and the text extracted from the speech recognition module in the model. We also adapt pre-trained models with self-supervised way such as Wav2vec 2.0 and HanBERT to the framework, considering limited Korean emotion data. Our experimental results show that the framework with pre-trained models yields better performance than a model trained with only speech on Korean multi-modal emotion dataset. The proposed learning methodology can minimize the performance degradation with poor performing speech recognition modules.

LEDPatNet19: Automated Emotion Recognition Model based on Nonlinear LED Pattern Feature Extraction Function using EEG Signals

Electroencephalography (EEG) signals collected from human brains have generally been used to diagnose diseases. Moreover, EEG signals can be used in several areas such as emotion recognition, driving fatigue detection. This work presents a new emotion recognition model by using EEG signals. The primary aim of this model is to present a highly accurate emotion recognition framework by using both a hand-crafted feature generation and a deep classifier. The presented framework uses a multilevel fused feature generation network. This network has three primary phases, which are tunable Q-factor wavelet transform (TQWT), statistical feature generation, and nonlinear textural feature generation phases. TQWT is applied to the EEG data for decomposing signals into different sub-bands and create a multilevel feature generation network. In the nonlinear feature generation, an S-box of the LED block cipher is utilized to create a pattern, which is named as Led-Pattern. Moreover, statistical feature extraction is processed using the widely used statistical moments. The proposed LED pattern and statistical feature extraction functions are applied to 18 TQWT sub-bands and an original EEG signal. Therefore, the proposed hand-crafted learning model is named LEDPatNet19. To select the most informative features, ReliefF and iterative Chi2 (RFIChi2) feature selector is deployed. The proposed model has been developed on the two EEG emotion datasets, which are GAMEEMO and DREAMER datasets. Our proposed hand-crafted learning network achieved 94.58%, 92.86%, and 94.44% classification accuracies for arousal, dominance, and valance cases of the DREAMER dataset. Furthermore, the best classification accuracy of the proposed model for the GAMEEMO dataset is equal to 99.29%. These results clearly illustrate the success of the proposed LEDPatNet19.