Proposed Emotion Recognition System Research Articles

Smart Classroom Monitoring Using Novel Real-Time Facial Expression Recognition System

Emotions play a vital role in education. Technological advancement in computer vision using deep learning models has improved automatic emotion recognition. In this study, a real-time automatic emotion recognition system is developed incorporating novel salient facial features for classroom assessment using a deep learning model. The proposed novel facial features for each emotion are initially detected using HOG for face recognition, and automatic emotion recognition is then performed by training a convolutional neural network (CNN) that takes real-time input from a camera deployed in the classroom. The proposed emotion recognition system will analyze the facial expressions of each student during learning. The selected emotional states are happiness, sadness, and fear along with the cognitive–emotional states of satisfaction, dissatisfaction, and concentration. The selected emotional states are tested against selected variables gender, department, lecture time, seating positions, and the difficulty of a subject. The proposed system contributes to improve classroom learning.

ViTFER: Facial Emotion Recognition with Vision Transformers

In several fields nowadays, automated emotion recognition has been shown to be a highly powerful tool. Mapping different facial expressions to their respective emotional states is the main objective of facial emotion recognition (FER). In this study, facial expression recognition (FER) was classified using the ResNet-18 model and transformers. This study examines the performance of the Vision Transformer in this task and contrasts our model with cutting-edge models on hybrid datasets. The pipeline and associated procedures for face detection, cropping, and feature extraction using the most recent deep learning model, fine-tuned transformer, are described in this study. The experimental findings demonstrate that our proposed emotion recognition system is capable of being successfully used in practical settings.

Normal Inverse Gaussian Features for EEG-Based Automatic Emotion Recognition

Electroencephalography (EEG)-based emotion recognition is crucial in the domain of Human-Computer Interaction (HCI), which gained significant attention in recent years. However, the non-stationarity and chaotic nature of the EEG signals pose challenges and restrict the state-of-the-art techniques from precisely identifying distinct emotional states from the EEG data and hence offer limited emotion recognition performance. To capture the underlying non-linear characteristics of EEG, this study employed a novel local mean decomposition (LMD) algorithm which decomposes EEG signals into product functions (PFs). Further PFs are modeled by Normal Inverse Gaussian (NIG) probability density function (PDF) parameters. Thus these PDF features are fed to an optimized Adaboost classifier developed with the help of a cross-validation approach. The novelty of the work lies in the NIG modeling of LMD domain PFs to identify specific emotions from the EEG signals. The significance of the NIG parameters is illustrated by qualitative, pictorial, and statistical analyses. To assess the efficiency of the proposed approach, intensive experiments are conducted on open-source datasets, SJTU Emotion EEG Dataset (SEED), SEED-IV, and Database for Emotion Analysis of Physiological Signals (DEAP). The emotion recognition performance is evaluated in terms of heat maps, receiver operating characteristics (ROC), and accuracy. The proposed emotion recognition system outperformed the state-of-art methods and achieved a maximum accuracy of 97.3%, 98%, and 98.6% with the cross-validation approach and 93.23%, 94.87%, and 95.58% for the cross-subject validation approach using DEAP and SEED, SEED-IV datasets, respectively.

MEC-Enabled Hierarchical Emotion Recognition and Perturbation-Aware Defense in Smart Cities

With the explosive growth of Internet of Things (IoT) devices and various emerging network technologies, IoT-enabled smart cities are further refined into health smart cities. For example, IoT devices can automatically recognize emotional states through collected facial expressions, which can further serve mental health assessment, human–computer interaction, etc. On the other hand, existing facial expression recognition algorithms emphasize the application of deep neural networks (DNNs), and it is difficult for resource-constrained IoT devices to provide sufficient computing resources to optimize parameters for DNN-based structures. To solve the challenge of resource constraints, we propose the hierarchical emotion recognition system enabled by mobile edge computing (MEC). Specifically, MEC nodes provide IoT devices with short-delay and high-performance computing services, satisfying the requirements of training DNN-based algorithms. Moreover, our proposed emotion recognition system leverages a pretrained feature extraction module on the remote cloud to accelerate optimization and provides a localization module for specific tasks of IoT devices. In addition to evaluating the accuracy and efficiency, we also clarify that the DNN-based emotion recognition system exposes obvious vulnerability to perturbation. Due to the uncertainty of the environment, it is common for facial expressions collected by IoT devices to be accompanied by perturbation. To address this issue, we propose the proactive perturbation-aware defense mechanism. It has been demonstrated that the newly proposed defense mechanism can maintain state-of-the-art performance on the publicly available LIRIS-CSE dataset while defending against known and unknown perturbation. This can promote the deployment of our proposed MEC-enabled hierarchical emotion recognition system and defense mechanism in real-world scenarios.

An Examination of Emotion Recognition using Machine Learning Algorithms on Different Speech Databases

The speech recognition system plays a vital role in understanding the emotions of natural language. The identification of emotions from speech is a challenging task. The performance of the speech recognition system is effects on the speech signals. The speech contains different emotions feelings. Many researchers introduced different emotion recognition techniques. However, these techniques achieved better performance but unsatisfied in identify emotion of natural languages. This paper proposed a novel speech recognition system, which identify the emotions based on the speech signals.The Mel Frequency Cepstral Coefficients (MFCC) features. On the resultant features of speech applied crossvalidation using the test emotions. The performance of the proposed system verify with the SVM and other two classifiers. The proposed emotion recognition system achieves better performance. The empirical results shows that the proposed system outperforms when compare with different classifiers and databases.

EEG-based emotion recognition using simple recurrent units network and ensemble learning

The purpose of this research is to develop an EEG-based emotion recognition system for identification of three emotions: positive, neutral and negative. Up to now, various modeling approaches for automatic emotion recognition have been reported. However, the time dependency property during emotion process has not been fully considered. In order to grasp the temporal information of EEG, we adopt deep Simple Recurrent Units (SRU) network which is not only capable of processing sequence data but also has the ability to solve the problem of long-term dependencies occurrence in normal Recurrent Neural Network (RNN). Before training the emotion models, Dual-tree Complex Wavelet Transform (DT-CWT) was applied to decompose the original EEG into five constituent sub-bands, from which features were then extracted using time, frequency and nonlinear analysis. Next, deep SRU models were established using four different features over five frequency bands and favorable results were found to be related to higher frequency bands. Finally, three ensemble strategies were employed to integrate base SRU models to get more desirable classification performance. We evaluate and compare the performance of shallow models, deep models and ensemble models. Our experimental results demonstrated that the proposed emotion recognition system based on SRU network and ensemble learning could achieve satisfactory identification performance with relatively economic computational cost.

Emotion Recognition From Body Movement

Automatic emotion recognition from the analysis of body movement has tremendous potential to revolutionize virtual reality, robotics, behavior modeling, and biometric identity recognition domains. A computer system capable of recognizing human emotion from the body can also significantly change the way we interact with the computers. One of the significant challenges is to identify emotion-specific features from a vast number of descriptors of human body movements. In this paper, we introduce a novel two-layer feature selection framework for emotion classification from a comprehensive list of body movement features. We used the feature selection framework to accurately recognize five basic emotions: happiness, sadness, fear, anger, and neutral. In the first layer, a unique combination of Analysis of Variance (ANOVA) and Multivariate Analysis of Variance (MANOVA) was utilized to eliminate irrelevant features. In the second layer, a binary chromosome-based genetic algorithm was proposed to select a feature subset from the relevant list of features that maximizes the emotion recognition rate. Score and rank-level fusion were applied to further improve the accuracy of the system. The proposed system was validated on proprietary and public datasets, containing 30 subjects. Different action scenarios, such as walking and sitting actions, as well as an action-independent case, were considered. Based on the experimental results, the proposed emotion recognition system achieved a very high emotion recognition rate outperforming all of the state-of-the-art methods. The proposed system achieved recognition accuracy of 90.0% during walking, 96.0% during sitting, and 86.66% in an action-independent scenario, demonstrating high accuracy and robustness of the developed method.

Emotion state identification based on heart rate variability and genetic algorithm.

The objective of this study is to develop an effective emotion recognition system based on ECG. The proposed emotion recognition system is capable of differentiating four kinds of emotions, namely neutral, happiness, stress, and sadness, based on the heart rate variability (HRV). Ten male subjects were involved in the study. Both visual and auditory stimuli were used to stimulate the emotions. Four categories of HRV features, namely time-domain, frequency-domain, Poincare plot, and differential features, were exploited to characterize the physiological changes during the affective stimuli. The support vector machine (SVM) was employed as the classifier. The genetic algorithm (GA) was exploited as feature selector. Without feature selector, only 52.2% recognition rate was achieved. However, with the GA feature selector, an optimal recognition rate of 90% was achieved. Compared with other user-independent systems published in the literature, the proposed method achieves an accuracy of 90% which is demonstrated to be the most effective for discriminating four kinds of emotions with user-independent design policy.

Classification of Music-Induced Emotions Based on Information Fusion of Forehead Biosignals and Electrocardiogram

Emotion recognition systems have been developed to assess human emotional states during different experiences. In this paper, an approach is proposed for recognizing music-induced emotions through the fusion of three-channel forehead biosignals (the left temporalis, frontalis, and right temporalis channels) and an electrocardiogram. The classification of four emotional states in an arousal–valence space (positive valence/low arousal, positive valence/high arousal, negative valence/high arousal, and negative valence/low arousal) was performed by employing two parallel support vector machines as arousal and valence classifiers. The inputs of the classifiers were obtained by applying a fuzzy-rough model feature evaluation criterion and sequential forward floating selection algorithm. An average classification accuracy of 88.78 % was achieved, corresponding to an average valence classification accuracy of 94.91 % and average arousal classification accuracy of 93.63 %. The proposed emotion recognition system may be useful for interactive multimedia applications or music therapy.

Recognition of emotions from video using neural network models

In this paper, facial features from the video sequence are explored for characterizing the emotions. The emotions considered for this study are Anger, Fear, Happy, Sad and Neutral. For carrying out the proposed emotion recognition study, the required video data is collected from the studio, Center for Education Technology (CET), at Indian Institute of Technology (IIT) Kharagpur. The dynamic nature of the grey values of the pixels within the eye and mouth regions are used as the features to capture the emotion specific knowledge from the facial expressions. Multiscale morphological erosion and dilation operations are used to extract features from eye and mouth regions, respectively. The features extracted from left eye, right eye and mouth regions are used to develop the separate models for each emotion category. Autoassociative neural network (AANN) models are used to capture the distribution of the extracted features. The developed models are validated using subject dependent and independent emotion recognition studies. The overall performance of the proposed emotion recognition system is observed to be about 87%.

A Hybrid Speech Emotion Recognition System Based on Spectral and Prosodic Features

In this paper, we present a hybrid speech emotion recognition system exploiting both spectral and prosodic features in speech. For capturing the emotional information in the spectral domain, we propose a new spectral feature extraction method by applying a novel non-uniform subband processing, instead of the mel-frequency subbands used in Mel-Frequency Cepstral Coefficients (MFCC). For prosodic features, a set of features that are closely correlated with speech emotional states are selected. In the proposed hybrid emotion recognition system, due to the inherently different characteristics of these two kinds of features (e.g., data size), the newly extracted spectral features are modeled by Gaussian Mixture Model (GMM) and the selected prosodic features are modeled by Support Vector Machine (SVM). The final result of the proposed emotion recognition system is obtained by combining the results from these two subsystems. Experimental results show that (1) the proposed non-uniform spectral features are more effective than the traditional MFCC features for emotion recognition; (2) the proposed hybrid emotion recognition system using both spectral and prosodic features yields the relative recognition error reduction rate of 17.0% over the traditional recognition systems using only the spectral features, and 62.3% over those using only the prosodic features.