Human Emotion Recognition Research Articles

Towards secure private and trustworthy human-centric embedded machine learning: An emotion-aware facial recognition case study

The use of artificial intelligence (AI) at the edge is transforming every aspect of the lives of human beings from scheduling daily activities to personalized shopping recommendations. Since the success of AI is to be measured ultimately in terms of how it benefits human beings, and that the data driving the deep learning-based edge AI algorithms are intricately and intimately tied to humans, it is important to look at these AI technologies through a human-centric lens. However, despite the significant impact of AI design on human interests, the security and trustworthiness of edge AI applications are not foolproof and ethicalneither foolproof nor ethical; Moreover, social norms are often ignored duringin the design, implementation, and deployment of edge AI systems. In this paper, we make the following two contributions: Firstly, we analyze the application of edge AI through a human-centric perspective. More specifically, we present a pipeline to develop human-centric embedded machine learning (HC-EML) applications leveraging a generic human-centric AI (HCAI) framework. Alongside, we also analyzediscuss the privacy, trustworthiness, robustness, and security aspects of HC-EML applications with an insider look at their challenges and possible solutions along the way. Secondly, to illustrate the gravity of these issues, we present a case study on the task of human facial emotion recognition (FER) based on AffectNet dataset, where we analyze the effects of widely used input quantization on the security, robustness, fairness, and trustworthiness of an EML model. We find that input quantization partially degrades the efficacy of adversarial and backdoor attacks at the cost of a slight decrease in accuracy over clean inputs. By analyzing the explanations generated by SHAP, we identify that the decision of a FER model is largely influenced by features such as eyes, alar crease, lips, and jaws. Additionally, we note that input quantization is notably biased against the dark skin faces, and hypothesize that low-contrast features of dark skin faces may be responsible for the observed trends. We conclude with precautionary remarks and guidelines for future researchers.

Human Emotions Recognition, Analysis and Transformation by the Bioenergy Field in Smart Grid Using Image Processing

The passage of electric signals throughout the human body produces an electromagnetic field, known as the human biofield, which carries information about a person’s psychological health. The human biofield can be rehabilitated by using healing techniques such as sound therapy and many others in a smart grid. However, psychiatrists and psychologists often face difficulties in clarifying the mental state of a patient in a quantifiable form. Therefore, the objective of this research work was to transform human emotions using sound healing therapy and produce visible results, confirming the transformation. The present research was based on the amalgamation of image processing and machine learning techniques, including a real-time aura-visualization interpretation and an emotion-detection classifier. The experimental results highlight the effectiveness of healing emotions through the aforementioned techniques. The accuracy of the proposed method, specifically, the module combining both emotion and aura, was determined to be ~88%. Additionally, the participants’ feedbacks were recorded and analyzed based on the prediction capability of the proposed module and their overall satisfaction. The participants were strongly satisfied with the prediction capability (~81%) of the proposed module and future recommendations (~84%). The results indicate the positive impact of sound therapy on emotions and the biofield. In the future, experimentation using different therapies and integrating more advanced techniques are anticipated to open new gateways in healthcare.

An Improved Face-Emotion Recognition to Automatically Generate Human Expression With Emoticons

Any human face image expression naturally identifies expressions of happy, sad etc.; sometimes human facial image expression recognition is complex, and it is a combination of two emotions. The existing literature provides face emotion classification and image recognition, and the study on deep learning using convolutional neural networks (CNN), provides face emotion recognition most useful for healthcare and with the most complex of the existing algorithms. This paper improves the human face emotion recognition and provides feelings of interest for others to generate emoticons on their smartphone. Face emotion recognition plays a major role by using convolutional neural networks in the area of deep learning and artificial intelligence for healthcare services. Automatic facial emotion recognition consists of two methods, such as face detection with Ada boost classifier algorithm and emotional classification, which consists of feature extraction by using deep learning methods such as CNN to identify the seven emotions to generate emoticons.

Humans read emotional arousal in monkey vocalizations: evidence for evolutionary continuities in communication.

Humans and other mammalian species communicate emotions in ways that reflect evolutionary conservation and continuity, an observation first made by Darwin. One approach to testing this hypothesis has been to assess the capacity to perceive the emotional content of the vocalizations of other species. Using a binary forced choice task, we tested perception of the emotional intensity represented in coos and screams of infant and juvenile female rhesus macaques (Macaca mulatta) by 113 human listeners without, and 12 listeners with, experience (as researchers or care technicians) with this species. Each stimulus pair contained one high- and one low-arousal vocalization, as measured at the time of recording by stress hormone levels for coos and the degree of intensity of aggression for screams. For coos as well as screams, both inexperienced and experienced participants accurately identified the high-arousal vocalization at significantly above-chance rates. Experience was associated with significantly greater accuracy with scream stimuli but not coo stimuli, and with a tendency to indicate screams as reflecting greater emotional intensity than coos. Neither measures of empathy, human emotion recognition, nor attitudes toward animal welfare showed any relationship with responses. Participants were sensitive to the fundamental frequency, noisiness, and duration of vocalizations; some of these tendencies likely facilitated accurate perceptions, perhaps due to evolutionary homologies in the physiology of arousal and vocal production between humans and macaques. Overall, our findings support a view of evolutionary continuity in emotional vocal communication. We discuss hypotheses about how distinctive dimensions of human nonverbal communication, like the expansion of scream usage across a range of contexts, might influence perceptions of other species' vocalizations.

Applying Self-Supervised Representation Learning for Emotion Recognition Using Physiological Signals

The use of machine learning (ML) techniques in affective computing applications focuses on improving the user experience in emotion recognition. The collection of input data (e.g., physiological signals), together with expert annotations are part of the established standard supervised learning methodology used to train human emotion recognition models. However, these models generally require large amounts of labeled data, which is expensive and impractical in the healthcare context, in which data annotation requires even more expert knowledge. To address this problem, this paper explores the use of the self-supervised learning (SSL) paradigm in the development of emotion recognition methods. This approach makes it possible to learn representations directly from unlabeled signals and subsequently use them to classify affective states. This paper presents the key concepts of emotions and how SSL methods can be applied to recognize affective states. We experimentally analyze and compare self-supervised and fully supervised training of a convolutional neural network designed to recognize emotions. The experimental results using three emotion datasets demonstrate that self-supervised representations can learn widely useful features that improve data efficiency, are widely transferable, are competitive when compared to their fully supervised counterparts, and do not require the data to be labeled for learning.

HUMAN EMOTION RECOGNITION SYSTEM USING DEEP LEARNING ALGORITHMS

The subject of research in the article is the software implementation of a neural image classifier. The work examines emotions as a special type of mental processes that express a person’s experience of his attitude to the surrounding world and himself. They can be expressed in different ways: facial expressions, posture, motor reactions, voice. However, the human face has the greatest expressiveness. Technologies for recognizing companies to improve customer service use human emotions make decisions about interviewing candidates and optimize the emotional impact of advertising. Therefore, the purpose of the work is to find and optimize the most satisfactory in terms of accuracy algorithm for classifying human emotions based on facial images. The following tasks are solved: review and analysis of the current state of the problem of "recognition of emotions"; consideration of classification methods; choosing the best method for the given task; development of a software implementation for the classification of emotions; conducting an analysis of the work of the classifier, formulating conclusions about the work performed, based on the received data. An image classification method based on a densely connected convolutional neural network is also used. Results: the results of this work showed that the method of image classification, based on a densely connected convolutional neural network, is well suited for solving the problems of emotion recognition, because it has a fairly high accuracy. The quality of the classifier was evaluated according to the following metrics: accuracy; confusion matrix; precision, recall, f1-score; ROC curve and AUC values. The accuracy value is relatively high – 63%, provided that the data set has unbalanced classes. AUC is also high at 89%. Conclusions. It can be concluded that the obtained model with weights has high indicators of recognition of human emotions, and can be successfully used for its purpose in the future.

Hybrid Model Recognition and Classification of Human Emotions in Thermal Images

Hybrid Model Recognition and Classification of Human Emotions in Thermal Images

Subject-independent EEG emotion recognition with hybrid spatio-temporal GRU-Conv architecture.

Recently, various deep learning frameworks have shown excellent performance in decoding electroencephalogram (EEG) signals, especially in human emotion recognition. However, most of them just focus on temporal features and ignore the features based on spatial dimensions. Traditional gated recurrent unit (GRU) model performs well in processing time series data, and convolutional neural network (CNN) can obtain spatial characteristics from input data. Therefore, this paper introduces a hybrid GRU and CNN deep learning framework named GRU-Conv to fully leverage the advantages of both. Nevertheless, contrary to most previous GRU architectures, we retain the output information of all GRU units. So, the GRU-Conv model could extract crucial spatio-temporal features from EEG data. And more especially, the proposed model acquires the multi-dimensional features of multi-units after temporal processing in GRU and then uses CNN to extract spatial information from the temporal features. In this way, the EEG signals with different characteristics could be classified more accurately. Finally, the subject-independent experiment shows that our model has good performance on SEED and DEAP databases. The average accuracy of the former is 87.04%. The mean accuracy of the latter is 70.07% for arousal and 67.36% for valence.

Context-dependent emotion recognition

Most previous methods for emotion recognition focus on facial emotion and ignore the rich context information that implies important emotion states. To make full use of the contextual information to make up for the facial information, we propose the Context-Dependent Net (CD-Net) for robust context-aware human emotion recognition. Inspired by the long-range dependency of the transformer, we introduce the tubal transformer which forms the shared feature representation space to facilitate the interactions among the face, body, and context features. Besides, we introduce the hierarchical feature fusion to recombine the enhanced multi-scale face, body, and context features for emotion classification. Experimentally, we verify the effectiveness of the proposed CD-Net on the two large emotion datasets, CAER-S and EMOTIC. On the one hand, the quantitative evaluation results demonstrate the superiority of the proposed CD-Net over other state-of-the-art methods. On the other hand, the visualization results show CD-Net can capture the dependencies among the face, body, and context components and focus on the important features related to the emotion.

Multichannel convolutional neural network for human emotion recognition from in-the-wild facial expressions

Multichannel convolutional neural network for human emotion recognition from in-the-wild facial expressions

FL-CapsNet: facial localization augmented capsule network for human emotion recognition

FL-CapsNet: facial localization augmented capsule network for human emotion recognition

Deep ganitrus algorithm for speech emotion recognition

Human emotion recognition with the evaluation of speech signals is an emerging topic in recent decades. Emotion recognition through speech signals is relatively confusing because of the speaking style, voice quality, cultural background of the speaker, environment, etc. Even though numerous signal processing methods and frameworks exists to detect and characterize the speech signal’s emotions, they do not attain the full speech emotion recognition (SER) accuracy and success rate. This paper proposes a novel algorithm, namely the deep ganitrus algorithm (DGA), to perceive the various categories of emotions from the input speech signal for better accuracy. DGA combines independent component analysis with fisher criterion for feature extraction and deep belief network with wake sleep for emotion classification. This algorithm is inspired by the elaeocarpus ganitrus (rudraksha seed), which has 1 to 21 lines. The single line bead is rarest to find, analogously finding a single emotion from the speech signal is also complex. The proposed DGA is experimentally verified on the Berlin database. Finally, the evaluation results were compared with the existing framework, and the test result accomplishes better recognition accuracy when compared with all other current algorithms.

Facial expression recognition via a jointly-learned dual-branch network

Human emotion recognition depends on facial expressions, and essentially on the extraction of relevant features. Accurate feature extraction is generally difficult due to the influence of external interference factors and the mislabelling of some datasets, such as the Fer2013 dataset. Deep learning approaches permit an automatic and intelligent feature extraction based on the input database. But, in the case of poor database distribution or insufficient diversity of database samples, extracted features will be negatively affected. Furthermore, one of the main challenges for efficient facial feature extraction and accurate facial expression recognition is the facial expression datasets, which are usually considerably small compared to other image datasets. To solve these problems, this paper proposes a new approach based on a dual-branch convolutional neural network for facial expression recognition, which is formed by three modules: The two first ones ensure features engineering stage by two branches, and features fusion and classification are performed by the third one. In the first branch, an improved convolutional part of the VGG network is used to benefit from its known robustness, the transfer learning technique with the EfficientNet network is applied in the second branch, to improve the quality of limited training samples in datasets. Finally, and in order to improve the recognition performance, a classification decision will be made based on the fusion of both branches’ feature maps. Based on the experimental results obtained on the Fer2013 and CK+ datasets, the proposed approach shows its superiority compared to several state-of-the-art results as well as using one model at a time. Those results are very competitive, especially for the CK+ dataset, for which the proposed dual branch model reaches an accuracy of 99.32, while for the FER-2013 dataset, the VGG-inspired CNN obtains an accuracy of 67.70, which is considered an acceptable accuracy, given the difficulty of the images of this dataset.

A Systematic Review on Emotion Recognition System Using Physiological Signals: Data Acquisition and Methodology

Emotion recognition systems (ERS) have become a popular research field to contribute to human-machine interaction in different areas. Different kinds of applications on ERS can serve different purposes. Artificial intelligence (AI) and the internet of things (IoT) are the technologies behind such applications. The main objective of this study is to enable researchers and developers to search for the most suitable options to develop an emotional state recognition system. More specifically, this paper presents work on ERS, which is built using physiological signals extracted from biosensors. It also presents details of how the extracted physiological signals are used to identify the user's emotional state. In this review, the sensors are categorized based on their modality: contact-based sensors and contactless sensors. Next, the ERS process is presented together with the reported results for each described technique. Articles from four different research databases were reviewed, of which 147 articles from 2009 to 2021 were referred to that are related to ERS using physiological signals. This paper should be significant for researchers developing systems that integrate human emotion recognition capability. The findings reported here can guide them in choosing suitable methods for their systems. Doi: 10.28991/ESJ-2022-06-05-017 Full Text: PDF

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

Enhancing the accuracy of a human emotion recognition method using spatial temporal graph convolutional networks

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Human emotion recognition by analyzing facial expressions, heart rate and blogs using deep learning method

Evaluating Ensemble Learning Methods for Multi-Modal Emotion Recognition Using Sensor Data Fusion.

Automatic recognition of human emotions is not a trivial process. There are many factors affecting emotions internally and externally. Expressing emotions could also be performed in many ways such as text, speech, body gestures or even physiologically by physiological body responses. Emotion detection enables many applications such as adaptive user interfaces, interactive games, and human robot interaction and many more. The availability of advanced technologies such as mobiles, sensors, and data analytics tools led to the ability to collect data from various sources, which enabled researchers to predict human emotions accurately. Most current research uses them in the lab experiments for data collection. In this work, we use direct and real time sensor data to construct a subject-independent (generic) multi-modal emotion prediction model. This research integrates both on-body physiological markers, surrounding sensory data, and emotion measurements to achieve the following goals: (1) Collecting a multi-modal data set including environmental, body responses, and emotions. (2) Creating subject-independent Predictive models of emotional states based on fusing environmental and physiological variables. (3) Assessing ensemble learning methods and comparing their performance for creating a generic subject-independent model for emotion recognition with high accuracy and comparing the results with previous similar research. To achieve that, we conducted a real-world study “in the wild” with physiological and mobile sensors. Collecting the data-set is coming from participants walking around Minia university campus to create accurate predictive models. Various ensemble learning models (Bagging, Boosting, and Stacking) have been used, combining the following base algorithms (K Nearest Neighbor KNN, Decision Tree DT, Random Forest RF, and Support Vector Machine SVM) as base learners and DT as a meta-classifier. The results showed that, the ensemble stacking learner technique gave the best accuracy of 98.2% compared with other variants of ensemble learning methods. On the contrary, bagging and boosting methods gave (96.4%) and (96.6%) accuracy levels respectively.

Performance evaluation of multi-channel electroencephalogram signal (EEG) based time frequency analysis for human emotion recognition

The automated detection of a human's emotional state by acquiring physiological or non-physiological cues is referred to as Emotion Recognition. The EEG-based approach is an effective mechanism that is extensively utilized for emotion identification in real-world settings. In this paper various classifiers are used to classify the EEG signal into three emotional states using SEED database, prepare for emotion study using physiological signals. DWT is used to decompose EEG signal in various frequency bands with “db6” as wavelet function for deriving various features like PSD, Energy, Standard Deviation, and Variance. Simultaneously we have derived various time domain features like Hjorth parameters, Maximum and Minimum value, Kurtosis, Skewness from EEG signal for recognition of emotion. The work is carried out with five electrode pairs Prefrontal (FP1-FP2), Frontal (F3-F4), Temporal (T3-T4), Parietal (P7-P8) and Occipital (O1-O2) out of 62 electrodes. Three classification methods, Support Vector Machine, K Nearest Neighbor and Decision Tree are used and their performances are compared for categorizing emotional state. The trial results show that maximum classification rate is 71.52% using decision tree and 60.19% using KNN. Electrodes FP1 and FP2 performs well in classification. Higher frequency spectrum like gamma and beta performs well in emotion recognition.

The Effect of Time Window Length on EEG-Based Emotion Recognition.

Various lengths of time window have been used in feature extraction for electroencephalogram (EEG) signal processing in previous studies. However, the effect of time window length on feature extraction for the downstream tasks such as emotion recognition has not been well examined. To this end, we investigate the effect of different time window (TW) lengths on human emotion recognition to find the optimal TW length for extracting electroencephalogram (EEG) emotion signals. Both power spectral density (PSD) features and differential entropy (DE) features are used to evaluate the effectiveness of different TW lengths based on the SJTU emotion EEG dataset (SEED). Different lengths of TW are then processed with an EEG feature-processing approach, namely experiment-level batch normalization (ELBN). The processed features are used to perform emotion recognition tasks in the six classifiers, the results of which are then compared with the results without ELBN. The recognition accuracies indicate that a 2-s TW length has the best performance on emotion recognition and is the most suitable to be used in EEG feature extraction for emotion recognition. The deployment of ELBN in the 2-s TW can further improve the emotion recognition performances by 21.63% and 5.04% when using an SVM based on PSD and DE features, respectively. These results provide a solid reference for the selection of TW length in analyzing EEG signals for applications in intelligent systems.

Facial Expression Recognition Using Convolutional Neural Network

Abstract: Human & computer interaction has been an important field of study for ages. Humans share universal and fundamental set of emotions which are exhibited through consistent facial expressions or emotion. If computer could understand the feelings of humans, it can give the proper services based on the feedback received. An algorithm that performs detection, extraction, and evaluation of these facial expressions will allow for automatic recognition of human emotion in images and videos. Automatic recognition of facial expressions can be an important component of natural human-machine interfaces; it may also be used in behavioural science and in clinical practices. In this model we give the overview of the work done in the past related to Emotion Recognition using Facial expressions along with our approach towards solving the problem. The approaches used for facial expression include classifiers like Support Vector Machine (SVM), Convolution Neural Network (CNN) are used to classify emotions based on certain regions of interest on the face like lips, lower jaw, eyebrows, cheeks and many more. Kaggle facial expression dataset with seven facial expression labels as happy, sad, surprise, fear, anger, disgust, and neutral is used in this project. The system achieved 56.77 % accuracy and 0.57 precision on testing dataset. Keywords: Facial Expression Recognition, Convolutional Neural Network, Deep Learning.