Facial Expression Signals Research Articles

Emotion recognition using cross-modal attention from EEG and facial expression

The fusion of facial expression and Electroencephalogram (EEG) signals in a multi-modal framework is a comprehensive and accurate method for emotion recognition. However, current methods often tend to directly concatenate two modalities, thereby overlooking the interplay between modalities. Furthermore, the selection of single static facial images for extracting facial features neglected the dynamic changes in facial expressions. These limitations have been identified as factors that constrain the accuracy and stability of the model. In this study, an innovative multimodal emotion recognition network was introduced, which using continuous facial expressions and EEG signals. It incorporated the cross-modal attention fusion mechanism to establish robust correlations between modal feature vectors, thereby generating amalgamated vectors with mutual information. Additionally, a Self-Attention Convolutional Long Short-Term Memory (SA-ConvLSTM) was employed to capture spatiotemporal information from facial expression images. The model proposed in this paper was experimentally evaluated on the DEAP and MAHNOB-HCI datasets. Its recognition accuracy was higher than the existing advanced research methods. At the same time, it still performed well in the Leave-One-Subject-Out (LOSO) experiment based on DEAP dataset. The experimental results showed the effectiveness of the proposed model in the task of multimodal emotion recognition.

Emotional analysis of joint sports quality expansion tasks based on multi-modal feature fusion

A multi-modal feature based motion emotion analysis model based on a fusion deep learning model is proposed for the problem of analyzing the motion emotions of participants in the joint exercise quality expansion task. This model involves three major modalities: EEG signals, peripheral physiological signals, and facial expression signals, and processes and fuses the information of these three main modalities to achieve the effect of processing multi-dimensional motor emotional information. At the same time, this study introduces the design concept of residual networks, using self attention modules and multi head mutual attention modules to process different modal features. The results showed that the combination of peripheral physiological modality and facial expression modality had the highest accuracy among the three modality combinations, with an accuracy rate of 88.8 %. The feature fusion method based on the cascaded residual attention mechanism module has better accuracy and F1 Score performance than other methods. In addition, different emotional states can be effectively identified and distinguished in these three modalities, indicating that the model has a wide range of possibilities in practical applications.

MGEED: A Multimodal Genuine Emotion and Expression Detection Database

Multimodal emotion recognition has attracted increasing interest from academia and industry in recent years, since it enables emotion detection using various modalities, such as facial expression images, speech and physiological signals. Although research in this field has grown rapidly, it is still challenging to create a multimodal database containing facial electrical information due to the difficulty in capturing natural and subtle facial expression signals, such as optomyography (OMG) signals. To this end, we present a newly developed Multimodal Genuine Emotion and Expression Detection (MGEED) database in this paper, which is the first publicly available database containing the facial OMG signals. MGEED consists of 17 subjects with over 150K facial images, 140K depth maps and different modalities of physiological signals including OMG, electroencephalography (EEG) and electrocardiography (ECG) signals. The emotions of the participants are evoked by video stimuli and the data are collected by a multimodal sensing system. With the collected data, an emotion recognition method is developed based on multimodal signal synchronisation, feature extraction, fusion and emotion prediction. The results show that superior performance can be achieved by fusing the visual, EEG and OMG features. The database can be obtained from <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/YMPort/MGEED</uri> .

Spontaneous Facial Behavior Analysis Using Deep Transformer-based Framework for Child–computer Interaction

A fascinating challenge in robotics-human interaction is imitating the emotion recognition capability of humans to robots with the aim to make human-robotics interaction natural, genuine and intuitive. To achieve the natural interaction in affective robots, human-machine interfaces, and autonomous vehicles, understanding our attitudes and opinions is very important, and it provides a practical and feasible path to realize the connection between machine and human. Multimodal interface that includes voice along with facial expression can manifest a large range of nuanced emotions compared to purely textual interfaces and provide a great value to improve the intelligence level of effective communication. Interfaces that fail to manifest or ignore user emotions may significantly impact the performance and risk being perceived as cold, socially inept, untrustworthy, and incompetent. To equip a child well for life, we need to help our children identify their feelings, manage them well, and express their needs in healthy, respectful, and direct ways. Early identification of emotional deficits can help to prevent low social functioning in children. In this work, we analyzed the child’s spontaneous behavior using multimodal facial expression and voice signal presenting multimodal transformer-based last feature fusion for facial behavior analysis in children to extract contextualized representations from RGB video sequence and Hematoxylin and eosin video sequence and then using these representations followed by pairwise concatenations of contextualized representations using cross-feature fusion technique to predict users emotions. To validate the performance of the proposed framework, we have performed experiments with the different pairwise concatenations of contextualized representations that showed significantly better performance than state-of-the-art method. Besides, we perform t-distributed stochastic neighbor embedding visualization to visualize the discriminative feature in lower dimension space and probability density estimation to visualize the prediction capability of our proposed model.

Stretchable Organohydrogel with Adhesion, Self-Healing, and Environment-Tolerance for Wearable Strain Sensors.

Stretchable hydrogels as landmark soft materials have been efficiently utilized in the field of wearable sensing devices. However, these soft hydrogels mostly cannot integrate transparency, stretchability, adhesiveness, self-healing, and environmental adaptability into one system. Herein, a fully physically cross-linked poly(hydroxyethyl acrylamide)-gelatin dual-network organohydrogel is prepared in a phytic acid-glycerol binary solvent via a rapid ultraviolet light initiation. The introduction of gelatin as the second network endows the organohydrogel with desirable mechanical performance (high stretchability up to 1240%). The presence of phytic acid not only synergizes with glycerol to impart environment-tolerance to the organohydrogel (from -20 to 60 °C) but also increases the conductivity. Moreover, the organohydrogel demonstrates a durable adhesive performance toward diverse substrates, a high self-healing efficiency through heat treatment, and favorable optical transparency (transmittance of 90%). Furthermore, the organohydrogel achieves high sensitivity (gauge factor of 2.18 at 100% strain) and rapid response time (80 ms) and could detect both tiny (a low detection limit of 0.25% strain) and large deformations. Therefore, the assembled organohydrogel-based wearable sensors are capable of monitoring human joint motions, facial expression, and voice signals. This work proposes a facile route for multifunctional organohydrogel transducers and promises the practical application of flexible wearable electronics in complex scenarios.

MMFN: Emotion recognition by fusing touch gesture and facial expression information

Human social interaction is a multimodal process that integrates several communication channels, such as speech, facial expressions, body gestures, and touch. For intelligent robots, the ability to recognize human emotions based on these channels helps in realizing natural human-robot interaction. The existing works on multimodal emotion recognition (MER) mainly focus on facial expression, speech, text, and electrophysiological signals. However, there are few works on the implementation of MER based on touch. In this work, we established a facial expression and touch gesture emotion (FETE) dataset comprising six basic discrete emotions, which was made publicly available for the first time. In addition, a multi-eigenspace based multimodal fusion network (MMFN) was proposed for tactile-visual bimodal emotion recognition. The proposed MMFN projects the tactile and visual modalities in multi-eigenspace to learn the specific and shared representations and employs multiple classifiers to decode these features. We demonstrated the effectiveness of our network through experiments on the FETE dataset. The tactile-visual dual-modal emotion recognition using MMFN achieved an accuracy of 80.18% and an improvement of approximately 15% over the single-modality results.

Basic emotions do not reliably co-occur with predicted facial expressions: Reply to Witkower et al. (2023).

Replies to the comments made by Witkower, et al. (see record 2023-63008-004) on the current authors original article (see record 2022-03375-001). A core assumption of Basic Emotion Theory is that the conscious experience of a basic emotion co-occurs with a facial expression signal of that same emotion. Our analysis of available evidence found co-occurrence in only 13% of cases-thus calling into question basic and applied studies in which the emotion is inferred from the face. Our second analysis counted as a co-occurrence even when only part of the facial signal was observed. Co-occurrence was found in only 23% of cases. Witkower et al.'s rebuttal failed to undermine these important findings. They claimed that similar degrees of correlation are found in other areas of psychology, but they confuse co-occurrence of two intrinsic manifestations of the same event (expression and experience of emotion) with the correlation between one potential causal antecedent and an observed event (e.g., effects of meditation on anxiety). Our results stand as a major challenge to Basic Emotion Theory. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Flexible strain sensors with high sensitivity and large working range prepared from biaxially stretched carbon nanotubes/polyolefin elastomer nanocomposites

AbstractFlexible strain sensors are a new generation of flexible and stretchable electronic devices that attracted increasing attention due to their practical applications in many fields. However, maintaining a wide detectable strain range while improving the sensitivity of flexible strain sensors remains challenging. In this study, flexible strain sensors with a large working range based on biaxially stretched carbon nanotubes (CNTs)/polyolefin elastomer (POE) nanocomposites were fabricated. Biaxial stretching was demonstrated to enhance the uniform dispersion and orientation of CNTs, thereby improving the performance of sensors. The optimal stretching ratios (SRs) of nanocomposites were investigated and the data revealed an increment in the sensitivity of sensors with SRs, while the working range first increased after biaxial stretching and decreased at higher SRs. Compared to the 9 wt% CNT/POE‐1.0 sensor with a gauge factor (GF) value of 2.37 and a detectable range of 0.5%–230%, the CNT/POE‐2.0 sensor exhibited an enhanced sensitivity (GF = 3935.12) coupled with a wider detectable range (0.5%–710%) and better stability. Besides, CNT/POE‐2.0 sensor also achieved the monitoring of head movements, mouth opening, facial expression, and physiological signals, showing a potential for use in wearable electronic products.

Multimodal Sentiment Sensing and Emotion Recognition Based on Cognitive Computing Using Hidden Markov Model with Extreme Learning Machine

In today's competitive business environment, exponential increase of multimodal content results in a massive amount of shapeless data. Big data that is unstructured has no specific format or organisation and can take any form, including text, audio, photos, and video. Many assumptions and algorithms are generally required to recognize different emotions as per literature survey, and the main focus for emotion recognition is based on single modality, such as voice, facial expression and bio signals. This paper proposed the novel technique in multimodal sentiment sensing with emotion recognition using artificial intelligence technique. Here the audio and visual data has been collected based on social media review and classified using hidden Markov model based extreme learning machine (HMM_ExLM). The features are trained using this method. Simultaneously, these speech emotional traits are suitably maximised. The strategy of splitting areas is employed in the research for expression photographs and various weights are provided to each area to extract information. Speech as well as facial expression data are then merged using decision level fusion and speech properties of each expression in region of face are utilized to categorize. Findings of experiments show that combining features of speech and expression boosts effect greatly when compared to using either speech or expression alone. In terms of accuracy, recall, precision, and optimization level, a parametric comparison was made.

Multi-feature fusing local directional ternary pattern for facial expressions signal recognition based on video communication system

In the field of Automatic Facial Expression Signal Recognition (AFESR) at video communication system, the fusing feature extraction is playing an extremely important role in recognition accuracy. This paper presents a new feature extraction method, Multi-Feature Fusing Local Directional Ternary Pattern (MFF-LDTP) which keeps more feature information and improvs the robustness under the uncontrollable and wild environment for AFESR. Firstly, the MFF-LDTP operator obtains the global feature of facial expression by Principal Components Analysis (PCA). Secondly, the MFF-LDTP enhances traditional Local Directional Ternary Pattern （LDTP）by using a “kirsch mask” to replace the Frei-Chen masks and selects the threshold for facial expression signal recognition. To effectively avoid generating invalid features, the MFF-LDTP extracts the local feature of eye and mouth which are significant regions by ELDTP. Thirdly, The MFF-LDTP final feature vector includes the linear connection of global and local features. The recognition rate for the extended JAFFE database is 96.5%. And the extended JAFFE includes captured sample images under an uncontrollable and wild environment. The experimental results show that the proposed MFF-LDTP achieved significant improvement and outperformed some state-of-the-art methods.

EEG-based cross-subject emotion recognition using Fourier-Bessel series expansion based empirical wavelet transform and NCA feature selection method

Automated emotion recognition using brain electroencephalogram (EEG) signals is predominantly used for the accurate assessment of human actions as compared to facial expression or speech signals. Various signal processing methods have been used for extracting representative features from EEG signals for emotion recognition. However, the EEG signals are non-stationary and vary across the subjects as well as in different sessions of the same subject; hence it exhibits poor generalizability and low classification accuracy for an emotion classification of cross subjects. In this paper, EEG signals-based automated cross-subject emotion recognition framework is proposed using the Fourier-Bessel series expansion-based empirical wavelet transform (FBSE-EWT) method. This method is used to decompose the EEG signals from each channel into four sub-band signals. Manually ten channels are selected from the frontal lobe, from which entropy and energy features are extracted from each sub-band signal. The subject variability is reduced using an average moving filter method on each channel to obtain the smoothened feature vector of size 80. The three feature selection techniques, such as neighborhood component analysis (NCA), relief-F, and mRMR, are used to obtain an optimal feature vector. The machine learning models, such as artificial neural network (ANN), k-nearest neighborhood (k-NN) with two (fine and weighted) functions, and ensemble bagged tree classifiers are trained by the obtained feature vectors. The experiments are performed on two publicly accessible databases, named SJTU emotion EEG dataset (SEED) and dataset for emotion analysis using physiological signals (DEAP). The training and testing of the models have been performed using 10-fold cross-validation and leave-one-subject-out-cross-validation (LOSOCV). The proposed framework based on FBSE-EWT and NCA feature selection approach shows superior results for classifying human emotions compared to other state-of-art emotion classification models.

Comparative Analysis of Emotion Classification Based on Facial Expression and Physiological Signals Using Deep Learning

This study aimed to classify emotion based on facial expression and physiological signals using deep learning and to compare the analyzed results. We asked 53 subjects to make facial expressions, expressing four types of emotion. Next, the emotion-inducing video was watched for 1 min, and the physiological signals were obtained. We defined four emotions as positive and negative emotions and designed three types of deep-learning models that can classify emotions. Each model used facial expressions and physiological signals as inputs, and a model in which these two types of input were applied simultaneously was also constructed. The accuracy of the model was 81.54% when physiological signals were used, 99.9% when facial expressions were used, and 86.2% when both were used. Constructing a deep-learning model with only facial expressions showed good performance. The results of this study confirm that the best approach for classifying emotion is using only facial expressions rather than data from multiple inputs. However, this is an opinion presented only in terms of accuracy without considering the computational cost, and it is suggested that physiological signals and multiple inputs be used according to the situation and research purpose.

The use of social referencing to teach safety skills to toddlers with autism

AbstractSocial referencing is a chain of behavior where the presence of an ambiguous stimulus evokes a gaze shift from the stimulus to another individual. The other individual's facial expression signals either the availability of reinforcement or punishment. The purpose was to teach two toddlers with autism to discriminate between safe and dangerous stimuli through a social referencing chain. Participants were trained using differential reinforcement and least‐to‐most prompting to gaze shift from an item inside of a lunchbox or a bin to an adult, and to respond differentially based on the adult's facial expression. Results showed acquisition of both the discrimination between safe and dangerous stimuli and the maintenance of a social referencing chain in the presence of unfamiliar stimuli. These findings are discussed as they relate to the implications of teaching socially valid safety skills to toddlers with autism.

Facial Expressions of Emotion Transmit Multiplexed Signals of Categorical and Dimensional Information

Human facial expressions are complex dynamic signals that are used to communicate emotions 1. As a highly evolved social species, human facial expressions likely signal complex information to inform adaptive response 2. However, a long-standing debate is what information such facial expressions transmit. One influential theory posits that facial expressions transmit discrete categorical information such as happy or anger 3–6, while another argues they transmit broad dimensional information such as valence and arousal 7–10 (see 11 for a review). Here, we tested an alternative that reconciles these contrasting theories – that facial expressions are multiplexed signals that jointly transmit categorical and dimensional information. Using a data-driven approach 12,13 and subjective affective perception, we first reconstructed the facial expression signals that transmit emotion category information (i.e., the six classic emotions, plus 19 complex emotions) and, separately, those that transmit affective dimensional information (i.e., valence and arousal). Cross-comparison showed that facial expressions of emotion categories are systematically embedded as discrete clusters within those of affective dimensions, suggesting that certain facial movements jointly transmit emotion categories and affective dimensions as multiplexed signals. Next, using a novel information-theoretic analysis 14,15, we disentangled the specific information that each facial movement transmits – emotion categories, affective dimensions, or both as multiplex signals. Results showed that facial expressions of emotion primarily comprise complexed multiplex signals that jointly transmit emotion category and affective dimensional information. Our results unite long-standing contrasted theories of emotion communication and provide a novel framework that can better characterize the complexities of human facial expression signalling.

Facial Expression and EEG Fusion for Investigating Continuous Emotions of Deaf Subjects

Emotion recognition has received increasing attention in human-computer interaction (HCI) and psychological assessment. Compared with single modal emotion recognition, the multimodal paradigm has an outperformance because of introducing complementary information for emotion recognition. However, current research is mainly focused on normal people, the deaf subjects need to understand emotional changes in real life. In this paper, we propose a multimodal continuous emotion recognition method for deaf subjects based on facial expression and electroencephalograph (EEG) signals. Twelve emotion movie clips as stimulus were selected and annotated by ten postgraduates who majored in psychology. The EEG signals and facial expressions of deaf subjects were collected when they watched the stimulus clips. The differential entropy (DE) features of EEG were extracted by time-frequency analysis and the six facial features were extracted by facial landmark. Long short-term memory networks (LSTM) were utilized to accomplish the feature level fusion and captured the temporal dynamics of emotions. The result shows that the EEG signal is better than the dynamic emotional capture of facial expressions and deaf subjects in continuous emotion recognition. Multi-modality can compensate for the information of a single modality, which achieves a better performance. Finally, from the neural activities of deaf subjects, the result reveals that the prefrontal lobe region may be strongly related to negative emotion processing, and the lateral temporal lobe region may be strongly related to positive emotion processing.

Introduction to the Special Issue on MMAC: Multimodal Affective Computing of Large-Scale Multimedia Data

The articles in this special section focus on multimodal affective computing of large scale multimedia data. Humans are emotional creatures. Emotion is present everywhere in our daily life and plays a vitally important role in our decisionmaking process. There are roughly two categories of modalities that are widely used by humans to express emotions: explicit affective cues and implicit affective stimuli. On the one hand, to distinguish whether an emotion is induced, one direct way is to check the physical changes of involved humans, such as facial expression, speech, action, gait, and physiological signals (e.g., electroencephalogram). These explicit affective cues can be directly observed and collected from an individual with specific sensors. On the other hand, the rapid development of digital photography and social networks has enabled humans to share their lives and expressing their opinions online using implicit affective stimuli, such as text, images, audios, and videos. The usergenerated content provides an implicit solution to analyze humans’ emotions. Affective computing of explicit and/or implicit large-scale multimedia data is rather challenging due to the following reasons. First, the development of affective analysis is constrained by the affective gap between low-level affective features and high-level emotions. Second, emotion is a subjective concept, and thus affective analysis involves multidisciplinary understanding of human perceptions and behaviors. Finally, emotions are often jointly expressed and perceived through multiple modalities. Multimodal data fusion and complementation need to be explored.

Emotion recognition by deeply learned multi-channel textual and EEG features

Human emotion recognition is a key technique in human–computer interaction. Traditional emotion recognition algorithms rely on external actions such as facial expression, which may fail to capture real human emotion since facial expression signals may be camouflaged. EEG signal is closely close to human emotion, which can directly reflect human emotion. In this paper, we propose to learn multi-channel features from the EEG signal for human emotion recognition, where the EEG signal is generated by sound signal stimulation. Specifically, we apply multi-channel EEG and textual feature fusion in time-domain to recognize different human emotions, where six statistical features in time-domain are fused to a feature vector for emotion classification. The textual feature extraction is based. And we conduct EEG&textual-based feature extraction from both time and frequency domain. Finally, we train SVM for human emotion recognition. Experimental on DEAP dataset show that compared with frequency domain feature-based emotion recognition algorithms, our proposed method improves recognition accuracy rate.

Federated Learning Meets Human Emotions: A Decentralized Framework for Human–Computer Interaction for IoT Applications

As stated by Spock, “change is the essential process of all existence,” which is reflected in everyday applications in our daily lives. We, as humans, just need to find a way to make the best use of the current technological advances. The pandemic has managed to exploit our deepest vulnerabilities and insecurities. We need to cope with a lot of things, just to be comfortable in the new normal. Hence, we can rely on technology, the greatest asset developed by humans. In this article, we discuss how we can enhance the work environment in offices post-pandemic. We combine federated learning with emotion analysis to create a state-of-the-art, simple, secure, and efficient emotion monitoring system. We combine facial expression and speech signals to find out macroexpressions and create an emotion index that is monitored to find the mental health of the user. Federated learning enables users to locally train the model without compromising his/her privacy. In place of sending data to the centralized server, the proposed scheme sends only model weights that are combined at the server to make a better global model, which is further pushed back to the users. This model is then trained interorganizational as it does not violate the privacy or data sharing to achieve optimal results. The data collected from users are monitored to analyze the mental health and presented with counseling solutions during low times. Technology is a panacea that has enabled us to survive in this pandemic, and by using our solution to improve work culture and the environment in post-pandemic times.

Smart Affect Recognition System for Real-Time Biometric Surveillance Using Hybrid Features and Multilayered Binary Structured Support Vector Machine

Abstract Human affect recognition (HAR) using images of facial expression and electrocardiogram (ECG) signal plays an important role in predicting human intention. This system improves the performance of the system in applications like the security system, learning technologies and health care systems. The primary goal of our work is to recognize individual affect states automatically using the multilayered binary structured support vector machine (MBSVM), which efficiently classify the input into one of the four affect classes, relax, happy, sad and angry. The classification is performed efficiently by designing an efficient support vector machine (SVM) classifier in multilayer mode operation. The classifier is trained using the 8-fold cross-validation method, which improves the learning of the classifier, thus increasing its efficiency. The classification and recognition accuracy is enhanced and also overcomes the drawback of ‘facial mimicry’ by using hybrid features that are extracted from both facial images (visual elements) and physiological signal ECG (signal features). The reliability of the input database is improved by acquiring the face images and ECG signals experimentally and by inducing emotions through image stimuli. The performance of the affect recognition system is evaluated using the confusion matrix, obtaining the classification accuracy of 96.88%.

CONTVERB: Continuous Virtual Emotion Recognition Using Replaceable Barriers for Intelligent Emotion-Based IoT Services and Applications

With a recent advent of technology, it is possible to recognize human emotion through various schemes which cover audio, visual data, facial expression and wireless signal and its reflection. When we design a virtual emotion system with an applicability of replaceable barrier concept in IoT environment, it is a critical issue to avoid the weak hole in the barriers. In this article, we introduce a continuous virtual emotion detection system toward suitable emotion-based services in advanced intelligent smart cities. Then, we formally define a MaxContLDA problem whose objective is to maximize the continuous lowest emotion detection accuracy to avoid weak detection holes in replaceable barriers. To solve the problem, we propose a novel approach using lowest edge replacement and implement it through extensive simulations with different scenarios. Also, we analyze the performance of the proposed approach which returns possible maximum lowest emotion detection accuracy value for those various simulation settings and scenarios. Furthermore, we discuss future research challenges, issues and plans with expanded perspectives.