Representation Of Facial Expressions Research Articles

Stress Sense: Enhanced Stress Detection and Management Via Image Processing

This project presents an innovative approach to stress detection by utilizing Convolutional Neural Networks (CNNs) to analyze emotional cues extracted from facial images. The proposed system employs CNNs, a class of deep learning models known for their efficacy in image recognition tasks, to automatically extract features from facial images. Through a combination of convolutional, pooling, and fully connected layers, the CNN learns hierarchical representations of facial expressions associated with various emotions, including those indicative of stress. The model is trained on a diverse dataset encompassing a wide range of facial expressions, allowing it to generalize well to unseen data. Transfer learning techniques may also be employed to leverage pre-trained CNN models, further enhancing performance with limited data.

FG-AGR: Fine-Grained Associative Graph Representation for Facial Expression Recognition in the Wild

Facial expression recognition (FER) in the wild is challenging due to various unconstrained conditions, i.e., occlusions and head pose variations. Previous methods tend to improve the performance of facial expression recognition through resorting to holistic methods or coarse local-based methods, while ignoring the local fine-grained feature structure knowledge and the correlation between features. In this paper, we propose a Fine-Grained Association Graph Representation (FG-AGR) framework which can capture the local fine-grained facial expression representation. Firstly, an Adaptive Salient Region Induction (ASRI) is designed for adaptively highlighting the local saliency regions of facial expressions combined with spatial location information. Based on this, a Local Fine-grained Feature Extraction (LFFE) based on Visual Transformers is introduced to further extract fine but discriminative fine-grained features of saliency regions. Thirdly, an Adaptive Graph Association Reasoning (AGAR) based on Graph Convolutional Network is constructed to learn associated fine-grained feature combinations. Extensive experiments demonstrate that our FG-AGR achieves superior performance compared to the state-of-the-art methods with 90.81% on RAF-DB, 64.91% on AffectNet-7, 60.69% on AffectNet-8 and 91.09% on FERPlus.

FM-3DFR: Facial Manipulation-Based 3-D Face Reconstruction.

3-D Morphable model (3DMM) has widely benefited 3-D face-involved challenges given its parametric facial geometry and appearance representation. However, previous 3-D face reconstruction methods suffer from limited power in facial expression representation due to the unbalanced training data distribution and insufficient ground-truth 3-D shapes. In this article, we propose a novel framework to learn personalized shapes so that the reconstructed model well fits the corresponding face images. Specifically, we augment the dataset following several principles to balance the facial shape and expression distribution. A mesh editing method is presented as the expression synthesizer to generate more face images with various expressions. Besides, we improve the pose estimation accuracy by transferring the projection parameter into the Euler angles. Finally, a weighted sampling method is proposed to improve the robustness of the training process, where we define the offset between the base face model and the ground-truth face model as the sampling probability of each vertex. The experiments on several challenging benchmarks have demonstrated that our method achieves state-of-the-art performance.

NA

NA

Decoding the temporal representation of facial expression in face-selective regions

The ability of humans to discern facial expressions in a timely manner typically relies on distributed face-selective regions for rapid neural computations. To study the time course in regions of interest for this process, we used magnetoencephalography (MEG) to measure neural responses participants viewed facial expressions depicting seven types of emotions (happiness, sadness, anger, disgust, fear, surprise, and neutral). Analysis of the time-resolved decoding of neural responses in face-selective sources within the inferior parietal cortex (IP-faces), lateral occipital cortex (LO-faces), fusiform gyrus (FG-faces), and posterior superior temporal sulcus (pSTS-faces) revealed that facial expressions were successfully classified starting from ∼100 to 150 ms after stimulus onset. Interestingly, the LO-faces and IP-faces showed greater accuracy than FG-faces and pSTS-faces. To examine the nature of the information processed in these face-selective regions, we entered with facial expression stimuli into a convolutional neural network (CNN) to perform similarity analyses against human neural responses. The results showed that neural responses in the LO-faces and IP-faces, starting ∼100 ms after the stimuli, were more strongly correlated with deep representations of emotional categories than with image level information from the input images. Additionally, we observed a relationship between the behavioral performance and the neural responses in the LO-faces and IP-faces, but not in the FG-faces and lpSTS-faces. Together, these results provided a comprehensive picture of the time course and nature of information involved in facial expression discrimination across multiple face-selective regions, which advances our understanding of how the human brain processes facial expressions.

Face2Nodes: Learning facial expression representations with relation-aware dynamic graph convolution networks

Face2Nodes: Learning facial expression representations with relation-aware dynamic graph convolution networks

Facial expression recognition based on Feature Pyramid Network

Facial expression recognition with significant implications across fields such as psychology, computer science, and artificial intelligence. This paper proposes a combination of a Feature Pyramid Network (FPN) and a Residual Network (ResNet) to construct a recognition model. The main objective of the proposed model is to refine the multi-level feature representation of facial expressions. This approach aims to provide a more holistic understanding of the diverse and complex nature of facial expressions, recognizing the intricate interplay between macro and micro-expressions. Experimental results underscore the model's considerable superiority over traditional methods, particularly in terms of accuracy and adaptability to objects of varying sizes and complexities. This comprehensive approach to facial expression recognition showcases the potential of integrating different neural network architectures, furthering our understanding of the subtleties of facial expressions. The research, therefore, presents a significant contribution to the field of facial expression recognition, demonstrating the efficacy of integrating multi-scale feature extraction techniques to improve model performance. It sets the stage for future research directions in this domain, paving the way for more sophisticated emotion recognition systems that can be deployed in real-world applications.

How to build rapport in online space: using online chat emoticons for qualitative interviewing in feminist research

PurposeThis study draws on the author's experiences building rapport through online chat for data collection for the author's doctoral dissertation. The author contacted ten Korean women via online chat to recruit and faced the most challenging situation; building rapport. As the Millennial generation is known as being tech-savvy or digital natives, the author actively used emoticons (pictorial representations of facial expressions using characters) with potential interviewees and completed ten interviews. Therefore, this paper offers a new qualitative interviewing method in feminist research.Design/methodology/approachThe paper briefly reviews the works of literature on interviewing women on sensitive topics and building rapport before the interview. Then, the author introduced using emoticons to create rapport during the data collection process and how a non-traditional approach positively impacts the interviewer and interviewee before, during and even after the interview.FindingsWomen participants' responses and behaviors differed after building a rapport through an online chat. They were willing to share their personal stories and memories with the interviewer even though the interviewer did not ask.Research limitations/implicationsThis study provides a stepping stone for developing an account of the new qualitative methodological approach, specifically feminist qualitative research.Originality/valueFew studies have described how qualitative researchers create a rapport in virtual space, specifically using emoticons. Also, this study suggests a new methodological approach since nonverbal communication in online chat is inevitable when interviewing people in qualitative research.

The visual representation of pain facial expressions: a high-definition transcranial direct current stimulation study

The visual representation of pain facial expressions: a high-definition transcranial direct current stimulation study

Robust facial expression recognition with Transformer Block Enhancement Module

Recently, facial expression recognition (FER) methods have achieved significant progress. However, FER is still challenged by factors such as uneven illumination and low-quality expression images. Exploring the potential of facial expression features to achieve robust FER is particularly important. Inspired by the Transformer’s excellent performance in modeling long-range dependencies and in vision tasks, this paper proposes a Transformer Block Enhancement Module (TBEM) for enhancing the feature representation of facial expressions. The proposed module contains a Channel Enhancement (CE) block and a Spatial Enhancement (SE) block. The CE block can adaptively enhance the expression features on the channel dimension by effectively leveraging the channel dependency information, while the SE block enhances the expression features on the spatial dimension by integrating the spatial dependency information. TBEM can output a more robust expression representation by combining CE and SE. The proposed artificial intelligence learning module greatly improves the recognition accuracy of FER engineering tasks. To further illustrate the application of TBEM in real-world FER engineering, three engineering problems are used for verification. Extensive experiments demonstrate that the proposed method improves FER performance by focusing on more accurate decisional features and can be easily embedded into regular convolutional neural network models to help them improve the accuracy on FER tasks by about 2.64%–3.03%. The results show the proposed method achieves accuracies of 90.57% on FERPlus, 89.41% on RAFDB basic and 68.43% on RAFDB compound, respectively. The proposed method also provides a meaningful reference for further research on applying Transformer to other tasks.

Dynamic facial expression recognition with pseudo‐label guided multi‐modal pre‐training

AbstractDue to the huge cost of manual annotations, the labelled data may not be sufficient to train a dynamic facial expression (DFR) recogniser with good performance. To address this, the authors propose a multi‐modal pre‐training method with a pseudo‐label guidance mechanism to make full use of unlabelled video data for learning informative representations of facial expressions. First, the authors build a pre‐training dataset of videos with aligned vision and audio modals. Second, the vision and audio feature encoders are trained through an instance discrimination strategy and a cross‐modal alignment strategy on the pre‐training data. Third, the vision feature encoder is extended as a dynamic expression recogniser and is fine‐tuned on the labelled training data. Fourth, the fine‐tuned expression recogniser is adopted to predict pseudo‐labels for the pre‐training data, and then start a new pre‐training phase with the guidance of pseudo‐labels to alleviate the long‐tail distribution problem and the instance‐class confliction. Fifth, since the representations learnt with the guidance of pseudo‐labels are more informative, a new fine‐tuning phase is added to further boost the generalisation performance on the DFR recognition task. Experimental results on the Dynamic Facial Expression in the Wild dataset demonstrate the superiority of the proposed method.

FER-CHC: Facial expression recognition with cross-hierarchy contrast

Facial expression recognition (FER) tasks with convolutional neural networks (CNNs) have seen remarkable progress. However, these CNN-based approaches do not well capture detailed and crucial features that can distinguish different facial expressions from a global perspective. There is still much room for improvement in the performance of existing CNN-based models for FER. To address this, we propose a novel cross-hierarchy contrast (CHC) framework called FER-CHC for FER tasks. FER-CHC employs a contrastive learning mechanism to utilize these crucial features in improving the performance of CNN-based models for FER. Specifically, FER-CHC utilizes CHC to regularize the feature learning of the backbone network and enhance global representations of facial expressions. The CHC captures common and differential features from different facial expressions with a cross-hierarchy contrast mechanism. Furthermore, a fusion network globally integrates the features learned from both the backbone network and CHC to learn a more robust feature representation. We conducted comprehensive experiments on six popular datasets: CK+, FER2013, FER+, RAF-DB, AffectNet, and JAFFE. The results show that our proposed FER-CHC achieves state-of-the-art performances on these datasets. Additionally, an ablation study was conducted to demonstrate the effectiveness of the proposed components in FER-CHC.

Multimodal Emotion Detection via Attention-Based Fusion of Extracted Facial and Speech Features.

Methods for detecting emotions that employ many modalities at the same time have been found to be more accurate and resilient than those that rely on a single sense. This is due to the fact that sentiments may be conveyed in a wide range of modalities, each of which offers a different and complementary window into the thoughts and emotions of the speaker. In this way, a more complete picture of a person's emotional state may emerge through the fusion and analysis of data from several modalities. The research suggests a new attention-based approach to multimodal emotion recognition. This technique integrates facial and speech features that have been extracted by independent encoders in order to pick the aspects that are the most informative. It increases the system's accuracy by processing speech and facial features of various sizes and focuses on the most useful bits of input. A more comprehensive representation of facial expressions is extracted by the use of both low- and high-level facial features. These modalities are combined using a fusion network to create a multimodal feature vector which is then fed to a classification layer for emotion recognition. The developed system is evaluated on two datasets, IEMOCAP and CMU-MOSEI, and shows superior performance compared to existing models, achieving a weighted accuracy WA of 74.6% and an F1 score of 66.1% on the IEMOCAP dataset and a WA of 80.7% and F1 score of 73.7% on the CMU-MOSEI dataset.

Differences Between East Asians and Westerners in the Mental Representations and Visual Information Extraction Involved in the Decoding of Pain Facial Expression Intensity.

The online version contains supplementary material available at 10.1007/s42761-023-00186-1.

A database of heterogeneous faces for studying naturalistic expressions

Facial expressions are thought to be complex visual signals, critical for communication between social agents. Most prior work aimed at understanding how facial expressions are recognized has relied on stimulus databases featuring posed facial expressions, designed to represent putative emotional categories (such as ‘happy’ and ‘angry’). Here we use an alternative selection strategy to develop the Wild Faces Database (WFD); a set of one thousand images capturing a diverse range of ambient facial behaviors from outside of the laboratory. We characterized the perceived emotional content in these images using a standard categorization task in which participants were asked to classify the apparent facial expression in each image. In addition, participants were asked to indicate the intensity and genuineness of each expression. While modal scores indicate that the WFD captures a range of different emotional expressions, in comparing the WFD to images taken from other, more conventional databases, we found that participants responded more variably and less specifically to the wild-type faces, perhaps indicating that natural expressions are more multiplexed than a categorical model would predict. We argue that this variability can be employed to explore latent dimensions in our mental representation of facial expressions. Further, images in the WFD were rated as less intense and more genuine than images taken from other databases, suggesting a greater degree of authenticity among WFD images. The strong positive correlation between intensity and genuineness scores demonstrating that even the high arousal states captured in the WFD were perceived as authentic. Collectively, these findings highlight the potential utility of the WFD as a new resource for bridging the gap between the laboratory and real world in studies of expression recognition.

Facial Expression Recognition Based on Deep Learning: An Overview

Recognizing facial expressions and emotions is a basic skill that is learned at an early age and it is important for human social interaction. Facial expressions are one of the most powerful natural and immediate means that humans use to express their feelings and intentions. Therefore, automatic emotion recognition based on facial expressions become an interesting area in research, which had been introduced and applied in many areas such as security, safety health, and human machine interface (HMI). Facial expression recognition transition from controlled environmental conditions and their improvement and succession of recent deep learning approaches from different areas made facial expression representation mostly based on using a deep neural network that is generally divided into two critical issues. These are a variation of expression and overfitting. Expression variations such as identity bias, head pose, illumination, and overfitting formed as a result of a lack of training data. This paper firstly discussed the general background and terminology utilized in facial expression recognition in field of computer vision and image processing. Secondly, we discussed general pipeline of deep learning. After that, for facial expression recognition to classify emotion there should be datasets in order to compare the image with the datasets for classifying the emotion. Besides that we summarized, discussed, and compared illustrated various recent approaches of researchers that have used deep techniques as a base for facial expression recognition, then we briefly presented and highlighted the classification of the deep feature. Finally, we summarized the most critical challenges and issues that are widely present for overcoming, improving, and designing an efficient deep facial expression recognition system.

Challenging the Classical View: Recognition of Identity and Expression as Integrated Processes.

Recent neuroimaging evidence challenges the classical view that face identity and facial expression are processed by segregated neural pathways, showing that information about identity and expression are encoded within common brain regions. This article tests the hypothesis that integrated representations of identity and expression arise spontaneously within deep neural networks. A subset of the CelebA dataset is used to train a deep convolutional neural network (DCNN) to label face identity (chance = 0.06%, accuracy = 26.5%), and the FER2013 dataset is used to train a DCNN to label facial expression (chance = 14.2%, accuracy = 63.5%). The identity-trained and expression-trained networks each successfully transfer to labeling both face identity and facial expression on the Karolinska Directed Emotional Faces dataset. This study demonstrates that DCNNs trained to recognize face identity and DCNNs trained to recognize facial expression spontaneously develop representations of facial expression and face identity, respectively. Furthermore, a congruence coefficient analysis reveals that features distinguishing between identities and features distinguishing between expressions become increasingly orthogonal from layer to layer, suggesting that deep neural networks disentangle representational subspaces corresponding to different sources.

Can Hierarchical Transformers Learn Facial Geometry?

Human faces are a core part of our identity and expression, and thus, understanding facial geometry is key to capturing this information. Automated systems that seek to make use of this information must have a way of modeling facial features in a way that makes them accessible. Hierarchical, multi-level architectures have the capability of capturing the different resolutions of representation involved. In this work, we propose using a hierarchical transformer architecture as a means of capturing a robust representation of facial geometry. We further demonstrate the versatility of our approach by using this transformer as a backbone to support three facial representation problems: face anti-spoofing, facial expression representation, and deepfake detection. The combination of effective fine-grained details alongside global attention representations makes this architecture an excellent candidate for these facial representation problems. We conduct numerous experiments first showcasing the ability of our approach to address common issues in facial modeling (pose, occlusions, and background variation) and capture facial symmetry, then demonstrating its effectiveness on three supplemental tasks.

Face identity and facial expression representations with adaptation paradigms: New directions for potential applications.

Adaptation and aftereffect are well-known procedures for exploring our neural representation of visual stimuli. It has been reported that they occur in face identity, facial expressions, and low-level visual features. This method has two primary advantages. One is to reveal the common or shared process of faces, that is, the overlapped or discrete representation of face identities or facial expressions. The other is to investigate the coding system or theory of face processing that underlies the ability to recognize faces. This study aims to organize recent research to guide the reader into the field of face adaptation and its aftereffect and to suggest possible future expansions in the use of this paradigm. To achieve this, we reviewed the behavioral short-term aftereffect studies on face identity (i.e., who it is) and facial expressions (i.e., what expressions such as happiness and anger are expressed), and summarized their findings about the neural representation of faces. First, we summarize the basic characteristics of face aftereffects compared to simple visual features to clarify that facial aftereffects occur at a different stage and are not inherited or combinations of low-level visual features. Next, we introduce the norm-based coding hypothesis, which is one of the theories used to represent face identity and facial expressions, and adaptation is a commonly used procedure to examine this. Subsequently, we reviewed studies that applied this paradigm to immature or impaired face recognition (i.e., children and individuals with autism spectrum disorder or prosopagnosia) and examined the relationships between their poor recognition performance and representations. Moreover, we reviewed studies dealing with the representation of non-presented faces and social signals conveyed via faces and discussed that the face adaptation paradigm is also appropriate for these types of examinations. Finally, we summarize the research conducted to date and propose a new direction for the face adaptation paradigm.

A discriminatively deep fusion approach with improved conditional GAN (im-cGAN) for facial expression recognition

Considering most deep learning-based methods heavily depend on huge labels, it is still a challenging issue for facial expression recognition to extract discriminative features of training samples with limited labels. Given above, we propose a discriminatively deep fusion (DDF) approach based on an improved conditional generative adversarial network (im-cGAN) to learn abstract representation of facial expressions. First, we employ facial images with action units (AUs) to train the im-cGAN to generate more labeled expression samples. Subsequently, we utilize global features learned by the global-based module and the local features learned by the region-based module to obtain the fused feature representation. Finally, we design the discriminative loss function (D-loss) that expands the inter-class variations while minimizing the intra-class distances to enhance the discrimination of fused features. Experimental results on JAFFE, CK+, Oulu-CASIA, and KDEF datasets demonstrate the proposed approach is superior to some state-of-the-art methods.