Expression Recognition Research Articles

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

Synchrony or asynchrony: development of facial expression recognition from childhood to adolescence based on large-scale evidence.

The development of facial expression recognition ability in children is crucial for their emotional cognition and social interactions. In this study, 510 children aged between 6 and 15 participated in a two forced-choice task of facial expression recognition. The findings supported that recognition of the six basic facial expressions reached a relatively stable mature level around 8-9 years old. Additionally, model fitting results indicated that children showed the most significant improvement in recognizing expressions of disgust, closely followed by fear. Conversely, recognition of expressions of happiness and sadness showed slower improvement across different age groups. Regarding gender differences, girls exhibited a more pronounced advantage. Further model fitting revealed that boys showed more pronounced improvements in recognizing expressions of disgust, fear, and anger, while girls showed more pronounced improvements in recognizing expressions of surprise, sadness, and happiness. These clear findings suggested the synchronous developmental trajectory of facial expression recognition from childhood to adolescence, likely influenced by socialization processes and interactions related to brain maturation.

Facial expression recognition ability and its neuropsychological mechanisms in children with attention deficit and hyperactive disorder.

Attention deficit and hyperactive disorder (ADHD) is a chronic neurodevelopmental disorder characterized by inattention, hyperactivity-impulsivity, and working memory deficits. Social dysfunction is one of the major challenges faced by children with ADHD. It has been found that children with ADHD can't perform as well as typically developing children on facial expression recognition (FER) tasks. Generally, children with ADHD have some difficulties in FER, while some studies suggest that they have no significant differences in accuracy of specific emotion recognition compared with typically developing children. The neuropsychological mechanisms underlying these difficulties are as follows. First, neuroanatomically. Compared to typically developing children, children with ADHD show smaller gray matter volume and surface area in the amygdala and medial prefrontal cortex regions, as well as reduced density and volume of axons/cells in certain frontal white matter fiber tracts. Second, neurophysiologically. Children with ADHD exhibit increased slow-wave activity in their electroencephalogram, and event-related potential studies reveal abnormalities in emotional regulation and responses to angry faces when facing facial stimuli. Third, psychologically. Psychosocial stressors may influence FER abilities in children with ADHD, and sleep deprivation in ADHD children may significantly increase their recognition threshold for negative expressions such as sadness and anger. This article reviews research progress over the past three years on FER abilities of children with ADHD, analyzing the FER deficit in children with ADHD from three dimensions: neuroanatomy, neurophysiology and psychology, aiming to provide new perspectives for further research and clinical treatment of ADHD.

Assessing children's outdoor thermal comfort with facial expression recognition: An efficient approach using machine learning

In this study, children's physiological indices and facial expressions during activities with distinct intensities in outdoor open spaces were real-timely measured. Correspondingly, meteorological measurements and questionnaire surveys were conducted to explore change laws of children's physiological feedback, facial expressions, and subjective perception. Then, a predictive model of children's outdoor thermal sensations was built with the coupling relations of physiological indices and facial expressions. Results showed that the skin temperature on the face tended to increase as thermal stress increased but gradually declined with increasing activity intensity. Mean ear skin temperature (Tear) was the most sensitive, followed by mean cheek temperature (Tcheek). When conducting moderate and vigorous intensity activities, the heart rate (HR) increased significantly within 5 min of exercise and then stabilized. The HR increase in vigorous intensity activities was greater than that during moderate intensity activities. With an increase in thermal stress and activity intensity, the proportion of negative emotions (NegEmo) in facial expressions gradually increased. Sadness and disgust expressions both increased significantly, followed by fear and anger. Thus, Tear and NegEmo expressed facially were important indices to predict children's outdoor thermal sensation. Finally, the predictive model for children's outdoor thermal sensation based on facial expressions, Tear, and HR allowed for non-invasive data collection was built integrating machine learning, and its accuracy reached as high as 97.1 %. Our results propose a faster feedback method for children's thermal perception based on facial expression recognition, which can real-timely monitor thermal comfort for children, ensuring their outdoor thermal health and safety.

Lightweight facial expression estimation for mobile computing in portable device

AbstractFacial expression recognition has been studied for many years, especially with the development of deep learning. However, the existing researches still have the following two issues. Firstly, the intensity of facial expression is neglected. Secondly, the deep learning based approaches cannot be directly deployed in the devices with limited resources. In order to tackle these two issues, this paper proposes a lightweight facial expression estimation method using a shallow ordinal regression algorithm, which is deployed in a portable smart device for mobile computing in IoTs. Compared with classification based facial expression recognition methods, ordinal regression considers the intensity of facial expression to achieve better mean absolute error (MAE), which is validated by experiments on several public facial expression datasets. The simulation in portable device also demonstrates its effectiveness for mobile computing.

Discriminative dictionary learning for nonnegative representation based classification

Representation based classification (RC) algorithms have been successfully applied to pattern classification. However, most existing RC algorithms are not robust to bad training samples, since they ignore learning more discriminative dictionary atoms and instead directly use training samples as dictionary atoms. In addition, in order to improve expression ability and recognition accuracy, RC algorithms often need to expand the number of dictionary atoms, resulting in a surge in storage and computing costs. To obtain a more discriminative and compact dictionary, this study proposes discriminative dictionary learning for nonnegative representation based classification (DDLNRC). Specifically, in the DDLNRC, this paper utilizes nonnegative constraint to obtain a nonnegative representation for each training sample on the dictionary. In the dictionary learning stage, for a training sample, the DDLNRC minimizes the intra-class reconstruction error of the training sample and simultaneously enlarges the distance between the training sample and the atom that has the most influence on inter-class reconstruction error. Experiments demonstrate the effectiveness of the DDLNRC. Combined with the deep neural network features, it also can achieve higher accuracy than using the Softmax.

Painful Experiences in Social Contexts Facilitate Sensitivity to Emotional Signals of Pain from Conspecifics in Laboratory Rats.

Previous studies demonstrated that laboratory rats could visually receive emotional pain signals from conspecifics through pictorial stimuli. The present study examined whether a prior painful emotional experience of the receiver influenced the sensitivity of emotional expression recognition in laboratory rats. The experiment comprised four phases: the baseline preference test, pain manipulation test, post-manipulation preference test, and state anxiety test. In the baseline phase, the rats explored an apparatus comprising two boxes to which pictures of pain or neutral expressions of other conspecifics were attached. In the pain manipulation phase, each rat was allocated to one of three conditions: foot shock alone (pained-alone; PA), foot shock with other unfamiliar conspecifics (pained-with-other; PWO), or no foot shock (control). In the post-manipulation phase, the animals explored the apparatus in the same manner as they did in the baseline phase. Finally, an open-field test was used to measure state anxiety. These findings indicate that rats in the PWO group stayed longer per entry in a box with photographs depicting a neutral disposition than in a box with photographs depicting pain after manipulation. The results of the open-field test showed no significant differences between the groups, suggesting that the increased sensitivity to pain expression in other individuals due to pain experiences in social settings was not due to increased primary state anxiety. Furthermore, the results indicate that rats may use a combination of self-painful experiences and the states of other conspecifics to process the emotional signal of pain from other conspecifics. In addition, changes in the responses of rats to facial expressions in accordance with social experience suggest that the expression function of rats is not only used for emotional expressions but also for communication.

Visual Field Restriction in the Recognition of Basic Facial Expressions: A Combined Eye Tracking and Gaze Contingency Study.

Uncertainties and discrepant results in identifying crucial areas for emotional facial expression recognition may stem from the eye tracking data analysis methods used. Many studies employ parameters of analysis that predominantly prioritize the examination of the foveal vision angle, ignoring the potential influences of simultaneous parafoveal and peripheral information. To explore the possible underlying causes of these discrepancies, we investigated the role of the visual field aperture in emotional facial expression recognition with 163 volunteers randomly assigned to three groups: no visual restriction (NVR), parafoveal and foveal vision (PFFV), and foveal vision (FV). Employing eye tracking and gaze contingency, we collected visual inspection and judgment data over 30 frontal face images, equally distributed among five emotions. Raw eye tracking data underwent Eye Movements Metrics and Visualizations (EyeMMV) processing. Accordingly, the visual inspection time, number of fixations, and fixation duration increased with the visual field restriction. Nevertheless, the accuracy showed significant differences among the NVR/FV and PFFV/FV groups, despite there being no difference in NVR/PFFV. The findings underscore the impact of specific visual field areas on facial expression recognition, highlighting the importance of parafoveal vision. The results suggest that eye tracking data analysis methods should incorporate projection angles extending to at least the parafoveal level.

Modelling appearance variations in expressive and neutral face image for automatic facial expression recognition

AbstractIn automatic facial expression recognition (AFER) systems, modelling the spatio‐temporal feature information in a specific manner, coalescing, and its effective utilization is challenging. The state‐of‐the‐art studies have examined integrating multiple features to enhance the recognition rate of AFER systems. However, the feature variations between expressive and neutral face images are not fully explored to identify the expression class. The proposed research presents an innovative approach to AFER by modelling appearance variations in both expressive and neutral face images. The prominent contributions of the work are developing a novel and hybrid feature space by integrating the discriminative feature distribution derived from expressive and neutral face images; preserving the highly discriminative latent feature distribution using autoencoders. Local binary pattern (LBP) and histogram of oriented gradients (HOG) are the feature descriptors employed to derive the discriminative texture and shape information, respectively. The component‐based approach is employed, wherein the features are derived from the salient facial regions instead of the whole face. The three‐stage stacked deep convolutional autoencoder (SDCA) and multi‐class support vector machine (MSVM) are employed to address dimensionality reduction and classification, respectively. The efficacy of the proposed model is substantiated by empirical findings, which establish its superiority in terms of accuracy in AFER tasks on widely recognized benchmark datasets.

Real-Time Emotion Recognition System Using Facial Expressions

This paper describes an emotion detection system based on real-time detection using image processing with human-friendly machine interaction. Facial detection has been around for decades. Taking a step ahead, Human expressions displayed by face and felt by the brain, captured via video, electric signal, or image form can be approximated. To recognize emotions via images or videos is a difficult task for the human eye and challenging for machines thus detection of emotion by a machine requires many image processing techniques for feature extraction. This paper proposes a system that has two main processes such as face detection and Facial expression recognition (FER). This research focuses on an experimental study on identifying facial emotions. The flow for an emotion detection system includes the image acquisition, preprocessing of an image, Face detection, feature extraction, and classification. To identify such emotions, the emotion detection system uses KNN Classifier for image classification, and Haar cascade algorithm an Object Detection Algorithm to identify faces in an image or a real-time video. This system works by taking live images from the webcam. The objective of this research is to produce an automatic facial emotion detection system to identify different emotions based on these experiments the system could identify several people that are sad, surprised, and happy, in fear, are angry, disgust etc.

A multi-featured expression recognition model incorporating attention mechanism and object detection structure for psychological problem diagnosis

Expression is the main method for judging the emotional state and psychological condition of the human body, and the prediction of changes in facial expressions can effectively determine the mental health of a person, thus avoiding serious psychological or psychiatric disorders due to early negligence. From a computer vision perspective, most researchers have focused on studying facial expression analysis, and in some cases, body posture is also considered. However their performance is more limited under unconstrained natural conditions, which requires more information to be used in human emotion analysis. In this paper, we design an Adaptive Multi-End Fusion Attention Mechanism suitable for extracting human body information based on the deep learning framework, depending on human expressions, postures and the environment they are in and add it to an object detection model to obtain the information we need from different regions of the human body and face and features of different sizes and use fusion networks for feature fusion and classification, and from different test methods to confirm that this fusion approach for expression recognition and prediction is feasible. This model achieves an average accuracy of 34.51 % in the Emotic contextual expression recognition dataset.

Application of facial expression recognition technology based on feature fusion in teaching

With the development of artificial intelligence technology, the digital transformation of student-oriented education becomes particularly important. How to promote real-time interaction between teachers and students in the classroom is an urgent issue which is needed to pay attention to. Based on the facial expression features of students in a classroom, this paper analyzes the changes in angles between facial expression feature points using Dlib. Additionally, this paper proposes a novel algorithm for extracting variable scale template edge trend features. The algorithm adaptively processes the template based on the edge trend features of expression feature points, and use the proposed template slope normalization algorithm to achieve multi-scale template edge trend extraction. Then, DNN are used to recognize different listening expressions. The experimental results show that the proposed algorithm has faster recognition speed and better robustness when applied to classroom expression recognition. By identifying students’ class status to remind teachers to adjust their class progress, the goal of improving classroom learning effectiveness is achieved

3D Facial Expression Recognition Based on Geometric Feature Fusion

In order to overcome the problems of missing important information and di-mensionality disaster of three-dimensional (3D) facial expression, a novel method based on geometric feature fusion is proposed. This approach initially extracts distance feature vectors and angle feature vectors from key facial regions such as the eyes, mouth, and eyebrows using direct geometric feature. Subsequently, the K-nearest neighbor (K-NN) algorithm is used to obtain the distance feature vector candidate sets and angle feature vector candidate sets separately. Finally, the maximum-minimum rule is utilized to fuse these candidate sets into a single feature vector, completing the recognition of 3D facial expression. Experimental results on the BU-3DFE database demonstrate that this method achieves on overall recognition rate of 97.6%, exhibiting excellent robustness to varia-tions in facial expressions. Furthermore, this approach can serve as a valuable reference for future research in the field of 3D face recognition.

Multi-CNN based logical reasoning system for facial expression recognition on small-sample datasets

Facial expression recognition has long been an area of great interest across a wide range of fields. Deep learning is commonly employed in facial expression recognition and demonstrates excellent performance in large-sample classification tasks. However, deep learning models often encounter challenges when confronted with small-sample expression classification problems, as they struggle to extract sufficient relevant features from limited data, resulting in subpar performance. This paper presents a novel approach called the Multi-CNN Logical Reasoning System, which is based on local area recognition and logical reasoning. It initiates the process by partitioning facial expression images into two distinct components: eye action and mouth action. Subsequently, it utilizes logical reasoning based on the inherent relationship between local actions and global expressions to facilitate facial expression recognition. Throughout the reasoning process, it not only incorporates manually curated knowledge but also acquires hidden knowledge from the raw data. Experimental results conducted on two small-sample datasets derived from the KDEF and RaFD datasets demonstrate that the proposed approach exhibits faster convergence and higher prediction accuracy when compared to classical deep learning-based algorithms.

Time to retire F1-binary score for action unit detection

Detecting action units is an important task in face analysis, especially in facial expression recognition. This is due, in part, to the idea that expressions can be decomposed into multiple action units. To evaluate systems that detect action units, F1-binary score is often used as the evaluation metric. In this paper, we argue that F1-binary score does not reliably evaluate these models due largely to class imbalance. Because of this, F1-binary score should be retired and a suitable replacement should be used. We justify this argument through a detailed evaluation of the negative influence of class imbalance on action unit detection. This includes an investigation into the influence of class imbalance in train and test sets and in new data (i.e., generalizability). We empirically show that F1-micro should be used as the replacement for F1-binary.

Decoding Group Emotional Dynamics in a Web-Based Collaborative Environment: A Novel Framework Utilizing Multi-Person Facial Expression Recognition

As a crucial form of non-verbal communication, facial expressions can convey an individual’s emotional state through a combination of facial muscle movements and provide an effective approach to understanding emotional shifts among group members to a certain extent. However, in real-time collaborative environments, the increasing influx of audiovisual information can overwhelm an individual’s limited visual attention, making it challenging to observe and analyze other group members’ facial expressions continually. The failure to promptly recognize and interpret facial expressions can disrupt individual and group-level comprehension and evaluation of emotional dynamics during collaborative interactions, consequently hindering subsequent emotion management and communication. This article delves into the classification of facial expression information and its formation mechanism to thoroughly analyze the relationship between group members’ emotional perceptions and expressions in the collaboration process. Leveraging the YOLO-FaceV2 face detection model, FaceNet face recognition model, and CERN expression recognition model, we present a novel Collaborative Emotion Analysis Framework (CEAF) for multi-person facial expression recognition. By employing Web real-time communication technology, this framework is integrated into an online group meetings system, which can identify the faces and expressions of the participants and provide real-time visual analysis of their expression information. After the conclusion of the meeting, the system can perform an emotional evaluation with pre-defined operators, providing invaluable insights into the emotional dynamics of the group and individual members throughout the collaborative process. Ultimately, validation through an online meeting instance indicates that this system can facilitate groups in interpreting emotional changes among collaborative group members to a considerable extent.

Human Emotion Recognition Based on Spatio-Temporal Facial Features Using HOG-HOF and VGG-LSTM

Human emotion recognition is crucial in various technological domains, reflecting our growing reliance on technology. Facial expressions play a vital role in conveying and preserving human emotions. While deep learning has been successful in recognizing emotions in video sequences, it struggles to effectively model spatio-temporal interactions and identify salient features, limiting its accuracy. This research paper proposed an innovative algorithm for facial expression recognition which combined a deep learning algorithm and dynamic texture methods. In the initial phase of this study, facial features were extracted using the Visual-Geometry-Group (VGG19) model and input into Long-Short-Term-Memory (LSTM) cells to capture spatio-temporal information. Additionally, the HOG-HOF descriptor was utilized to extract dynamic features from video sequences, capturing changes in facial appearance over time. Combining these models using the Multimodal-Compact-Bilinear (MCB) model resulted in an effective descriptor vector. This vector was then classified using a Support Vector Machine (SVM) classifier, chosen for its simpler interpretability compared to deep learning models. This choice facilitates better understanding of the decision-making process behind emotion classification. In the experimental phase, the fusion method outperformed existing state-of-the-art methods on the eNTERFACE05 database, with an improvement margin of approximately 1%. In summary, the proposed approach exhibited superior accuracy and robust detection capabilities.

Structural self-contrast learning based on adaptive weighted negative samples for facial expression recognition

Structural self-contrast learning based on adaptive weighted negative samples for facial expression recognition

CSINet: Channel–Spatial Fusion Networks for Asymmetric Facial Expression Recognition

Occlusion or posture change of the face in natural scenes has typical asymmetry; however, an asymmetric face plays a key part in the lack of information available for facial expression recognition. To solve the problem of low accuracy of asymmetric facial expression recognition, this paper proposes a fusion of channel global features and a spatial local information expression recognition network called the “Channel–Spatial Integration Network” (CSINet). First, to extract the underlying detail information and deepen the network, the attention residual module with a redundant information filtering function is designed, and the backbone feature-extraction network is constituted by module stacking. Second, considering the loss of information in the local key area of face occlusion, the channel–spatial fusion structure is constructed, and the channel features and spatial features are combined to enhance the accuracy of occluded facial recognition. Finally, before the full connection layer, more local spatial information is embedded into the global channel information to capture the relationship between different channel–spatial targets, which improves the accuracy of feature expression. Experimental results on the natural scene facial expression data sets RAF-DB and FERPlus show that the recognition accuracies of the modeling approach proposed in this paper are 89.67% and 90.83%, which are 13.24% and 11.52% higher than that of the baseline network ResNet50, respectively. Compared with the latest facial expression recognition methods such as CVT, PACVT, etc., the method in this paper obtains better evaluation results of masked facial expression recognition, which provides certain theoretical and technical references for daily facial emotion analysis and human–computer interaction applications.

Multi-Representation Joint Dynamic Domain Adaptation Network for Cross-Database Facial Expression Recognition

In order to obtain more fine-grained information from multiple sub-feature spaces for domain adaptation, this paper proposes a novel multi-representation joint dynamic domain adaptation network (MJDDAN) and applies it to achieve cross-database facial expression recognition. The MJDDAN uses a hybrid structure to extract multi-representation features and maps the original facial expression features into multiple sub-feature spaces, aligning the expression features of the source domain and target domain in multiple sub-feature spaces from different angles to extract features more comprehensively. Moreover, the MJDDAN proposes the Joint Dynamic Maximum Mean Difference (JD-MMD) model to reduce the difference in feature distribution between different subdomains by simultaneously minimizing the maximum mean difference and local maximum mean difference in each substructure. Three databases, including eNTERFACE, FABO, and RAVDESS, are used to design a large number of cross-database transfer learning facial expression recognition experiments. The accuracy of emotion recognition experiments with eNTERFACE, FABO, and RAVDESS as target domains reach 53.64%, 43.66%, and 35.87%, respectively. Compared to the best comparison method chosen in this article, the accuracy rates were improved by 1.79%, 0.85%, and 1.02%, respectively.