SLNL: Soft Label Regularization For Semi-Supervised Facial Expression Recognition With Negative Label Learning

Impaired facial expression recognition in schizophrenia patients contributes to abnormal social functioning and may predict functional outcome in these patients. Facial expression processing involves individual neural networks that have been shown to malfunction in schizophrenia. Whether these patients have a selective deficit in facial expression recognition or a more global impairment in face processing remains controversial. To investigate whether patients with schizophrenia exhibit a selective impairment in facial emotional expression recognition, compared with patients with major depression and healthy control subjects. We studied performance in facial expression recognition and facial sex recognition paradigms, using original morphed faces, in a population with schizophrenia (n=29) and compared their scores with those of depression patients (n=20) and control subjects (n=20). Schizophrenia patients achieved lower scores than both other groups in the expression recognition task, particularly in fear and disgust recognition. Sex recognition was unimpaired. Facial expression recognition is impaired in schizophrenia, whereas sex recognition is preserved, which highly suggests an abnormal processing of changeable facial features in this disease. A dysfunction of the top-down retrograde modulation coming from limbic and paralimbic structures on visual areas is hypothesized.

BackgroundA history of Childhood Trauma (CT), i.e., physical or emotional abuse or neglect, and sexual abuse, is reportedly more prevalent in individuals suffering from psychosis than in the general population. Crucial questions remain unclear about the nature of interpersonal functioning in CT survivors, involving the capacity to understand and interpret other people′s thoughts and feelings, especially in individuals with First-Episode of Schizophrenia (FESz). We investigated the Theory of Mind (ToM) performance of patients with FESz related to CT in comparison to healthy controls (HC).MethodsParticipants (n=77) completed the Eye Task Revised (RMET) and the Childhood Experience of Care Abuse Questionnaire (CECA-Q). The Word Accentuation Test (TAP) was used to estimate a premorbid IQ. Seven-teen patients with FESz (Mean age = 24.9, SD = 5.4, Male = 79.6%; Education = 10.7, SD = 1.5 years) were recruited at the First-Episode Psychosis Program, Hospital 12 de Octubre Madrid, and 60 HC (Mean age = 27.6, SD = 7.2; Male = 45.6%; Education = 14.5, SD = 2.8 years) were healthy volunteers. Between-group comparisons were made using ANCOVA, considering group and CT as fixed factors. Age, years of education and IQ were included as covariates.ResultsPreliminary results showed that compared to controls, patients with FESz performed worse on the recognition and interpretation of facial expressions, in both male and female faces (p < .001). Patients with FESz did not perform differently than HC in the recognition and interpretation of positive facial expressions (p = .074). However, lower interpretation of negative facial expressions (p < .001) and of neutral facial expressions (p < .001) was shown in patients with FESz compared to HC. Higher interpretation of facial expressions was shown in FESz patients with CT (n = 12), only of female faces (p < .001), compared to patients without CT (n = 7). It was also shown higher interpretation of facial expressions in HC with CT (n = 28), only of negative facial expressions (p = .014), compared to HC without CT (n = 32). Female patients with FESz performed worse on the recognition and interpretation of negative (p = .024) and neutral faces (p < .001), only of male faces (p = .038), compared to female HC. Male patients with FESz performed worse on the recognition and interpretation of positive (p = .038) and negative facial expressions (p = .001) of male faces (p < .001), compared to male HC. In comparison to male FESz patients without CT, male FESz patients with CT showed higher interpretation of female faces (p = .030). In comparison to male HC without CT, male HC with CT showed higher interpretation of male faces (p = .031).DiscussionAccording to previous research, our preliminary findings indicated theory of mind deficits in patients with FESz. Interestingly, in our study the alterations on the interpretation and recognition of facial expressions were shown only of negative and neutral, but not of positive facial expressions. Furthermore, and contrary to literature, we found more interpretation and recognition of facial expressions in patients and healthy controls survivors of CT. However, the above-mentioned was specifically observed of female faces in patients and of negative facial expressions in healthy controls. In addition, female and male patients and healthy controls seem to interpret differently facial expressions related to childhood trauma. Nevertheless, increasing our sample size would give us the opportunity to draw further conclusions about how adverse experiences during childhood may influence social abilities in patients with FESz.

Background: Individuals with child maltreatment (CM) experiences show alterations in emotion recognition (ER). However, previous research has mainly focused on populations with specific mental disorders, which makes it unclear whether alterations in the recognition of facial expressions are related to CM, to the presence of mental disorders or to the combination of CM and mental disorders, and on ER of emotional, rather than neutral facial expressions. Moreover, commonly, recognition of static stimulus material was researched. Objective: We assessed recognition of dynamic (closer to real life) negative, positive and neutral facial expressions in individuals characterised by CM, rather than a specific mental disorder. Moreover, we assessed whether they show a negativity bias for neutral facial expressions and whether the presence of one or more mental disorders affects recognition. Methods: Ninety-eight adults with CM experiences (CM+) and 60 non-maltreated (CM−) adult controls watched 200 non-manipulated coloured video sequences, showing 20 neutral and 180 emotional facial expressions, and indicated whether they interpreted each expression as neutral or as one of eight emotions. Results: The CM+ showed significantly lower scores in the recognition of positive, negative and neutral facial expressions than the CM− group (p < .050). Furthermore, the CM+ group showed a negativity bias for neutral facial expressions (p < .001). When accounting for mental disorders, significant effects stayed consistent, except for the recognition of positive facial expressions: individuals from the CM+ group with but not without mental disorder scored lower than controls without mental disorder. Conclusions: CM might have long-lasting influences on the ER abilities of those affected. Future research should explore possible effects of ER alterations on everyday life, including implications of the negativity bias for neutral facial expressions on emotional wellbeing and relationship satisfaction, providing a basis for interventions that improve social functioning.

We proposed a facial motion tracking and expression recognition system based on video data. By a 3D deformable facial model, the online statistical model (OSM) and cylinder head model (CHM) were combined to track 3D facial motion in the framework of particle filtering. For facial expression recognition, a fast and efficient algorithm and a robust and precise algorithm were developed. With the first, facial animation and facial expression were retrieved sequentially. After that facial animation was obtained, facial expression was recognized by static facial expression knowledge learned from anatomical analysis. With the second, facial animation and facial expression were simultaneously retrieved to increase the reliability and robustness with noisy input data. Facial expression was recognized by fusing static and dynamic facial expression knowledge, the latter of which was learned by training a multi-class expressional Markov process using a video database. The experiments showed that facial motion tracking by OSM+CHM is more pose robust than that by OSM, and the facial expression score of the robust and precise algorithm is higher than those of other state-of-the-art facial expression recognition methods.

Recently, deep learning based facial expression recognition (FER) methods have attracted considerable attention and they usually require large-scale labelled training data. Nonetheless, the publicly available facial expression databases typically contain a small amount of labelled data. In this paper, to overcome the above issue, we propose a novel joint deep learning of facial expression synthesis and recognition method for effective FER. More specifically, the proposed method involves a two-stage learning procedure. Firstly, a facial expression synthesis generative adversarial network (FESGAN) is pre-trained to generate facial images with different facial expressions. To increase the diversity of the training images, FESGAN is elaborately designed to generate images with new identities from a prior distribution. Secondly, an expression recognition network is jointly learned with the pre-trained FESGAN in a unified framework. In particular, the classification loss computed from the recognition network is used to simultaneously optimize the performance of both the recognition network and the generator of FESGAN. Moreover, in order to alleviate the problem of data bias between the real images and the synthetic images, we propose an intra-class loss with a novel real data-guided back-propagation (RDBP) algorithm to reduce the intra-class variations of images from the same class, which can significantly improve the final performance. Extensive experimental results on public facial expression databases demonstrate the superiority of the proposed method compared with several state-of-the-art FER methods.

This study presents a new color-depth based face database gathered from different genders and age ranges from Iranian subjects. Using suitable databases, it is possible to validate and assess available methods in different research fields. This database has application in different fields such as face recognition, age estimation and Facial Expression Recognition and Facial Micro Expressions Recognition. Image databases based on their size and resolution are mostly large. Color images usually consist of three channels namely Red, Green and Blue. But in the last decade, another aspect of image type has emerged, named “depth image”. Depth images are used in calculating range and distance between objects and the sensor. Depending on the depth sensor technology, it is possible to acquire range data differently. Kinect sensor version 2 is capable of acquiring color and depth data simultaneously. Facial expression recognition is an important field in image processing, which has multiple uses from animation to psychology. Currently, there is a few numbers of color-depth (RGB-D) facial micro expressions recognition databases existing. With adding depth data to color data, the accuracy of final recognition will be increased. Due to the shortage of color-depth based facial expression databases and some weakness in available ones, a new and almost perfect RGB-D face database is presented in this paper, covering Middle-Eastern face type. In the validation section, the database will be compared with some famous benchmark face databases. For evaluation, Histogram Oriented Gradients features are extracted, and classification algorithms such as Support Vector Machine, Multi-Layer Neural Network and a deep learning method, called Convolutional Neural Network or are employed. The results are so promising.

Face processing and efficient recognition of facial expressions are impaired following right but not left anteromedial temporal lobe resections: Behavioral and fMRI evidence

Recognition of facial expression has many useful applications that have drawn researcher’s interest over the past decade. Extraction of features is a major step in the analysis of expression which leads to fast and accurate recognition of expression. Recognition of facial expressions is not an easy issue for methods of machine learning, as different people can vary in the way they show their expressions and for one expression the image of the same person can differ for brightness, background and position. Recognition of facial expression is therefore still a challenging computer vision problem. In this thesis work, we aim to design a robust technique of automatic facial expression analysis and recognition using zone based active and salient patches of the human face by combining various techniques from computer vision and pattern recognition. Expression recognition is closely related to face recognition where a lot of research has been done and a vast array of algorithms have been introduced. Facial expression recognition (FER) can also be considered as a special case of a pattern recognition problem and many techniques are available. In the designing of an FER system, we divided the system into 4 modules, i.e. preprocessing, active and salient patch extraction and classification. Voila Jones algorithm is used for face detection and after that features are extracted from the facial patches. The active facial patches are located on the facial regions that during different expressions undergo a major change. The active patches are located after detection of facial landmarks and hybrid features are determined from these patches. The use of small parts of face instead of the whole face for extracting features reduces the computational cost. Zoning is applied and got remarkable results. The dimensionality of the function is reduced by using linear discriminant analysis, which is further defined using the support vector machine (SVM). On the basis of classification expression is recognized. We evaluated our algorithm on Extended Cohn-Kanade (CK+) dataset.

Facial expression analysis and recognition could help humanize computers and design a new generation of human computer interface. A number of techniques were successfully exploited for facial expression recognition (Chang et al., 2004; Cohen et al., 2004; Cohen et al., 2003; Gu & Ji, 2004; and Littlewort et al, 2004), including feature estimation by optical flow (Mase, 1999; Yacoob & S. Davis, 2006), dynamic model ( Essa & Pentland, 1997), eigenmesh method (Matsuno et all.) and neural networks (Rosenblum et all., 1996). The excellent review of recent advances in this field can be found in (Y. Tian et al., 2001; Pantic & Rothkrantz, 2000; Zhao et al., 2000). The conventional methods on facial expression recognition concern themselves with extracting the expression data to describe the change of facial features, such as Action Units (AUs) defined in Facial Action Coding System (FACS) (Donato et al., 1999). Although the FACS is the most successful and commonly used technique for facial expression representation and recognition, the difficulty and complexity of the AUs extraction limit its application. As quoted by most previous works (Essa & Pentland, 1997; Yacoob & Davis, 1996), capturing the subtle change of facial skin movements is a dif cult task due to the difficulty to implement such an implicit representation. Currently, feature-based approaches (Reinders et al., 1995; Terzopoulos & Waters, 1993) and spatio-temporal based approaches (Essa & Pentland) are commonly used. Yacoob & Davis, 1996 integrated spatial and temporal information and studied the temporal model of each expression for the recognition purpose, a high recognition rate was achieved. Colmenarez et al. used a probabilistic framework based on the facial feature position and appearances to recognize the facial expressions, the recognition performance was improved, but only the feature regions other than the surface information were explored. Recently, Tian, Kanade and Cohn (Tian et al., 2001) noticed the importance of the transient features (i.e., furrow information) besides the permanent features (i.e, eyes, lips and brows) in facial expression recognition. They explored the furrow information for improving the accuracy of the AU parameters, an impressive result was achieved in recognizing a large variety of subtle expressions. To our knowledge, little investigation has been conducted on combining texture analysis and surface structure analysis for modeling and recognizing facial expressions. A detailed higher level face representation and tracking system is in high demand. In this paper, we explore the active texture information and facial surface features to meet the challenge – modeling the facial expression with sufficient accuracy.

An Intelligent Biometrics systems aims at localizing and detecting human faces from supplied images so that further recognition of persons and their facial expression recognition will be easy. The area of human-computer interaction (HCI) will be much more effective if a computer is able to recognize the emotional state of human being. Emotional states have a greater effect on the face which can tell about mood of a person. So if we can recognize facial expressions, we will know something about the human’s emotions and mood. This paper focuses on the novel Hybrid Facial Geometry Algorithm (HFGA) and comparative analysis of Facial Geometry algorithm and HFGA for facial feature extraction and its use to classify facial expressions. Feed forward back propagation neural network (BPNN) and Adaptive Neuro-Fuzzy Inference System (ANFIS) are used as classifiers for expression classification and recognition. Experimentations are carried out using Japanese Female Facial Expression (JAFFE) database. Experimental results shows that average recognition efficiency from 95.33% to 93.33% is achieved for 30 to 75 test samples using BPNN and 95.71% to 95.33% with ANFIS approach.

The perception and recognition of facial expressions are crucial for parenting. This study investigated whether and how maternal nurturing experience and trait anxiety influence the perception and recognition of infant and adult facial expressions. This was assessed by comparing the performance of primiparous mothers (n = 25) and non-mothers (n = 28) on an emotional face perception task. Trait anxiety was measured using the Japanese version of the State-Trait Anxiety Inventory (STAI). We found that mothers had higher recognition accuracy for facial expressions, but only of adults, not infants. Moreover, as trait anxiety increased, so did mothers’ sensitivity in perceiving facial expressions of both infants and adults. These findings suggest that maternal nurturing experience does enhance the recognition of adult emotional expressions, and an optimal level of maternal trait anxiety may enhance mothers’ sensitivity toward infants’ and adults’ emotional signals.

The role of the right prefrontal cortex in recognition of facial emotional expressions in depressed individuals: fNIRS study

The Latent Dirichlet Allocation (LDA) is a model proposed recently which extracts latent topics from text data. The paper uses the LDA model in facial expression recognition but not document recognition to excavate the distribution relations between the Action Unit (AU) and expressions. Using the LDA to set up a model between AU and the facial expression of distribution relations, we can use the LDA model to ensure the unknown category of expression sequence of proportion which belongs to six basic expressions. This is the first LDA-based solution to facial complex expression recognition. We have validated that the complex facial expression recognition based on the LDA model can be used and is reasonable.

With the work and study pressure on people increasing and the importance of mental health problems on the rise, facial expression analysis plays an important role in mental health auxiliary diagnosis and treatment (MHADT). This study proposes a facial expression localization and recognition model that integrates Main Directional Mean Optical Flow (MDMO) with the Transformer architecture. It addresses the problems of insufficient generalization ability and limited temporal modeling ability of traditional methods in facial expression recognition. The study is based on the authoritative Audio/Visual Emotion Challenge 2019 Depression Detection Sub-challenge (AVEC2019 DDS) dataset in the mental health field, which contains 163 training samples, 56 validation samples, and 56 test samples. With Explicit Shape Regression, Local Binary Patterns, Mnemonic Descent Method, Convolutional Neural Network, and benchmark models as comparison objects, it systematically evaluates the model's performance in error, accuracy, and processing speed. The results show that the model achieves the best performance in both facial expression localization and recognition tasks. The validation set errors are 7.26 and 6.85, the localization accuracy reaches 89.6%, and the recognition accuracy reaches 88.9%, which is significantly better than other methods. At the same time, its image processing time is 55 milliseconds (ms) and 44ms, balancing high precision and real-time performance. The study indicates that the fusion of MDMO and Transformer can effectively capture the spatial and temporal features of facial expressions. Thus, this fusion method provides an efficient, stable, and scalable technical solution for emotion recognition in MHADT. This improves both the FER and localization accuracy and effect. Besides, it provides a novel reference in both approaches and application level for the intelligent development of mental health evaluation.

Pediatric brain tumor survivors (PBTS) experience deficits in social functioning. Facial expression and identity recognition are key components of social information processing and are widely studied as an index of social difficulties in youth with autism spectrum disorder (ASD) and other neurodevelopmental conditions. This study evaluated facial expression and identity recognition among PBTS, youth with ASD, and typically developing (TD) youth, and the associations between these face processing skills and social impairments. PBTS (N = 54; ages 7-16) who completed treatment at least 2 years prior were matched with TD (N = 43) youth and youth with ASD (N = 55) based on sex and IQ. Parents completed a measure of social impairments and youth completed a measure of facial expression and identity recognition. Groups significantly differed on social impairments (p < .001), with youth with ASD scoring highest followed by PBTS and lastly TD youth. Youth with ASD performed significantly worse on the two measures of facial processing, while TD youth and PBTS were not statistically different. The association of facial expression recognition and social impairments was moderated by group, such that PBTS with higher levels of social impairment performed worse on the expression task compared to TD and ASD groups (p < .01, η2 = 0.07). Variability in face processing may be uniquely important to the social challenges of PBTS compared to other neurodevelopmental populations. Future directions include prospectively examining associations between facial expression recognition and social difficulties in PBTS and face processing training as an intervention for PBTS.