Facial Expression Modeling Research Articles

An Empirical Study of Portrait Rights on Emotional Evolution in Virtual Social Scenarios by Transformer and Cloud Computing

Facial expression recognition technology can improve the possibility of fugitives and persons subject to execution being discovered by public security and judicial personnel in the virtual world, thereby improving the actual effectiveness and credibility of the law. There may also be a risk of portrait rights infringement with this technology, and the user of the technology needs to inform the person whose facial expression is being extracted in advance and obtain permission. With the rapid progression of deep learning and artificial intelligence, 3D facial expression modeling has garnered increased significance in computer vision and graphics. We propose an innovative approach combining Transformer models with cloud computing to simulate facial expression evolution in virtual social environments. Leveraging Transformer-based feature extraction, our model integrates emotional cues from various modalities to accurately capture subtle changes over time.

Machine learning model of facial expression outperforms models using analgesia nociception index and vital signs to predict postoperative pain intensity: a pilot study.

Few studies have evaluated the use of automated artificial intelligence (AI)-based pain recognition in postoperative settings or the correlation with pain intensity. In this study, various machine learning (ML)-based models using facial expressions, the analgesia nociception index (ANI), and vital signs were developed to predict postoperative pain intensity, and their performances for predicting severe postoperative pain were compared. In total, 155 facial expressions from patients who underwent gastrectomy were recorded postoperatively; one blinded anesthesiologist simultaneously recorded the ANI score, vital signs, and patient self-assessed pain intensity based on the 11-point numerical rating scale (NRS). The ML models' area under the receiver operating characteristic curves (AUROCs) were calculated and compared using DeLong's test. ML models were constructed using facial expressions, ANI, vital signs, and different combinations of the three datasets. The ML model constructed using facial expressions best predicted an NRS ≥ 7 (AUROC 0.93) followed by the ML model combining facial expressions and vital signs (AUROC 0.84) in the test-set. ML models constructed using combined physiological signals (vital signs, ANI) performed better than models based on individual parameters for predicting NRS ≥ 7, although the AUROCs were inferior to those of the ML model based on facial expressions (all P < 0.050). Among these parameters, absolute and relative ANI had the worst AUROCs (0.69 and 0.68, respectively) for predicting NRS ≥ 7. The ML model constructed using facial expressions best predicted severe postoperative pain (NRS ≥ 7) and outperformed models constructed from physiological signals.

Hybridization of Deep Sequential Network for Emotion Recognition Using Unconstraint Video Analysis

The reliable way to discern human emotions in various circumstances has been proven to be through facial expressions. Facial expression recognition (FER) has emerged as a research topic to identify various essential emotions in the present exponential rise in research for emotion detection. Happiness is one of these basic emotions everyone may experience, and facial expressions are better at detecting it than other emotion-measuring methods. Most techniques have been designed to recognize various emotions to achieve the highest level of general precision. Maximizing the recognition accuracy for a particular emotion is challenging for researchers. Some techniques exist to identify a single happy mood recorded in unrestricted video. Still, they are all limited by the processing of extreme head posture fluctuations that they need to consider, and their accuracy still needs to be improved. This research proposes a novel hybrid facial emotion recognition using unconstraint video to improve accuracy. Here, a Deep Belief Network (DBN) with long short-term memory (LSTM) is employed to extract dynamic data from the video frames. The experiments conducted uses decision-level and feature-level fusion techniques are applied unconstrained video dataset. The outcomes show that the proposed hybrid approach may be more precise than some existing facial expression models.

Characteristics of healthy Japanese young adults with respect to recognition of facial expressions: a preliminary study

BackgroundEmotional cognitive impairment is a core phenotype of the clinical symptoms of psychiatric disorders. The ability to measure emotional cognition is useful for assessing neurodegenerative conditions and treatment responses. However, certain factors such as culture, gender, and generation influence emotional recognition, and these differences require examination. We investigated the characteristics of healthy young Japanese adults with respect to facial expression recognition.MethodsWe generated 17 models of facial expressions for each of the six basic emotions (happiness, sadness, anger, fear, disgust, and surprise) at three levels of emotional intensity using the Facial Acting Coding System (FACS). Thirty healthy Japanese young adults evaluated the type of emotion and emotional intensity the models represented to them.ResultsAssessment accuracy for all emotions, except fear, exceeded 60% in approximately half of the videos. Most facial expressions of fear were rarely accurately recognized. Gender differences were observed with respect to both faces and participants, indicating that expressions on female faces were more recognizable than those on male faces, and female participants had more accurate perceptions of facial emotions than males.ConclusionThe videos used may constitute a dataset, with the possible exception of those that represent fear. The subject’s ability to recognize the type and intensity of emotions was affected by the gender of the portrayed face and the evaluator’s gender. These gender differences must be considered when developing a scale of facial expression recognition.

Testing, explaining, and exploring models of facial expressions of emotions.

Models are the hallmark of mature scientific inquiry. In psychology, this maturity has been reached in a pervasive question-what models best represent facial expressions of emotion? Several hypotheses propose different combinations of facial movements [action units (AUs)] as best representing the six basic emotions and four conversational signals across cultures. We developed a new framework to formalize such hypotheses as predictive models, compare their ability to predict human emotion categorizations in Western and East Asian cultures, explain the causal role of individual AUs, and explore updated, culture-accented models that improve performance by reducing a prevalent Western bias. Our predictive models also provide a noise ceiling to inform the explanatory power and limitations of different factors (e.g., AUs and individual differences). Thus, our framework provides a new approach to test models of social signals, explain their predictive power, and explore their optimization, with direct implications for theory development.

Retracted: Impact of Activation, Optimization, and Regularization Methods on the Facial Expression Model Using CNN

Retracted: Impact of Activation, Optimization, and Regularization Methods on the Facial Expression Model Using CNN

EMOSENSE – Multi-Modal Emotion Recognition to Identify Emotions

An extended study has been done over the past years to better comprehend human emotions. The embracement of technology to recognize and react to human emotions has become a required component of society. We present a fully functional multi-modal emotion recognition system in this study that integrates data from text, voice, facial expressions, and body language. In this study, the automatic classification of anger, fear, joy, sadness, surprise, disgust, and neutral emotions from text, facial expressions, voice, and body movements have been studied on the TESS, MELD, FER2013, and EDNLP datasets. Random Forest Classifier has been used for the classification of emotions using body language, VGG16 pre-trained model for facial emotion classification, Logistic Resgression for text emotion classification, and CNN for voice emotion classification. The logistic regression model for text emotion prediction leverages natural language processing (NLP) techniques to extract emotions from textual data. The CNN-based voice model utilizes speech recognition and emotion recognition algorithms to analyze audio signals and detect emotional cues in the speaker's voice. The facial expression model employs a combination of CNN-based VGG16 pre-trained model and modified convolutional layers to detect emotions. Meanwhile, the Random Forest Clasifier model is used to capture and interpret non-verbal cues, such as gestures, posture, and overall body movements, to enrich the emotion detection process. The real strength of our proposed system lies in its ability to synergistically combine information from multiple modalities.

Modeling and Analysis of Discrete Facial Expressions with Dense Optical Flow-derived Features

Facial expressions (FEs) are one of the most preeminent means of conveying one’s emotions and are pivotal to nonverbal communication. Potential applications in a wide range of areas in computer vision have lent a strong impetus to research in the domain of automatic facial expression recognition. This work discusses the effectiveness of two optical flow-based features for modeling the FEs associated with prototypic emotions based on the pattern of nonrigid deformable motion of facial components occurring during their portrayal. The discernible motion patterns are categorized into distinct discrete classes with the descriptive features indicating the global spatial distribution of deformation derived from the dense optical flow field associated with emotional and neutral face images. Results obtained with evaluation on images and video clips taken from Extended Cohn-Kanade, Japanese Female Facial Expressions, and Dynamic Karolinska Directed Emotional Faces datasets with multi-class support vector machine and [Formula: see text]-nearest neighbor classifiers are competent with the state-of-the-art techniques and concordant with empirical psychological studies in emotion science.

Looking fear in the face: Adults but not adolescents gaze at social threat during observational learning

Observational threat learning is an indirect pathway to learn about what is safe versus dangerous. Visual gaze patterns may be an important measure to understand the underlying mechanisms of social threat learning. However, little research has considered this type of learning or attention allocation during observational learning, and even less has examined it across development. The study examined visual gaze patterns during observational threat acquisition amongst adolescents and adults. Ninety-three adolescents (13–17 years) and 78 adults (18–34 years) underwent a differential observational acquisition task by watching a video wherein a learning model was presented with colored bells. One bell (conditioned stimulus, CS+) was associated with an electric stimulation to the learning model's arm (social unconditioned stimulus, social US), while the other bell was not (CS-). Eye tracking was used during each trial. First, all participants, regardless of age, fixated longer on the face of the learning model when the CS+ was presented than when the CS- was presented. Second, adolescents averted their gaze from the learning model's face when the learning model received an electrical stimulation, whereas adults fixated more on the learning model's face when the stimulation was administered. Finally, adolescents understood the CS+-US contingency less than adults, stemming from the different gaze pattern for the age groups to the social US. Results replicate previous findings in adults and extend them to adolescents, emphasizing the importance of the learning model's facial expressions in conveying information on what is safe versus dangerous in the environment. Moreover, results showed developmental differences in gaze patterns during observational threat learning; this translated into poorer understanding of the CS+-US association amongst adolescents.

Impact of Activation, Optimization, and Regularization Methods on the Facial Expression Model Using CNN.

When it comes to conveying sentiments and thoughts, facial expressions are quite effective. For human-computer collaboration, data-driven animation, and communication between humans and robots to be successful, the capacity to recognize emotional states in facial expressions must be developed and implemented. Recently published studies have found that deep learning is becoming increasingly popular in the field of image categorization. As a result, to resolve the problem of facial expression recognition (FER) using convolutional neural networks (CNN), increasingly substantial efforts have been made in recent years. Facial expressions may be acquired from databases like CK+ and JAFFE using this novel FER technique based on activations, optimizations, and regularization parameters. The model recognized emotions such as happiness, sadness, surprise, fear, anger, disgust, and neutrality. The performance of the model was evaluated using a variety of methodologies, including activation, optimization, and regularization, as well as other hyperparameters, as detailed in this study. In experiments, the FER technique may be used to recognize emotions with an Adam, Softmax, and Dropout Ratio of 0.1 to 0.2 when combined with other techniques. It also outperforms current FER techniques that rely on handcrafted features and only one channel, as well as has superior network performance compared to the present state-of-the-art techniques.

Online English Teaching Quality Assessment Based on K-Means and Improved SSD Algorithm

Classroom teaching quality is a key content to measure the teaching level, and the teaching effect can be intuitively reflected from the students’ listening state. In order to improve the teaching quality, this paper proposes an online English teaching quality evaluation model based on K-means and an improved SSD algorithm. In the SSD algorithm, the backbone network is replaced by DenseNet with a dense connection to improve detection accuracy. The network structure of quadratic regression is designed to solve the problem of unbalance between positive and negative samples in the default box of the candidate region. A feature graph scaling method is used to fuse feature graphs without introducing additional parameters. The number of default boxes and the optimal aspect ratio were obtained by k-means clustering analysis. Finally, the state of students in the teaching process is predicted through the dual-mode recognition model of facial expression and posture, and the state of students in class is judged. Experimental comparison and analysis were conducted on the public data set and a self-built classroom teaching video data set. Experiments show that compared with other comparison algorithms, the algorithm presented in this paper performs better in the index of detection accuracy.

Improving the Facial Expression Recognition and Its Interpretability via Generating Expression Pattern-map

Facial expression recognition focuses on extracting expression-related features on a face. In this paper, a novel method is proposed for facial expression modeling based on the following two aspects: seeking expression-related regions more accurately, and enhancing expression features more discriminating. To this end, we design a model containing three submodules: the Expression Feature Extractor (EFE), the Expression Mask Refiner (EMR), and the Expression Pattern-Map Generator (EPMG). The EFE module is the backbone that extracts expression features and generates a coarse attention mask which roughly indicates expression-related regions. The EMR module refines the mask to be more precise by modeling the relationship among expression-related regions, and generates the masked features. The EPMG module utilizes the masked features to further model the fusion and extraction process which obtains a compact and discriminating expression-salient embedding for recognition, and generates an expression pattern-map. We propose the concept of the expression pattern-map, which provides a unified visualization of expression features and improves the interpretability of facial expression recognition. Our model is evaluated on four public datasets (CK+, Oulu-CASIA, RAF-DB, AffectNet), and achieves the competitive performance compared with the state-of-the-art.

Review on facial expression modeling

Facial modeling has been an ongoing research for many years and still shows research trend due to its relevance to current technology. Many applications incorporate facial expression modeling with the help of facial tracking, facial animation and facial recognition. The existing performance of the modeling method faces the challenges to perform well due to many factors. Currently, the use of 2D images and videos as inputs for modeling process are gaining popularity. However, current technologies and development had extended the trends towards acquiring 3D human data. This paper provides an overview on variety of modeling techniques based on human facial model that can lead to future research.

Customized Facial Expression Analysis in Video

<i>Existing computer vision-based emotion recognition systems are trained to classify images of faces into a very limited number of emotions. In this work, we take a different approach and train a convolutional neural network (CNN) that is able to distinguish subtle differences in facial expressions appearing in both individual images and videos. For this effect, we train a feature embedding network that maps a facial image into a position in embedding space, such that it is positioned close to similar facial expressions compared with other expressions. We train the feature embedding network using triplet loss on the publicly available facial expression comparison (FEC) data set. The proposed facial expression model is lightweight (4.7M parameters) and obtains a triplet prediction accuracy of 84.5&#x0025;&#x2014;very close to the average human performance of 86.2&#x0025;. A key contribution of this work is adapting the model that has been trained on images to work reliably on video content. To this end, we propose a novel automatic face quality assessment (FaceQA), which allows us to filter out faces that are unsuited for expression analysis. We validate the proposed approach on several applications, including searching video content for a specific expression from a single query face, generating facial expression statistics of various actors appearing in a video, as well as generating facial expression summaries</i>.

High-fidelity 3D real-time facial animation using infrared structured light sensing system

The acquisition of 3D facial models is crucial in the gaming and film industries. In this study, we developed a facial acquisition system based on infrared structured light sensors to obtain facial-expression models with high fidelity and accuracy. First, we employed time-multiplexing structured light to obtain an accurate and dense point cloud. The template model was then warped to the captured facial expression. A model-tracking method based on optical flow was applied to track the registered 3D model displacement. Finally, the live 3D mesh was textured using high-resolution images captured by three color cameras. Experiments were conducted on human faces to demonstrate the performance of the proposed system and methods.

Facial Expression Modeling and Synthesis for Patient Simulator Systems: Past, Present, and Future

Clinical educators have used robotic and virtual patient simulator systems (RPS) for dozens of years, to help clinical learners (CL) gain key skills to help avoid future patient harm. These systems can simulate human physiological traits; however, they have static faces and lack the realistic depiction of facial cues, which limits CL engagement and immersion. In this article, we provide a detailed review of existing systems in use, as well as describe the possibilities for new technologies from the human–robot interaction and intelligent virtual agents communities to push forward the state of the art. We also discuss our own work in this area, including new approaches for facial recognition and synthesis on RPS systems, including the ability to realistically display patient facial cues such as pain and stroke. Finally, we discuss future research directions for the field.

Recognition method of unspecified face expressions based on machine learning

Traditional face recognition methods usually complete facial expression recognition for designated faces, and the pixel set at the edge of face image is chaotic, which leads to poor accuracy of unspecified facial expression recognition. In order to improve the accuracy of unknown facial expression recognition, a method of unknown facial expression recognition based on machine learning is proposed. The feature detection model of unspecified facial expressions is constructed, and the features are divided into regional blocks. Fusion block feature information establishes a spatial feature projection model, weights the feature information entropy, extracts statistical features and edge information entropy features, reorganises features and matching edge pixel sets and completes the recognition of various facial expression features. Experimental results show that the accuracy of this method is significant, reaching 1, which effectively improves the recognition efficiency and anti-interference performance.

Altering sensorimotor simulation impacts early stages of facial expression processing depending on individual differences in alexithymic traits

Simulation models of facial expressions suggest that posterior visual areas and brain areas underpinning sensorimotor simulations might interact to improve facial expression processing. According to these models, facial mimicry, a manifestation of sensorimotor simulation, may contribute to the visual processing of facial expressions by influencing early stages. The aim of this study was to assess whether and how sensorimotor simulation influences early stages of face processing, also investigating its relationship with alexithymic traits given that previous studies have suggested that individuals with high levels of alexithymic traits (vs. individuals with low levels of alexithymic traits) tend to use sensorimotor simulation to a lesser extent. We monitored P1 and N170 ERP components of the event-related potentials (ERP) in participants performing a fine discrimination task of facial expressions and animals, as a control condition. In half of the experiment, participants could freely use their facial mimicry whereas in the other half they had their facial mimicry blocked by a gel. Our results revealed that only individuals with lower compared to high alexithymic traits showed a larger modulation of the P1 amplitude as a function of the mimicry manipulation selectively for facial expressions (but not for animals), while we did not observe any modulation of the N170. Given the null results at the behavioural level, we interpreted the P1 modulation as compensative visual processing in individuals with low levels of alexithymia under conditions of interference on the sensorimotor processing, providing a preliminary evidence in favor of sensorimotor simulation models.

Happy Emotion Recognition From Unconstrained Videos Using 3D Hybrid Deep Features

Facial expressions have been proven to be the most effective way for the brain to recognize human emotions in a variety of contexts. With the exponentially increasing research for emotion detection in recent years, facial expression recognition has become an attractive, hot research topic to identify various basic emotions. Happy emotion is one of such basic emotions with many applications, which is more likely recognized by facial expressions than other emotion measurement instruments (e.g., audio/speech, textual and physiological sensing). Nowadays, most methods have been developed for identifying multiple types of emotions, which aim to achieve the best overall precision for all emotions; it is hard for them to optimize the recognition accuracy for single emotion (e.g., happiness). Only a few methods are designed to recognize single happy emotion captured in the unconstrained videos; however, their limitations lie in that the processing of severe head pose variations has not been considered, and the accuracy is still not satisfied. In this paper, we propose a Happy Emotion Recognition model using the 3D hybrid deep and distance features (HappyER-DDF) method to improve the accuracy by utilizing and extracting two different types of deep visual features. First, we employ a hybrid 3D Inception-ResNet neural network and long-short term memory (LSTM) to extract dynamic spatial-temporal features among sequential frames. Second, we detect facial landmarks’ features and calculate the distance between each facial landmark and a reference point on the face (e.g., nose peak) to capture their changes when a person starts to smile (or laugh). We implement the experiments using both feature-level and decision-level fusion techniques on three unconstrained video datasets. The results demonstrate that our HappyER-DDF method is arguably more accurate than several currently available facial expression models.

Multi-Feature Tensor Neighborhood Preserving Embedding for 3D Facial Expression Recognition

To investigate an effective representation model for 3D facial expression recognition, this paper proposes a multi-feature tensor neighborhood preserving embedding (MFTNPE) method, which seeks various attribute features from raw textured shape scan models of 3D facial expression to construct a multi-feature tensor model. With MFTNPE method, we can project the model into a tensor subspace, explore the interactions and correlations among multimodal data and then extract discriminative low-dimensional tensor features for 3D facial expression recognition. In the study, we first introduce a tensor model based on multi-feature integration for 3D facial expression recognition. Via exploring the textural and geometric features of 3D facial expression data, an original multi-feature tensor model is constructed by stacking various attribute maps orderly. Second, we utilize the tensor completion approach as the scheme of data preprocessing to correct the original multi-feature tensor model. Due to the presence of imperfect data in the projected attribute maps, the tensor completion can predict the missing data in tensor by the trace norm of multi-feature tensor model along each mode. Third, we redefine a weighted matrix for use in the orthogonal-based tensor neighborhood preserving embedding algorithm. In the weighted matrix, the class information is used to guarantee the correct neighborhood relationships for multi-feature tensor samples. This effectively avoids the disturbances and confusions between similar samples while preserving spatial structure and inter-class information. Finally, we conduct extensive experiments on two 3D facial expression recognition datasets. The experimental results show that the proposed method has much better efficiency and performance compared with the state-of-the-art methods.