Action Unit Research Articles

Feature fusion for human compound emotion recognition: a fusion of facial expression texture and action unit data

Feature fusion for human compound emotion recognition: a fusion of facial expression texture and action unit data

Orangutans and chimpanzees produce morphologically varied laugh faces in response to the age and sex of their social partners

Laugh faces of humans play a key role in everyday social interactions as a pervasive tool of communication across contexts. Humans often vary the degree of mouth opening and teeth exposure when producing these facial expressions, which may depend on who their social partner is (e.g., their gender and age as well as their social relationship), serving this way different functions. Although it was found that laugh faces show evolutionary continuity across humans and non-human great apes according to the Principle of Maximum Parsimony, little is known about the function of laugh face variations from an evolutionary perspective. Hence, the present work examined the morphology of laugh faces in orangutan and chimpanzee dyadic play to test if they are modified with dependence on the playmate’s characteristics (sex, age and social relationship). In total, we analysed over 600 facial expressions of 14 orangutans and 17 chimpanzees by coding the specific muscle activations (Action Units, i.e. AUs) contributing to these expressions, using OrangFACS and ChimpFACS, respectively. Our results suggest that age difference and, to a lesser extent, playmate sex influence laugh face morphology in both taxa, but in opposite ways. While the orangutans of our study seem to expose their upper teeth (with AU10) and to pull the mouth corners (with AU12) more towards weaker partners (younger and female), possibly to communicate non-hostility, the chimpanzees showed both upper and lower teeth exposure (with AU10 and AU16) more often when interacting with the stronger partners (older individuals), possibly to communicate submissiveness. These findings suggest that the ability of humans to modify laugh faces with dependence on social partner characteristics has most likely evolved from pre-existing traits, going back at least to the last common ancestor of today’s great apes, including humans.

Observer-generated maps of diagnostic facial features enable categorization and prediction of emotion expressions

According to one prominent model, facial expressions of emotion can be categorized into depicting happiness, disgust, anger, sadness, fear and surprise. One open question is which facial features observers use to recognize the different expressions and whether the features indicated by observers can be used to predict which expression they saw.We created fine-grained maps of diagnostic facial features by asking participants to use mouse clicks to highlight those parts of a face that they deem useful for recognizing its expression. We tested how well the resulting maps align with models of emotion expressions (based on Action Units) and how the maps relate to the accuracy with which observers recognize full or partly masked faces.As expected, observers focused on the eyes and mouth regions in all faces. However, each expression deviated from this global pattern in a unique way, allowing to create maps of diagnostic face regions. Action Units considered most important for expressing an emotion were highlighted most often, indicating their psychological validity. The maps of facial features also allowed to correctly predict which expression a participant had seen, with above-chance accuracies for all expressions. For happiness, fear and anger, the face half which was highlighted the most was also the half whose visibility led to higher recognition accuracies.The results suggest that diagnostic facial features are distributed in unique patterns for each expression, which observers seem to intuitively extract and use when categorizing facial displays of emotion.

Employing the Artificial Intelligence Object Detection Tool YOLOv8 for Real-Time Pain Detection: A Feasibility Study.

Effective pain management is crucial for patient care, impacting comfort, recovery, and overall well-being. Traditional subjective pain assessment methods can be challenging, particularly in specific patient populations. This research explores an alternative approach using computer vision (CV) to detect pain through facial expressions. The study implements the YOLOv8 real-time object detection model to analyze facial expressions indicative of pain. Given four pain datasets, a dataset of pain-expressing faces was compiled, and each image was carefully labeled based on the presence of pain-associated Action Units (AUs). The labeling distinguished between two classes: pain and no pain. The pain category included specific AUs (AU4, AU6, AU7, AU9, AU10, and AU43) following the Prkachin and Solomon Pain Intensity (PSPI) scoring method. Images showing these AUs with a PSPI score above 2 were labeled as expressing pain. The manual labeling process utilized an open-source tool, makesense.ai, to ensure precise annotation. The dataset was then split into training and testing subsets, each containing a mix of pain and no-pain images. The YOLOv8 model underwent iterative training over 10 epochs. The model's performance was validated using precision, recall, and mean Average Precision (mAP) metrics, and F1 score. When considering all classes collectively, our model attained a mAP of 0.893 at a threshold of 0.5. The precision for "pain" and "nopain" detection was 0.868 and 0.919, respectively. F1 scores for the classes "pain", "nopain", and "all classes" reached a peak value of 0.80. Finally, the model was tested on the Delaware dataset and in a real-world scenario. Despite limitations, this study highlights the promise of using real-time computer vision models for pain detection, with potential applications in clinical settings. Future research will focus on evaluating the model's generalizability across diverse clinical scenarios and its integration into clinical workflows to improve patient care.

Stress recognition identifying relevant facial action units through explainable artificial intelligence and machine learning

Stress recognition identifying relevant facial action units through explainable artificial intelligence and machine learning

Depression Classification Based on Facial Action Unit Intensity Features Using CNN-Poolingless Framework

Depression Classification Based on Facial Action Unit Intensity Features Using CNN-Poolingless Framework

Development of the calf grimace scale for pain and stress assessment in castrated Angus beef calves

Grimace scales have been used to assess pain in various animal species. This study aimed to develop the calf grimace scale (CGS), evaluate its responsiveness and the effect of external factors (change of environment and dam separation, and restraint) on CGS. Sixty-nine Angus calves, 6–8 weeks old, were randomly allocated into castrated (n = 34) and sham castrated (n = 35) groups. Images were extracted from videos pre- (M1-M4), during- (M5), and post-castration/sham castration (M6, M7). Six facial action units (FAUs) were identified: ear position, orbital tightening, tension above the eye, nostril dilation, straining of chewing muscle, and mouth opening. Final CGS median scores increased after castration (P < 0.001) for both non-restrained (M7 versus M2) and restrained (M6 versus M3) calves, indicating scale responsiveness. Final CGS median scores increased (P < 0.001) when calves were subjected to external factors before castration (M1 [baseline] versus M2 and M3). However, there was no difference (P > 0.05) in CGS median scores before and after sham castration, regardless of restraint (M3 versus M6, and M2 versus M7), indicating that the external factors may have reached a maximum effect. The CGS is composed of six FAUs, responsive to acute pain and can identify stress unrelated to pain.

How Wanghong Live Streamers Affect Online Users’ Purchasing Behavior on Live Streaming Platforms: A PLS-SEM Approach

ABSTRACT Considering the distinct effects of the Wanghong group in retaining online consumers, more and more live streamers tend to accept the Wanghong economy and aim to become Wanghong live streamers. Although existing scholars have investigated the relationship between live streamers and online consumers, limited studies have discussed how Wanghong live streamers affect online consumers’ watching interest and lead to their final purchasing behaviors. To explore their purchasing decision-making process, this paper draws on the Attention-Interest-Desire-Action (AIDA) model and divides the process into Attention, Interest, Desire, and Action units. Through the data analysis based on the partial least squares path modeling and variance-based structural equation modeling (PLS-SEM), both attractive appearance and social interaction designed by Wanghongs positively affect online consumers’ watching interest, generate their purchasing desires, and cause their final purchasing behaviors. The research results are helpful for related scholars and network industries to understand the interactive mechanism between online consumers and Wanghongs. Exploring the distinct effect of this group will be beneficial in driving the development of the Wanghong economy and improving the environment of live streaming economy.

Biodegradable Polymers: Environmental Sustainability, Challenges, and Considerations

The increasing global production of polymers and plastics has led to the development of a new environmentally friendly material that is degraded in the environment by enzymatic action and/or smaller units are created by the breakdown of microorganisms such as bacteria, fungi, and algae. In the present review, the analysis of biodegradable polymers was presented on the global production of these polymers and industrial and environmental applications. In addition, the work focuses on sustainability in the environment, because the biodegradable polymers have become spread as sustainable alternative options to the non-biodegradable polymers. Additionally, the study provided an explanation of how to produce biodegradable polymers using agricultural waste, including tomato pomace and pineapple peel waste. Generally, the creation and application of biodegradable polymers need to take into account their environmental effect and performance along with aspects like cost, availability, recyclability, and printing methods.

Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample

Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample

Microexpression Recognition Method Based on ADP-DSTN Feature Fusion and Convolutional Block Attention Module

Microexpressions are subtle facial movements that occur within an extremely brief time frame, often revealing suppressed emotions. These expressions hold significant importance across various fields, including security monitoring and human–computer interaction. However, the accuracy of microexpression recognition is severely constrained by the inherent characteristics of these expressions. To address the issue of low detection accuracy regarding the subtle features present in microexpressions’ facial action units, this paper proposes a microexpression action unit detection algorithm, Attention-embedded Dual Path and Shallow Three-stream Networks (ADP-DSTN), that incorporates an attention-embedded dual path and a shallow three-stream network. First, an attention mechanism was embedded after each Bottleneck layer in the foundational Dual Path Networks to extract static features representing subtle texture variations that have significant weights in the action units. Subsequently, a shallow three-stream 3D convolutional neural network was employed to extract optical flow features that were particularly sensitive to temporal and discriminative characteristics specific to microexpression action units. Finally, the acquired static facial feature vectors and optical flow feature vectors were concatenated to form a fused feature vector that encompassed more effective information for recognition. Each facial action unit was then trained individually to address the issue of weak correlations among the facial action units, thereby facilitating the classification of microexpression emotions. The experimental results demonstrated that the proposed method achieved great performance across several microexpression datasets. The unweighted average recall (UAR) values were 80.71%, 89.55%, 44.64%, 80.59%, and 88.32% for the SAMM, CASME II, CAS(ME)3, SMIC, and MEGC2019 datasets, respectively. The unweighted F1 scores (UF1) were 79.32%, 88.30%, 43.03%, 81.12%, and 88.95%, respectively. Furthermore, when compared to the benchmark model, our proposed model achieved better performance with lower computational complexity, characterized by a Floating Point Operations (FLOPs) value of 1087.350 M and a total of 6.356 × 106 model parameters.

CNN-Based Voice Recognition by Self-Governing Robots to Improve Computer-Human Communication

Robots must have the ability to recognize human emotions to engage with individuals and to plan their movements autonomously. Nonverbal signals encompassing pitch, loudness, spectrum, and speech speed are successful methods for transmitting emotions to most individuals. Provided this situation, a machine could have the capability of qualified figure out emotions by deploying the traits of spoken communication, and these potentially propagate vital information concerning the speaker's emotional state. More precisely, a combination of numerous facial action units can be employed to describe a human's emotion. In this paper, we propose a deep Convolutional Neural Network-based system that can identify sentiments in real-time and with a high accuracy rate. This investigation establishes an entirely novel speech-emotion detection system founded on Convolutional Neural Networks (CNNs). With the support of a top-tier GPU, a model is developed and nourished raw speech from a specific set for training, classification, and testing purposes. We further analyze the speech data and incorporate the information from the visual and audio sources to enhance the recognition system's accuracy. The benefits of the proposed method for emotion identification and the implications of blending visual and aural suggestions are made clear by the experimental results. Convolutional neural networks (CNNs) are taught on grayscale images from the softmax dataset in the present work. To acquire the best accuracy, we experimented with different baselines and max pooling layers, finally acquiring 89.98% accuracy. Dropout is one approach we have used to ward off overfitting.

Project: Malecón Buenaventura, Urban Regeneration For The Southern Area of The Island of Cascajal

Objective: The objective of this research is to generate the solution from the urban planning of informal settlements located in the southern area of Cascajal Island, which have been settled in an ancestral way on low-water strips which are declared as goods for public use by the Colombian State, being the same in constant illegality; establishing the following specific objectives: Interpret the current regulations, given in Law 185/59, Decree Law 2424/84 and Agreement 03/01; constituting the baseline of urban planning, to generate new urban fiscal land, and minimize the non-mitigable risk. Carry out urban planning; taking into account the instruments of complementary planning; developed through six units of urban actions. Theoretical reference: On the goods of public use (bajamares) it is impossible to establish legal processes for formal habitats; therefore, they must be converted into fiscal assets in favor of the District of Buenaventura, to solve the settlements towards the required formality, taking into account the incidence towards the normative framework of the current territorial order given by Law 185/59 and Decree Law 2324/84 Method: The methodological approach was qualitative of analytic-relational type, through the normative interpretation of Law 185/59 and Decree Law 2324/84 in relation to the current territorial order; and whose data collection instruments were the documentation of normative files and government sources of district level linked to the ownership of fiscal assets Result and conclusion: The expected results showed the proposed resolution as a dynamic of the urban law towards the specific implication for the territorial ordering; which raises that they must contemplate the particularities of the territories and from there generate the solutions that solve the socioeconomic, urban and environmental problems that by the inability of the State itself are perpetuated in time and where the same rules do not allow to generate the solutions. Implications of the research: The urban solution proposed, demonstrates the effectiveness within urban law, of the specific regulatory interpretations on the territorial ordering Originality/Value: The research on the solution of informal settlements on low-water areas in the city of Buenaventura (southern area of Cascajal Island); presents a significant originality and value for the welfare state of the poorest and most vulnerable communities, contributing to their quality of life.

Paying attention to uncertainty: A stochastic multimodal transformers for post-traumatic stress disorder detection using video

Background and Objectives:Post-traumatic stress disorder is a debilitating psychological condition that can manifest following exposure to traumatic events. It affects individuals from diverse backgrounds and is associated with various symptoms, including intrusive thoughts, nightmares, hyperarousal, and avoidance behaviors. Methods:To address this challenge this study proposes a decision support system powered by a novel multimodal deep learning approach, based on a stochastic Transformer and video data. This Transformer has the ability to take advantage of its stochastic activation function and layers that allow it to learn sparse representations of the inputs. The method leverages a combination of low-level features extracted using three modalities, including Mel-frequency cepstral coefficients extracted from audio recordings, Facial Action Units captured from facial expressions, and textual data obtained from the audio transcription. By considering these modalities, our proposed model captures a comprehensive range of information related to post-traumatic stress disorder symptoms, including vocal cues, facial expressions, and linguistic content. Results:The deep learning model was trained and evaluated on the eDAIC dataset, which consists of clinical interviews with individuals with and without post-traumatic disorder. The model achieved state-of-the-art results, demonstrating its effectiveness in accurately detecting PTSD, showing an impressive Root Mean Square Error of 1.98, and a Concordance Correlation Coefficient of 0.722, signifying the model’s superior performance compared to existing approaches. Conclusion:This work introduces a new method for post-traumatic stress disorder detection from videos by utilizing a multimodal stochastic Transformer model. The model makes use of a variety of modalities, such as text, audio, and visual data, to gather comprehensive and varied information in order to make the detection.

FacePsy: An Open-Source Affective Mobile Sensing System - Analyzing Facial Behavior and Head Gesture for Depression Detection in Naturalistic Settings

Depression, a prevalent and complex mental health issue affecting millions worldwide, presents significant challenges for detection and monitoring. While facial expressions have shown promise in laboratory settings for identifying depression, their potential in real-world applications remains largely unexplored due to the difficulties in developing efficient mobile systems. In this study, we aim to introduce FacePsy, an open-source mobile sensing system designed to capture affective inferences by analyzing sophisticated features and generating real-time data on facial behavior landmarks, eye movements, and head gestures - all within the naturalistic context of smartphone usage with 25 participants. Through rigorous development, testing, and optimization, we identified eye-open states, head gestures, smile expressions, and specific Action Units (2, 6, 7, 12, 15, and 17) as significant indicators of depressive episodes (AUROC=81%). Our regression model predicting PHQ-9 scores achieved moderate accuracy, with a Mean Absolute Error of 3.08. Our findings offer valuable insights and implications for enhancing deployable and usable mobile affective sensing systems, ultimately improving mental health monitoring, prediction, and just-in-time adaptive interventions for researchers and developers in healthcare.

Multimodal Fusion for Talking Face Generation Utilizing Speech-Related Facial Action Units

Talking face generation is to synthesize a lip-synchronized talking face video by inputting an arbitrary face image and corresponding audio clips. The current talking face model can be divided into four parts: visual feature extraction, audio feature processing, multimodal feature fusion, and rendering module. For the visual feature extraction part, existing methods face the challenge of complex learning task with noisy features, this article introduces an attention-based disentanglement module to disentangle the face into Audio-face and Identity-face using speech-related facial action unit (AU) information. For the multimodal feature fusion part, existing methods ignore not only the interaction and relationship of cross-modal information but also the local driving information of the mouth muscles. This study proposes a novel generative framework that incorporates a dilated non-causal temporal convolutional self-attention network as a multimodal fusion module to enhance the learning of cross-modal features. The proposed method employs both audio- and speech-related facial AUs as driving information. Speech-related AU information can facilitate more accurate mouth movements. Given the high correlation between speech and speech-related AUs, we propose an audio-to-AU module to predict speech-related AU information. Finally, we present a diffusion model for the synthesis of talking face images. We verify the effectiveness of the proposed model on the GRID and TCD-TIMIT datasets. An ablation study is also conducted to verify the contribution of each component. The results of quantitative and qualitative experiments demonstrate that our method outperforms existing methods in terms of both image quality and lip-sync accuracy. Code is available at https://mftfg-au.github.io/Multimodal_Fusion/.

Prototype Facial Response to Cute Stimuli: Expression and Recognition.

Cute, kindchenschema stimuli can evoke a suite of cognitive, physiological, and behavioral tendencies thought to promote caregiving. This research investigated facial expression elements associated with this response to cuteness and assessed the recognizability of an expression combining these elements. In Studies 1 and 2, participants at a community outreach event (Study 1, n = 19) and undergraduate students (Study 2, n = 103) showed spontaneous facial displays while watching videos/photos of baby humans and animals. These were Facial Action Coding System (FACS)-coded, revealing characteristic and statistically distinctive action unit elements of facial expression responses to cuteness. In six follow-up online studies (combined N = 962), including replications with Syrian refugees (n = 103) and Chinese samples (n = 222), a "cuteness prototype" expression combining all elements identified across Studies 1 and 2 (i.e., oblique brows, chin raise, lip tightening, and Duchenne smile) was commonly interpreted as a response to cuteness. These findings add to a growing literature about caregiving-focused motivational states and associated emotion/affect.

Third Social Pathway computes Dynamic Action Unit Features for Emotion Decision Behavior

Third Social Pathway computes Dynamic Action Unit Features for Emotion Decision Behavior

BFFN: A novel balanced feature fusion network for fair facial expression recognition

Facial expression recognition (FER) technology has become increasingly mature and applicable in recent years. However, it still suffers from the bias of expression class, which can lead to unfair decisions for certain expression classes in applications. This study aims to mitigate expression class bias through both pre-processing and in-processing approaches. First, we analyze the output of existing models and demonstrate the existence of obvious class bias, particularly for underrepresented expressions. Second, we develop a class-balanced dataset constructed through data generation, mitigating unfairness at the data level. Then, we propose the Balanced Feature Fusion Network (BFFN), a class fairness-enhancing network. The BFFN mitigates the class bias by adding facial action units (AU) to enrich expression-related features and allocating weights in the AU feature fusion process to improve the extraction ability of underrepresented expression features. Finally, extensive experiments on datasets (RAF-DB and AffectNet) provide evidence that our BFFN outperforms existing FER models, improving the fairness by at least 16%.

Comparative analysis of artificial intelligence and expert assessments in detecting neonatal procedural pain

Assessing pain in newborns in the NICU is crucial due to their frequent exposure to painful stimuli, yet it's challenging due to the subjective nature of current methods. This study aimed to evaluate the effectiveness of an AI system designed for automatic facial recognition by comparing its performance with the expert opinion of health care provider. This is a secondary analysis from an eye-tracking study, assessing neonatal pain evaluations by healthcare professionals. The performance of AI software, FaceReader 9, was compared to experts' evaluations using a visual-analog scale, focusing on identifying specific facial action units associated with different pain levels. The study found significant differences in AI-generated metrics—arousal and valence—across three stimulus types: non-noxious thermal, short-noxious, and prolonged-noxious, with p-values below 0.001. A strong correlation (r = 0.84, p ≤ .001) was observed between AI metrics and expert ratings. Eleven facial action units were identified as relevant to describe neonatal pain. The findings highlight the AI system's potential in accurately detecting and analyzing newborn facial expressions in response to varying pain intensities, demonstrating a significant correlation with healthcare professionals' assessments. This suggests that AI technology could enhance objective pain assessment in neonates.