Facial Expressions Research Articles

Expressive 3D Facial Animation Generation Based on Local-to-Global Latent Diffusion.

3D Facial animations, crucial to augmented and mixed reality digital media, have evolved from mere aesthetic elements to potent storytelling media. Despite considerable progress in facial animation of neutral emotions, existing methods still struggle to capture the authenticity of emotions. This paper introduces a novel approach to capture fine facial expressions and generate facial animations using audio synchronization. Our method consists of two key components: First, the Local-to-global Latent Diffusion Model (LG-LDM) tailored for authentic facial expressions, which can integrate audio, time step, facial expressions, and other conditions towards possible encoding of emotionally rich yet latent features in response to possibly noisy raw audio signals. The core of LG-LDM is our carefully designed Facial Denoiser Model (FDM) for aligning the local-to-global animation feature with audio. Second, we redesign an Emotion-centric Vector Quantized-Variational AutoEncoder framework (EVQ-VAE) to finely decode the subtle differences under different emotions and reconstruct the final 3D facial geometry. Our work significantly contributes to the key challenges of emotionally realistic 3D facial animation for audio synchronization and enhances the immersive experience and emotional depth in augmented and mixed reality applications. We provide a reproducibility kit including our code, dataset, and detailed instructions for running the experiments. This kit is available at https://github.com/wangxuanx/Face-Diffusion-Model.

Functional and Aesthetic Lower Lip Reconstruction With Depressor Anguli Oris Muscle Flap and Reverse Superior Labial Artery Flap.

The lower lip is an important anatomical structure involved in daily activities such as eating, drinking, and speaking, as well as in facial expressions. Lower lip defects should be repaired to preserve lip function without disrupting the aesthetic harmony of the lips within the face. This goal is difficult for plastic surgeons because, for functional and aesthetic repair, it is necessary to replace the mucosa, muscle (especially contractile), and skin tissues present in the lip. Subtotal lower lip repair was performed using an innervated depressor anguli oris muscle flap and a desepithelialized reverse superior labial artery flap in seven male patients with a mean age of 63 years. A depressor anguli oris muscle flap was used to repair the orbicularis oris muscle defect. The desepithelialized part of the reverse superior labial artery flap was used for mucosal defect repair, whereas the remaining part was used for skin defect repair. Patients were followed up for an average of 15 months. Patient lips, epithelialization, sensation, and sphincter function were clinically evaluated. Electromyography (EMG) was performed at 6 and 12 months to evaluate the function of the depressor anguli oris muscle flap. All the reverse superior labial artery flaps survived without any loss. Trapdoor deformity developed in the flaps of two patients. The desepithelialized parts of the flaps were completely epithelialized (covered with mucosa) by approximately 6th week. In the EMG controls of the patients at 6 months, the depressor anguli oris muscle was contractible. Sensation regained in the operated lip during the follow-up period. The combination of an innervated depressor anguli oris muscle flap and a desepithelialized reverse superior labial artery flap is an important surgical option for the functional and aesthetic repair of subtotal lower lip defects.

DFCNet +: Cross-modal dynamic feature contrast net for continuous sign language recognition

In sign language communication, the combination of hand signs and facial expressions is used to convey messages in a fluid manner. Accurate interpretation relies heavily on understanding the context of these signs. Current methods, however, often focus on static images, missing the continuous flow and the story that unfolds through successive movements in sign language. To address this constraint, our research introduces the Dynamic Feature Contrast Net Plus (DFCNet+), a novel model that incorporates both dynamic feature extraction and cross-modal learning. The dynamic feature extraction module of DFCNet+ uses dynamic trajectory capture to monitor and record motion across frames and apply key features as an enhancement tool that highlights pixels that are critical for recognizing important sign language movements, allowing the model to follow the temporal variation of the signs. In the cross-modal learning module, we depart from the conventional approach of aligning video frames with textual descriptions. Instead, we adopt a gloss-level alignment, which provides a more detailed match between the visual signals and their corresponding text glosses, capturing the intricate relationship between what is seen and the associated text. The enhanced proficiency of DFCNet+ in discerning inter-frame details translates to heightened precision on benchmarks such as PHOENIX14, PHOENIX14-T and CSL-Daily. Such performance underscores its advantage in dynamic feature capture and inter-modal learning compared to conventional approaches to sign language interpretation. Our code is available at https://github.com/fyzjut/DFCNet_Plus.

Leveraging the learning focal point algorithm for emotional intelligence

<span>One of the secrets of the success of the education process is taking into account the learner’s feelings. That is, the teacher must be characterized by high emotional intelligence (EI) to understand the student’s feelings in order to facilitate the indoctrination process for him. Within the framework of the project to create a robot teacher, we had to add this feature because of its importance. In this article, we create a computer application that classifies students' emotions based on deep learning and learning focal point (LFP) algorithm by analyzing facial expressions. That is, the robot will be able to know whether the student is happy, excited, or sad in order to deal with him appropriately.</span>

Accident Prevention System for Cars, Buses, Trucks

Driver drowsiness is a significant factor in road accidents, contributing to a large number of crashes annually. To enhance road safety, the development of a Drivers Drowsiness Detection System is essential. This system builds to detect early signs of lazyness in drivers and provide timely alerts to prevent accidents. The proposed system uses to track key indicators of drowsiness, such as eye closure, head position, yawning, and blink rate. By utilizing a camera-based real-time monitoring system, it continuously analyzes the driver’s facial expressions. The system employs cadcade algorithms to distinguish a featurebased object detection algorithm to detect objects from images between alert and drowsy states based on collected data. When drowsiness is detected, the system triggers alerts like audio warnings, seat vibrations, or even vehicle slowing mechanisms, depending on the integration. This technology has the potential to significantly reduce the number of road accidents related to driver fatigue and could be applied in both personal vehicles and commercial fleets. The implementation of such systems represents a critical step toward improving road safety and decreasing driver-related incidents. Nowadays it is very challenging to stay active all the time due to busy schedules. Falling asleep while driving can lead to serious consequences, accidents, and even death. This situation is much more common and therefore it is very important to solve this problem. So, to solve this problem, we developed a sleep alarm system for drivers. This system alerts the user when he falls asleep at the wheel, thus the main concern is preventing accidents and saving lives. This system is useful for long- distance travelers and late-night drivers.

Measuring palatability of pet food products: Sensory components, evaluations, challenges, and opportunities.

The pet food industry is a growing business launching a variety of new products in the market. The acceptability or preference of pet food samples has traditionally been measured using either one-bowl or two-bowl tests. Academic researchers and professionals in the pet food industry have explored other methods, including the cognitive palatability assessment protocols and the ranking test, to evaluate more than two samples. A variety of approaches and perspectives were also utilized to predict palatability and key sensory attributes of pet foods, including descriptive sensory analysis by human-trained panelists and pet food caregivers' perceptions of pet food. This review article examined a range of testing methods for evaluating the palatability of pet foods, specifically targeting products for dogs and/or cats. It outlined the advantages and disadvantages of each method. Additionally, the review provided in-depth insights into the key sensory attributes of pet foods and the methodologies for assessing palatability. It also explored pets' behavioral responses and facial expressions in relation to different pet foods. Furthermore, this review discussed current challenges and future opportunities in pet food development, including the use of instrumental analyses and artificial intelligence-based approaches.

Facial Expression Recognition-You Only Look Once-Neighborhood Coordinate Attention Mamba: Facial Expression Detection and Classification Based on Neighbor and Coordinates Attention Mechanism

In studying the joint object detection and classification problem for facial expression recognition (FER) deploying the YOLOX framework, we introduce a novel feature extractor, called neighborhood coordinate attention Mamba (NCAMamba) to substitute for the original feature extractor in the Feature Pyramid Network (FPN). NCAMamba combines the background information reduction capabilities of Mamba, the local neighborhood relationship understanding of neighborhood attention, and the directional relationship understanding of coordinate attention. The resulting FER-YOLO-NCAMamba model, when applied to two unaligned FER benchmark datasets, RAF-DB and SFEW, obtains significantly improved mean average precision (mAP) scores when compared with those obtained by other state-of-the-art methods. Moreover, in ablation studies, it is found that the NCA module is relatively more important than the Visual State Space (VSS), a version of using Mamba for image processing, and in visualization studies using the grad-CAM method, it reveals that regions around the nose tip are critical to recognizing the expression; if it is too large, it may lead to erroneous prediction, while a small focused region would lead to correct recognition; this may explain why FER of unaligned faces is such a challenging problem.

What is in a look? The accountability of gaze in trajectories to conflict

This study investigates the role of gaze in initiating episodes of conflict by examining, using multimodal conversation analysis, a set of cases in which a recipient is prompted to speak by another’s extended gaze. In these cases, this recipient response may be, e.g., “What,” or a more elaborate demand for an account, such as “Why are you looking at me like that for?” Here we investigate the characteristics of the gaze that prompts such responses, and what actions such responses constitute. While “What” compositionally resembles other-initiated repair, its sequential position characterizes it as a so-called “go-ahead” action. In these cases, the sequential positioning of such gazes, constituting it structurally as a so-called “pre,” alongside its durational characteristics and facial expression, are examined to identify the normative associations of gaze and subsequent conduct that make such gazes accountable.

Virtual Children’s Hospital for Learning Pediatric Nursing

Abstract Background In the context of a funded Erasmus+ program, a platform of a Virtual Children’s Hospital for Learning Pediatric Nursing - VLPN is being developed. The education of nurses working with children, adolescents and families, needs to include specialized knowledge, skills, attitudes and behaviors that will empower them to provide optimal and high-quality care. The virtual platform enables the improvement of pediatric nursing competences of the nursing students through interactive and self-paced activities and content. Objectives 1. To develop an immersive and interactive virtual reality learning environment (VRLE), a virtual children’s hospital and new teaching and learning methods and practices, utilizing virtual reality and simulation in teaching. 2. To promote the international collaboration of students and lecturers through the VRLE Results The results are preliminary. 1. The VRLE includes educational materials developed to be integrated in the pediatric nursing studies, easy to access and use by the students and teachers with innovative self-pace learning material i.e. videos, 3D-models, interactive tests. Digital patients can react to the provided care with speech, movement, facial expressions and gestures, something that is not possible with the manikins in the Labs. 2. Nursing students and lecturers from different countries can participate in joint remote seminars or workshops in the VRLE, sharing and comparing their country and culture-specific differences and similarities in the nursing practices. VRLE and digital materials enables synchronous and asynchronous learning. Conclusions VRLE is highly conducive to clinical and procedure-focused training by enabling simulation. VR methods offer nursing students the opportunity to practice skills in a safe and risk-free environment in an immersive way. It can be accessed by multiple students, teachers or nurses at the same time from different locations in different countries. Key messages • New innovative VRLE enhances nursing students’ knowledge and skills that are needed when working with sick children and their families. • International collaboration, interaction and mobilization are enabled by using a virtual platform instead of traveling, thus strengthening the co-operation in European higher educational institutes.

Audio-Driven Facial Animation with Deep Learning: A Survey

Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology.

Autism Identification Based on the Intelligent Analysis of Facial Behaviors: An Approach Combining Coarse- and Fine-Grained Analysis

Background: Facial behavior has emerged as a crucial biomarker for autism identification. However, heterogeneity among individuals with autism poses a significant obstacle to traditional feature extraction methods, which often lack the necessary discriminative power. While deep-learning methods hold promise, they are often criticized for their lack of interpretability. Methods: To address these challenges, we developed an innovative facial behavior characterization model that integrates coarse- and fine-grained analyses for intelligent autism identification. The coarse-grained analysis provides a holistic view by computing statistical measures related to facial behavior characteristics. In contrast, the fine-grained component uncovers subtle temporal fluctuations by employing a long short-term memory (LSTM) model to capture the temporal dynamics of head pose, facial expression intensity, and expression types. To fully harness the strengths of both analyses, we implemented a feature-level attention mechanism. This not only enhances the model’s interpretability but also provides valuable insights by highlighting the most influential features through attention weights. Results: Upon evaluation using three-fold cross-validation on a self-constructed autism dataset, our integrated approach achieved an average recognition accuracy of 88.74%, surpassing the standalone coarse-grained analysis by 8.49%. Conclusions: This experimental result underscores the improved generalizability of facial behavior features and effectively mitigates the complexities stemming from the pronounced intragroup variability of those with autism, thereby contributing to more accurate and interpretable autism identification.

Implementation of the YOLO Method for Detection of Human Emotions Based on Facial Mimics

Emotion detection through facial expression recognition plays an important role in everyday life, such as how to respond correctly to emotional expressions in social interactions, so that you can establish and build verbal or nonverbal communication with other people and so on. Facial expressions are facial changes in response to a person's emotional state, intentions, or social communication. Face detection is the first step that must be taken in facial analysis, including facial expression recognition. There are many methods that can be used to carry out the face detection process, such as the YOLO method. This YOLO method reframes object detection as a single regression problem, directly from image pixels to bounding box coordinates and class probabilities. By using the YOLO method, the process only needs to look once at the input image, to predict what objects are in the image and where those objects are. Based on the results of the tests carried out, the YOLO method can be used to detect human facial expressions with a success rate of 80%, with neutral, surprise and disgust facial expressions having a good level of accuracy and fear facial expressions having a poor accuracy level. The YOLO method is able to detect facial expressions of humans who wear accessories such as glasses.

Human-machine interaction with intelligent machine learning algorithms for smart classrooms

ABSTRACT The modern technology of learning and teaching in education has moved on to smart classes. The introduction of artificial intelligence (AI) in the education sector has essentially taken over traditional classrooms and transformed how education leads to admiration. Various data types, such as videos, images, sound, text, and biometric data like eye-tracking potentially captured through smart classrooms, pose new issues and challenges for artificial intelligence algorithms. Hence, this paper suggests the Machine Learning Assisted Intelligent Human-Machine Interaction (MLIHMI) model for the smart classroom monitoring system. The real-time interactive dimension signifies the capability to support the Smart classroom’s teaching and human-computer interactions, including smooth interaction, convenient operation, and interactive monitoring. This paper used ML and HMI models to introduce a new effective database for online learning and smart classroom environments utilising the learners’ body postures, facial expressions, and hand gestures. The numerical findings show that the suggested system achieves less error rate of 10.1% and enhances the accuracy rate of 96.5%, with the highest classification ratio of 95.6%, prediction rate of 93.6%, the detection rate of 97.3%, and engagement rate of 81.3% in student gesture and postures recognition and the identification of facial expressions with the high performance of 93.3% when compared to other existing models.

Decoding the influence of emotional and attentional states on self-control using facial analysis

Self-control plays a pivotal role in pursuing long-term goals related to health and financial well-being. While ample evidence suggests that humans are prone to occasional self-control lapses, little is known about how changes in emotional and attentional states affect the ability to maintain self-control. In two studies (N1 = 109 and N2 = 90), we used emotion recognition software to decode participants’ facial expressions while manipulating their attentional and emotional states during a Psychomotor Vigilance Task (PVT) before exerting self-control in a subsequent task. Our findings reveal dissociable roles of attention and valence in maintaining self-control, depending on the distinct demands of the task. Specifically, performance in a subsequent cognitive task was predominantly associated with changes in attentional states during the PVT rather than valence. Conversely, preferences in a subsequent social task were associated with changes in valence states during the PVT, while attention showed no effect. This dissociation between attention-invoked and emotion-invoked lapses in self-control holds significant implications for psychological and economic models of self-control, ultimately contributing to the development of policies to mitigate the detrimental consequences of individual self-control lapses.

Attendance Marking using Face Detection

The demand for automated attendance management systems has significantly increased in educational institutions and corporate environments where manual methods are prone to inefficiencies and errors. This paper presents a real-time, facial recognition-based attendance system developed using Flask, OpenCV, and machine learning techniques. The system employs OpenCV’s Haar Cascade classifier for face detection and the K-Nearest Neighbours (KNN) algorithm for face recognition. During user registration, the system captures 10 images per individual to ensure robust recognition across various facial expressions. The web interface, built using Flask, facilitates user interaction for managing attendance records stored in CSV format. The system achieves an accuracy of 92% under optimal lighting conditions, though performance decreases to 80% in low-light environments. Recognition is completed in approximately 2 seconds, making the system suitable for real-time applications in classrooms or offices. Despite handling real-time image streaming efficiently, challenges such as reduced accuracy when handling obstructions like masks or glasses remain. Future enhancements will explore the integration of deep learning models, such as Convolutional Neural Networks (CNNs), to improve robustness and scalability. This system demonstrates the potential of facial recognition technology for automating attendance tracking, with significant applications in both educational and corporate sectors

Social threat avoidance depends on action-outcome predictability

Avoiding threatening individuals is pivotal for adaptation to our social environment. Yet, it remains unclear whether social threat avoidance is subtended by goal-directed processes, in addition to stimulus-response associations. To test this, we manipulated outcome predictability during spontaneous approach/avoidance decisions from avatars displaying angry facial expressions. Across three virtual reality experiments, we showed that participants avoided more often when they could predict the outcome of their actions, indicating goal-directed processes. However, above-chance avoidance rate when facing unpredictable outcomes suggested that stimulus-response associations also played a role. We identified two latent classes of participants: the “goal-directed class” showed above-chance avoidance only in the predictable condition, while the “stimulus-response class” showed no credible difference between conditions but had a higher overall avoidance rate. The goal-directed class exhibited greater cardiac deceleration in the predictable condition, associated with better value integration in decision-making. Computationally, this class had an increased drift-rate in the predictable condition, reflecting increased value estimation of threat avoidance. In contrast, the stimulus-response class showed higher responsiveness to threat, indicated by increased drift-rate for avoidance and increased muscular activity at response time. These results support the central role of goal-directed processes in social threat avoidance and reveal its physiological and computational correlates.

Integration of facial expression detection into the design and assessment of programming teaching materials

Integration of facial expression detection into the design and assessment of programming teaching materials

Facial expression morphing: enhancing visual fidelity and preserving facial details in CycleGAN-based expression synthesis

Recent advancements in facial expression synthesis using deep learning, particularly with Cycle-Consistent Adversarial Networks (CycleGAN), have led to impressive results. However, a critical challenge persists: the generated expressions often lack the sharpness and fine details of the original face, such as freckles, moles, or birthmarks. To address this issue, we introduce the Facial Expression Morphing (FEM) algorithm, a novel post-processing method designed to enhance the visual fidelity of CycleGAN-based outputs. The FEM method blends the input image with the generated expression, prioritizing the preservation of crucial facial details. We experimented with our method on the Radboud Faces Database (RafD) and evaluated employing the Fréchet Inception Distance (FID) standard benchmark for image-to-image translation and introducing a new metric, FSD (Facial Similarity Distance), to specifically measure the similarity between translated and real images. Our comprehensive analysis of CycleGAN, UNet Vision Transformer cycle-consistent GAN versions 1 (UVCGANv1) and 2 (UVCGANv2) reveals a substantial enhancement in image clarity and preservation of intricate details. The average FID score of 31.92 achieved by our models represents a remarkable 50% reduction compared to the previous state-of-the-art model’s score of 63.82, showcasing the significant advancements made in this domain. This substantial enhancement in image quality is further supported by our proposed FSD metric, which shows a closer resemblance between FEM-processed images and the original faces.

Mapping perceived sentiments in university campuses with varied landscape metrics and climatic conditions

A sustainable university campus should accommodate students to experience positive emotions, which can be evoked by sustainable landscape with green and blue spaces (GBS). This effect is location-dependent because local vegetative type is climate-determinative, but evidence is not sufficient for sentiments of people experiencing campus landscapes. Forty-seven university campuses were selected along a latitudinal gradient in mainland China, and 100 subjects were chosen per campus (50 indoor and 50 outdoor). Photos of the subjects’ faces on Sina Weibo were collected. Facial expressions were assigned happy, sad, and neutral scores (n = 4,334). The average temperature (AveT) and blue space area (BlueA) showed negative relationships with latitude, thereby generating neutral emotion scores for subjects at indoor and outdoor locations. The ratio of green space area to host campus was the only landscape metric that depressed the presentation of happiness and enhanced sadness levels. Large water bodies should be built on campuses to induce calmness, and a high ratio of green spaces should be avoided. Mapping results show that campuses in eastern regions (Beijing and Liaoning) tend to elicit positive sentiments more frequently.

Occluded facial expression recognition based on deep learning

Abstract. When employing Convolutional Neural Networks (CNNs) for facial expression recognition, several challenges are often encountered, such as facial occlusions, the limited size of reliable expression datasets, and inadequate precision in recognition outcomes. This paper preprocesses the dataset to enhance its reliability. By leveraging data synthesis and augmentation techniques, it employs a method of randomly generating occlusion blocks to integrate and expand the dataset. Based on the ResNet-18 network, the model is optimized by incorporating an attention mechanism, thereby improving the network's precision and robustness in recognizing facial expressions.