Facial Movements Research Articles

Speech synthesis from three-axis accelerometer signals using conformer-based deep neural network

Silent speech interfaces (SSIs) have emerged as innovative non-acoustic communication methods, and our previous study demonstrated the significant potential of three-axis accelerometer-based SSIs to identify silently spoken words with high classification accuracy. The developed accelerometer-based SSI with only four accelerometers and a small training dataset outperformed a conventional surface electromyography (sEMG)-based SSI. In this study, motivated by the promising initial results, we investigated the feasibility of synthesizing spoken speech from three-axis accelerometer signals. This exploration aimed to assess the potential of accelerometer-based SSIs for practical silent communication applications. Nineteen healthy individuals participated in our experiments. Five accelerometers were attached to the face to acquire speech-related facial movements while the participants read 270 Korean sentences aloud. For the speech synthesis, we used a convolution-augmented Transformer (Conformer)-based deep neural network model to convert the accelerometer signals into a Mel spectrogram, from which an audio waveform was synthesized using HiFi-GAN. To evaluate the quality of the generated Mel spectrograms, ten-fold cross-validation was performed, and the Mel cepstral distortion (MCD) was chosen as the evaluation metric. As a result, an average MCD of 5.03 ± 0.65 was achieved using four optimized accelerometers based on our previous study. Furthermore, the quality of generated Mel spectrograms was significantly enhanced by adding one more accelerometer attached under the chin, achieving an average MCD of 4.86 ± 0.65 (p < 0.001, Wilcoxon signed-rank test). Although an objective comparison is difficult, these results surpass those obtained using conventional SSIs based on sEMG, electromagnetic articulography, and electropalatography with the fewest sensors and a similar or smaller number of sentences to train the model. Our proposed approach will contribute to the widespread adoption of accelerometer-based SSIs, leveraging the advantages of accelerometers like low power consumption, invulnerability to physiological artifacts, and high portability.

CSLNSpeech: Solving the extended speech separation problem with the help of Chinese sign language

Previous audio-visual speech separation methods synchronize the speaker's facial movement and speech in the video to self-supervise the speech separation. In this paper, we propose a model to solve the speech separation problem assisted by both face and sign language, which we call the extended speech separation problem. We design a general deep learning network to learn the combination of three modalities, audio, face, and sign language information, to solve the speech separation problem better. We introduce a large-scale dataset named the Chinese Sign Language News Speech (CSLNSpeech) dataset to train the model, in which three modalities coexist: audio, face, and sign language. Experimental results show that the proposed model performs better and is more robust than the usual audio-visual system. In addition, the sign language modality can also be used alone to supervise speech separation tasks, and introducing sign language helps hearing-impaired people learn and communicate. Last, our model is a general speech separation framework and can achieve very competitive separation performance on two open-source audio-visual datasets. The code is available at https://github.com/iveveive/SLNSpeech

Facial expressions in different communication settings: A case of whispering and speaking with a face mask in Farsi

Abstract This study addresses the importance of orofacial gestures and acoustic cues to execute prosodic patterns under different communicative settings in Farsi. Given that Farsi lacks morpho-syntactic markers for polar questions, we aim to determine whether specific facial movements accompany the prosodic correlates of questionhood in Farsi under conditions of degraded information, that is, whispering and wearing face masks. We hypothesise speakers will employ the most pronounced facial expressions when whispering questions with a face mask to compensate for the absence of F0, reduced intensity and lower face invisibility. To this end, we conducted an experiment with 10 Persian speakers producing 10 pairs of statements and questions in normal and whispered speech modes with and without face masks. Our results provide support to our hypotheses that speakers will intensify their orofacial expressions when confronted with marked conditions. We interpreted our results in terms of the ‘hand in hand’ and ‘trade-off’ hypothesis. In whispered speech, the parallel realisation of longer word duration and orofacial expressions may be a compensatory mechanism for the limited options to convey intonation. Also, the lower face coverage is mutually compensated for by word duration and intensified upper facial expressions, all of which in turn support the trade-off hypothesis.

Digital assessment of nonverbal behaviors forecasts first onset of depression.

Adolescence is marked by a sharp increase in the incidence of depression, especially in females. Identification of risk for depressive disorders (DD) in this key developmental stage can help prevention efforts, mitigating the clinical and public burden of DD. While frequently used in diagnosis, nonverbal behaviors are relatively understudied as risk markers for DD. Digital technology, such as facial recognition, may provide objective, fast, efficient, and cost-effective means of measuring nonverbal behavior. Here, we analyzed video-recorded clinical interviews of 359 never-depressed adolescents females via commercially available facial emotion recognition software. We found that average head and facial movements forecast future first onset of depression (AUC = 0.70) beyond the effects of other established self-report and physiological markers of DD risk. Overall, these findings suggest that digital assessment of nonverbal behaviors may provide a promising risk marker for DD, which could aid in early identification and intervention efforts.

Normal Facial Function in the Sunnybrook Facial Grading System: Is the 100% Score too Restrictive?

The primary objective of this study was to describe the distribution of the Sunnybrook Facial Grading System scores in the general population. Secondary objectives included analyzing the effects of age, gender and facial side on these scores. Two speech therapists specializing in facial motor skill assessment evaluated the healthy participants using the Sunnybrook Facial Grading System, first with the right hemiface as a reference and then with the left hemiface as a reference. The study was conducted in our ENT department from September 2022 to June 2023. One hundred eleven healthy participants were included (57 women and 54 men), aged 18 to 79 years. SFGS composite scores (SFGS-Total) and sub-scores at rest (SFGS-Rest) and in movement (SFGS-Movement) were studied according to three age categories (18-39 years) (40-59 years) and (60-79 years) and gender. Inter-rater reliability was collected between the two evaluators with Cronbach's alpha. Taking the right hemiface as a reference, SFGS-Total scores ranged from 65% to 100% (median = 96, IQR [91-100]). When the left hemiface was considered as the reference, scores ranged from 78% to 100% (median = 95; IQR [90-100]). Right and left SFGS-Total scores showed high inter-rater reliability (respectively α = 0.953 and α = 0.926). There was a slight negative correlation between age and SFGS-Total scores. By embracing a more realistic approach that acknowledges natural variations and asymmetry in facial movements, we can enhance patient care and promote a more holistic understanding of facial rehabilitation outcomes.

Facial Expressions of Emotion in Children with Cleft Lip and Palate.

To examine the facial movements children with cleft lip and palate (CLP) employ to express basic emotions. Ability of observers to interpret facial expressions of children with CLP was also considered. Prospective case-control design. Outpatient craniofacial anomalies clinic. Twenty-five children with CLP (age 8 to 12) and 25 age/sex-matched controls. Children were video recorded making facial expressions representing anger, disgust, fear, happiness, sadness, and surprise. Magnitude of children's facial movements was quantified and compared using OpenFace. Subsequently, emotion videos were presented to 19 adults who were asked to identify the emotion conveyed in each facial expression. Accuracy of emotion recognition was compared across groups. Compared with controls, children with CLP employed significantly (P < .05) smaller magnitude superior and lateral perioral movements to express disgust (Cohen's d = .50), happiness (Cohen's d = 1.1), and fear (Cohen's d = .93). For disgust and sadness, children with CLP employed significantly greater magnitude movements of the nose and chin, presumably to compensate for reduced perioral range of motion. For anger, happiness, and sadness, children with CLP employed smaller magnitude movements of the upper face when compared with controls. Observers identified disgust (OR = 1.26), and fear (OR = 2.44) significantly less accurately in children with CLP when compared with controls. Children with CLP employed different facial movements to express certain emotions. Observers less accurately identified some emotions conveyed by facial expressions in children with CLP when compared with controls, likely due in part to differences in facial movements. Future research should explore the implications of these differences for social communication.

A Louder Call for the Integration of Multiple Nonverbal Channels in the Study of Affect.

Affective science has increasingly sought to represent emotional experiences multimodally, measuring affect through a combination of self-report ratings, linguistic output, physiological measures, and/or nonverbal expressions. However, despite widespread recognition that non-facial nonverbal cues are an important facet of expressive behavior, measures of nonverbal expressions commonly focus solely on facial movements. This Commentary represents a call for affective scientists to integrate a larger range of nonverbal cues-including gestures, postures, and vocal cues-alongside facial cues in efforts to represent the experience of emotion and its communication. Using the measurement and analysis of vocal cues as an illustrative case, the Commentary considers challenges, potential solutions, and the theoretical and translational significance of working to integrate multiple nonverbal channels in the study of affect.

Wireless high-resolution surface facial electromyography mask for discrimination of standardized facial expressions in healthy adults

Wired high resolution surface electromyography (sEMG) using gelled electrodes is a standard method for psycho-physiological, neurological and medical research. Despite its widespread use electrode placement is elaborative, time-consuming, and the overall experimental setting is prone to mechanical artifacts and thus offers little flexibility. Wireless and easy-to-apply technologies would facilitate more accessible examination in a realistic setting. To address this, a novel smart skin technology consisting of wireless dry 16-electrodes was tested. The soft electrode arrays were attached to the right hemiface of 37 healthy adult participants (60% female; 20 to 57 years). The participants performed three runs of a standard set of different facial expression exercises. Linear mixed-effects models utilizing the sEMG amplitudes as outcome measure were used to evaluate differences between the facial movement tasks and runs (separately for every task). The smart electrodes showed specific activation patterns for each of the exercises. 82% of the exercises could be differentiated from each other with very high precision when using the average muscle action of all electrodes. The effects were consistent during the 3 runs. Thus, it appears that wireless high-resolution sEMG analysis with smart skin technology successfully discriminates standard facial expressions in research and clinical settings.

A survey on deep learning based reenactment methods for deepfake applications

AbstractAmong the sectors that deep learning has transformed, deepfake, a novel method of manipulating multimedia, deserves particular attention. The long‐term objective of many researchers is to seamlessly mimic human facial movement or whole‐body activity, referred to as reenactment. Deepfake progress has made this goal much more feasible in recent years. Yet, achieving more realistic facial and body reenactment remains a challenging task. The primary focus of this study is to explore the current capability of the reenactment techniques and expand them further to attain greater results. The analysis offers a thorough overview of the various techniques involved, the challenges addressed, the datasets utilized, and the metrics employed by the underlying methods of reenactment technologies. The study also addresses the potential risks and their mitigating strategies to ensure responsible reenactment techniques. To the best of the authors' knowledge, this is the first survey paper that delves deeper into the topic of deepfake reenactment.

Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flow

This paper presents a production-oriented 4D facial reconstruction pipeline designed to produce high-fidelity facial mesh sequences with a consistently structured topology, while preserving the wireframe structure specified by artists. We have designed and developed a compact, efficient, and fast optical capture system based on synchronized camera arrays for high-precision dynamic 3D facial imaging. Unlike prevailing methods that primarily concentrate on single-frame reconstruction, often reliant on labor-intensive manual annotation, our framework exploits the constraint of appearance consistency to autonomously establish feature correspondence and uphold temporal coherence within the mesh. Consequently, our approach eliminates mesh drifting and jitter, enabling full parallelization for dynamic facial expression capture. The proposed pipeline decouples the non-linear deformation of facial expressions from the rigid movements of the skull through a stable external device. Leveraging progressive retopology, our methodology employs artist-guided templates as priors, ensuring the preservation of wireframe structures across the result sequence. Progressive retopology is achieved by constraining different fine-grained features of 3D landmarks, scan surface shapes, and appearance textures. The results of our study showcase facial mesh sequences with production-quality topology, adept at faithfully reproducing character expressions from photographs while achieving artist-friendly stable facial movements.

Semantic-aware hyper-space deformable neural radiance fields for facial avatar reconstruction

High-fidelity facial avatar reconstruction from monocular videos is a prominent research problem in computer graphics and computer vision. Recent advancements in the Neural Radiance Field (NeRF) have demonstrated remarkable proficiency in rendering novel views and garnered attention for its potential in facial avatar reconstruction. However, previous methodologies have overlooked the complex motion dynamics present across the head, torso, and intricate facial features. Additionally, a deficiency exists in a generalized NeRF-based framework for facial avatar reconstruction adaptable to either 3DMM coefficients or audio input. To tackle these challenges, we propose an innovative framework that leverages semantic-aware hyper-space deformable NeRF, facilitating the reconstruction of high-fidelity facial avatars from either 3DMM coefficients or audio features. Our framework effectively addresses both localized facial movements and broader head and torso motions through semantic guidance and a unified hyper-space deformation module. Specifically, we adopt a dynamic weighted ray sampling strategy to allocate varying degrees of attention to distinct semantic regions, enhancing the deformable NeRF framework with semantic guidance to capture fine-grained details across diverse facial regions. Moreover, we introduce a hyper-space deformation module that enables the transformation of observation space coordinates into canonical hyper-space coordinates, allowing for the learning of natural facial deformation and head-torso movements. Extensive experiments validate the superiority of our framework over existing state-of-the-art methods, demonstrating its effectiveness in producing realistic and expressive facial avatars. Our code is available at https://github.com/jematy/SAHS-Deformable-Nerf.

Duration-aware and mode-aware micro-expression spotting for long video sequences

Micro-expressions (MEs) are unconscious, instant and slight facial movements, revealing people’s true emotions. Locating MEs is a prerequisite of classifying them, while only a few researches focus on this task. Among them, sliding window based methods are the most prevalent. Due to the differences of individual physiological and psychological mechanisms, and some uncontrollable factors, the durations and transition modes of different MEs fluctuate greatly. Limited to fixed window scale and mode, traditional sliding window based ME spotting methods fail to capture the motion changes of all MEs exactly, resulting in performance degradation. In this paper, an ensemble learning based duration & mode-aware (DMA) ME spotting framework is proposed. Specifically, we exploit multiple sliding windows of different scales and modes to generate multiple weak detectors, each of which accommodates to MEs with certain duration and transition mode. Additionally, to get a more comprehensive strong detector, we integrate the analysis results of multiple weak detectors using a voting based aggregation module. Furthermore, a novel interval generation scheme is designed to merge close peaks and their neighbor frames into a complete ME interval. Experimental results on two long video databases show the promising performance of our proposed DMA framework compared with state-of-the-art methods. The codes are available at https://github.com/TJUMMG/DMA-ME-Spotting.

GAN-Based High-Quality Face-Swapping Composite Network

Face swapping or face replacement is a challenging task that involves transferring a source face to a target face while maintaining the target’s facial motion and expression. Although many studies have made a lot of encouraging progress, we have noticed that most of the current solutions have the problem of blurred images, abnormal features, and unnatural pictures after face swapping. To solve these problems, in this paper, we proposed a composite face-swapping generation network, which includes a face extraction module and a feature fusion generation module. This model retains the original facial expression features, as well as the background and lighting of the image while performing face swapping, making the image more realistic and natural. Compared with other excellent models, our model is more robust in terms of face identity, posture verification, and image quality.

Micro and macro facial expressions by driven animations in realistic Virtual Humans

Computer Graphics (CG) advancements have allowed the creation of more realistic Virtual Humans (VH) through modern techniques for animating the VH body and face, thereby affecting perception. From traditional methods, including blend shapes, to driven animations using facial and body tracking, these advancements can potentially enhance the perception of comfort and realism in relation to VHs. Previously, Psychology studied facial movements in humans, with some works separating expressions into macro and micro expressions. Also, some previous CG studies have analyzed how macro and micro expressions are perceived, replicating psychology studies in VHs, encompassing studies with realistic and cartoon VHs, and exploring different VH technologies. However, instead of using facial tracking animation methods, these previous studies animated the VHs using blendshapes interpolation. To understand how the facial tracking technique alters the perception of VHs, this paper extends the study to macro and micro expressions, employing two datasets to transfer real facial expressions to VHs and analyze how their expressions are perceived. Our findings suggest that transferring facial expressions from real actors to VHs significantly diminishes the accuracy of emotion perception compared to VH facial animations created by artists.

EEG correlates of static and dynamic face perception: the role of naturalistic motion

Much of our understanding of how the brain processes dynamic faces comes from research that compares static photographs to dynamic morphs, which exhibit simplified, computer-generated motion. By comparing static, video recorded, and dynamic morphed expressions, we aim to identify the neural correlates of naturalistic facial dynamism, using time-domain and time-frequency analysis. Dynamic morphs were made from the neutral and peak frames of video recorded transitions of happy and fearful expressions, which retained expression change and removed asynchronous and non-linear features of naturalistic facial motion. We found that dynamic morphs elicited increased N400 amplitudes and lower LPP amplitudes compared to other stimulus types. Video recordings elicited higher LPP amplitudes and greater frontal delta activity compared to other stimuli. Thematic analysis of participant interviews using a large language model revealed that participants found it difficult to assess the genuineness of morphed expressions, and easier to analyse the genuineness of happy compared to fearful expressions. Our findings suggest that animating real faces with artificial motion may violate expectations (N400) and reduce the social salience (LPP) of dynamic morphs. Results also suggest that delta oscillations in the frontal region may be involved with the perception of naturalistic facial motion in happy and fearful expressions. Overall, our findings highlight the sensitivity of neural mechanisms required for face perception to subtle changes in facial motion characteristics, which has important implications for neuroimaging research using faces with simplified motion.

Facial micro-expression recognition based on motion magnification network and graph attention mechanism

Micro-expression is extensively studied due to their ability to fully reflect individuals' genuine emotions. However, accurate micro-expression recognition is a challenging task due to the subtle motion of facial muscle. Therefore, this paper introduces a Graph Attention Mechanism-based Motion Magnification Guided Micro-Expression Recognition Network (GAM-MM-MER) to amplify delicate muscle motions and focus on key facial landmarks. First, we propose a Swin Transformer-based network for micro-expression motion magnification (ST-MEMM) to enhance the subtle motions in micro-expression videos, thereby unveiling imperceptible facial muscle movements. Then, we propose a graph attention mechanism-based network for micro-expression recognition (GAM-MER), which optimizes facial key area maps and prioritizes adjacent nodes crucial for mitigating the influence of noisy neighbors, while attending to key feature information. Finally, experimental evaluations conducted on the CASME II and SAMM datasets demonstrate the high accuracy and effectiveness of the proposed network compared to state-of-the-art approaches. The results of our network exhibit significant superiority over existing methods. Furthermore, ablation studies provide compelling evidence of the robustness of our proposed network, substantiating its efficacy in micro-expression recognition.

How an Android Expresses “Now Loading…”: Examining the Properties of Thinking Faces

The “thinking face” is a facial signal used to convey being in thought. For androids, the thinking face may be important to achieve natural human–robot interaction. However, the facial pattern necessary for portraying the thinking face remains unclear and has not yet been investigated in androids. The current study aims to (a) identify the facial patterns when people are engaged in answering complex questions (i.e., thinking face) and (b) clarify whether implementing the observed thinking faces in an android can facilitate natural human–robot interaction. In Study 1, we analyze the facial movements of 40 participants after they are prompted with difficult questions and indicate five facial patterns that corresponded to thinking faces. In Study 2, we further focus on the pattern of furrowing of the brows and narrowing of the eyes among the observed thinking facial patterns and implement this pattern in an android. The results show that thinking faces enhance the perception of being in thought, genuineness, human-likeness, and appropriateness in androids while decreasing eeriness. The free-description data also revealed that negative emotions are attributed to the thinking face. In Study 3, we compared the thinking vs. neutral faces in a question–answer situation. The results showed that the android's thinking face facilitated the perception of being in thought and human-likeness. These findings suggest that the thinking face of androids can facilitate natural human–robot interaction.

Bridging the Gap: Deep Learning Techniques for American Sign Language Recognition

Abstract: Communication stands upon the pillars of verbal and non-verbal conversations, and hence it holds the basis of human social relationships. Along with words, gestures are another component of nonverbal communication that achieves the purpose of conveying the intended sense and bridging the gab of the languages and cultures. People with speech or hearing problems usually read manual or verbal signs that often don't make sense to a hearing-challenged person. Gestures are the first sign of instruction that overcomes the speech gap. A kaleidoscopic patchwork of facial expressions comprising of facial movements and body language! Such variations occurring in linguistic areas overall are not surprising, as community cultures and tongues around the planet typically shape their language. In the United States and Canada, American Sign Language (ASL) is common and is an independent language from what is heard in the surrounding community but is also used as a way of communication between individuals and in groups that are deaf and hearing alike. There are some restrictions. Common language review and adequate practice are of crucial importance here, that is why it is so hard for deaf people to work outside. The accessibility of the translation tools decreases radically, which means it will be difficult to communicate, and it will further be hard to understand and to be understood. By integrating the AI technologies as neural networks and deep learning into the goals, the system will proceed to bridging various communication channels from manual writing to voice operation. The task goes with webcam installation together with gesture capture and then as an input it goes to the system. The proposed model will be divided into several stages namely, data acquisition, pre-training to the neural network, testing and the post-testing phases. This research project will do that through developing digital technology which in turn will enhance accessibility, encourage integration and let people who are film-blind or deaf to associate with the environment.

Rigid facial motion at study facilitates the holistic processing of own-race faces during the structural encoding stage

Holistic processing is a fundamental element of face-recognition studies. Some behavioral studies have investigated the impact of rigid facial motion on holistic face processing, yet it is still unclear how rigid motion affects the time course of holistic face processing for different face races. The current study investigated this issue, using the composite face effect (CFE) as a direct measure of holistic processing. Participants were asked to match the identity of the top half of a static composite face with the study face during the test stage, where the study face was either static or rigidly-moving. ERP results showed that rigidly-moving study faces elicited a larger CFE relative to static study faces in the N170 component when recognizing own-race faces. The amplitude of P1, N170 and P2 components indicated that rigid motion facilitated holistic face processing, with differences observed between the hemispheres over time. Specifically, the CFE was only observed after exposure to rigidly-moving faces in the P1 and P2 components of the right hemisphere. Additionally, a greater CFE was observed following exposure to rigidly-moving faces compared to static faces, particularly in the N170 component of the left hemisphere. This study suggests that holistic processing is a fundamental aspect of face perception that applies to both static and moving faces, not just static ones. Furthermore, rigid facial motion improves holistic processing of own-race faces during the structural encoding stage. These findings provide evidence of distinct neural mechanisms underlying the holistic processing of static and moving faces.

FaceMotionPreserve: a generative approach for facial de-identification and medical information preservation

Telemedicine and video-based diagnosis have raised significant concerns regarding the protection of facial privacy. Effective de-identification methods require the preservation of diagnostic information related to normal and pathological facial movements, which play a crucial role in the diagnosis of various movement, neurological, and psychiatric disorders. In this work, we have developed FaceMotionPreserve , a deep generative model-based approach that transforms patients’ facial identities while preserving facial dynamics with a novel face dynamic similarity module to enhance facial landmark consistency. We collected test videos from patients with Parkinson’s disease recruited via telemedicine for evaluation of model performance and clinical applicability. The performance of FaceMotionPreserve was quantitatively evaluated based on neurologist diagnostic consistency, critical facial behavior fidelity, and correlation of general facial dynamics. In addition, we further validated the robustness and advancements of our model in preserving medical information with clinical examination videos from a different cohort of patients. FaceMotionPreserve is applicable to real-time integration, safeguarding facial privacy while retaining crucial medical information associated with facial movements to address concerns in telemedicine, and facilitating safer and more collaborative medical data sharing.