Non-rigid Facial Motion Research Articles

The Remove-the-Mask Open-Source head and neck Surface-Guided radiation therapy system

Background and PurposeSurface Guided Radiotherapy (SGRT) for head and neck radiotherapy is challenging as obstructions are common and non-rigid facial motion can compromise surface accuracy. The purpose of this work was to develop and benchmark the Remove the Mask (RtM) SGRT system, an open-source system especially designed to address the challenges faced in radiotherapy of head and neck cancer. Materials and MethodsThe accuracy of the RtM SGRT system was benchmarked using a head phantom positioned on a robotic motion platform capable of sub-millimetre accuracy which was used to induce unidirectional shifts and to reproduce three real head motion traces. We also assessed the accuracy of the system in ten humans volunteers. The ground truth motion of the volunteers was obtained using a commercial motion capture system with an accuracy < 0.3 mm. ResultsThe mean tracking error of the RtM SGRT system for the ten volunteers was of −0.1 ± 0.4 mm −0.6 ± 0.6 mm and 0.3 ± 0.2 mm, and 0.0 ± 0.2° 0.0 ± 0.1° and 0.0 ± 0.2° for translations and rotations along the left–right, superior-inferior and anterior-posterior axes respectively and we also found similar results in measurements with the head phantom. Forced facial motion was associated with lower tracking accuracy. The RtM SGRT system achieved submillimetre accuracy. ConclusionThe RtM SGRT system is a low-cost, easy to build and open-source SGRT system that can achieve an accuracy that meets international commissioning guidelines. Its open-source and modular design allows for the development and easy translation of novel surface tracking techniques.

Head2Head++: Deep Facial Attributes Re-Targeting

Facial video re-targeting is a challenging problem aiming to modify the facial attributes of a target subject in a seamless manner by a driving monocular sequence. We leverage the 3D geometry of faces and Generative Adversarial Networks (GANs) to design a novel deep learning architecture for the task of facial and head reenactment. Our method is different to purely 3D model-based approaches, or recent image-based methods that use Deep Convolutional Neural Networks (DCNNs) to generate individual frames. We manage to capture the complex non-rigid facial motion from the driving monocular performances and synthesise temporally consistent videos, with the aid of a sequential Generator and an ad-hoc Dynamics Discriminator network. We conduct a comprehensive set of quantitative and qualitative tests and demonstrate experimentally that our proposed method can successfully transfer facial expressions, head pose and eye gaze from a source video to a target subject, in a photo-realistic and faithful fashion, better than other state-of-the-art methods. Most importantly, our system performs end-to-end reenactment in nearly real-time speed (18 fps).

Anterior superior temporal sulcus is specialized for non-rigid facial motion in both monkeys and humans

Facial motion plays a fundamental role in the recognition of facial expressions in primates, but the neural substrates underlying this special type of biological motion are not well understood. Here, we used fMRI to investigate the extent to which the specialization for facial motion is represented in the visual system and compared the neural mechanisms for the processing of non-rigid facial motion in macaque monkeys and humans. We defined the areas specialized for facial motion as those significantly more activated when subjects perceived the motion caused by dynamic faces (dynamic faces ​> ​static faces) than when they perceived the motion caused by dynamic non-face objects (dynamic objects ​> ​static objects). We found that, in monkeys, significant activations evoked by facial motion were in the fundus of anterior superior temporal sulcus (STS), which overlapped the anterior fundus face patch. In humans, facial motion activated three separate foci in the right STS: posterior, middle, and anterior STS, with the anterior STS location showing the most selectivity for facial motion compared with other facial motion areas. In both monkeys and humans, facial motion shows a gradient preference as one progresses anteriorly along the STS. Taken together, our results indicate that monkeys and humans share similar neural substrates within the anterior temporal lobe specialized for the processing of non-rigid facial motion.

An Amalgamated Probabilistic Structure for Unconstrained Facial Activity Utilizing Dynamic Bayesian Network

Outward appearance was a characteristic and incredible method for human correspondence. Perceiving unconstrained facial activities in any case, is trying because of unpretentious facial miss happening, visit head developments and vague and unsure facial movement estimation. In light of these difficulties, ebb and flow look into in outward appearance acknowledgment is restricted to presented articulations and frequently in frontal view. An unconstrained outward appearance is described by inflexible head developments and non-rigid facial strong developments. All the more critically, it is the intelligible and steady spatiotemporal collaborations among unbending and non-rigid facial movements that produce an important outward appearance. Perceiving this reality, we present a bound together probabilistic facial activity model dependent on the Dynamic Bayesian system (DBN) to all the while and intelligibly speak to unbending and non-rigid facial movements, their spatiotemporal conditions, and their picture estimations. Propelled AI techniques are acquainted with gain proficiency with the model dependent on both preparing information and emotional earlier information. Given the model and the estimations of facial movements, facial activity acknowledgment is practiced through probabilistic surmising by deliberately incorporating visual estimations with the facial activity model. Analyses show that contrasted with the best in class strategies, the proposed framework yields huge enhancements in perceiving both inflexible and non-rigid facial movements, particularly for unconstrained outward appearances.

Robust heart-rate estimation from facial videos using Project_ICA

Objective: Remote photoplethysmography (rPPG) can achieve non-contact measurement of heart rate (HR) from a continuous video sequence by scanning the skin surface. However, practical applications are still limited by factors such as non-rigid facial motion and head movement. In this work, a detailed system framework for remotely estimating heart rate from facial videos under various movement conditions is described. Approach: After the rPPG signal has been obtained from a defined region of the facial skin, a method, termed ‘Project_ICA’, based on a skin reflection model, is employed to extract the pulse signal from the original signal. Main results: To evaluate the performance of the proposed algorithm, a dataset containing 112 videos including the challenges of various skin tones, body motion and HR recovery after exercise was created from 28 participants. Significance: The results show that Project_ICA, when evaluated by several criteria, provides a more accurate and robust estimate of HR than most existing methods, although problems remain in obtaining reliable measurements from dark-skinned subjects.

A comprehensive survey on automatic facial action unit analysis

Facial Action Coding System is the most influential sign judgment method for facial behavior, and it is a comprehensive and anatomical system which could encode various facial movements by the combination of basic AUs (Action Units). AUs define certain facial configurations caused by contraction of one or more facial muscles, and they are independent of interpretation of emotions. However, automatic facial action unit recognition remains challenging due to several open questions such as rigid and non-rigid facial motions, multiple AUs detection, intensity estimation and naturalistic context application. This paper introduces recent advances in automatic facial action unit recognition, focusing on the fundamental components of face detection and registration, facial action representation, feature selection and classification. The comprehensive analysis of facial representations is presented according to the facial data properties (image and video, 2D and 3D) and characteristics of facial features (predesign and learning, appearance and geometry, hybrid and fusion). Facial action unit recognition involves AUs occurrence detection, AUs temporal segment detection and AUs intensity estimation. We discussed the role of each component, main techniques with their characteristics, challenges and potential directions of facial action unit analysis.

An Approach to Detect the Region of Interest of Expressive Face Images

On human face, non-rigid facial movements due to facial expressions cause noticeable alterations in their usual shapes, which sometimes create occlusions in facial feature areas making face recognition as a difficult problem. The paper presents an automatic Region of Interest (ROI) detection technique of six universal expressive face images. The proposed technique is a facial geometric based hybrid approach. The localization accuracy was evaluated by rectangular error measure and was tested on Japanese Female Facial Expression (JAFFE) database. The average localization accuracy of all detected facial regions is 94%.

Distinct Representations for Rigid and Non-Rigid Facial Movements in Face-Selective Regions of the Human Brain

Distinct Representations for Rigid and Non-Rigid Facial Movements in Face-Selective Regions of the Human Brain

Automatic expression spotting in videos

In this paper, we propose a novel solution for the problem of segmenting macro- and micro-expression frames (or retrieving the expression intervals) in video sequences, which is a prior step for many expression recognition algorithms. The proposed method exploits the non-rigid facial motion that occurs during facial expressions by capturing the optical strain corresponding to the elastic deformation of facial skin tissue. The method is capable of spotting both macro-expressions which are typically associated with expressed emotions and rapid micro- expressions which are typically associated with semi-suppressed macro-expressions. We test our algorithm on several datasets, including a newly released hour-long video with two subjects recorded in a natural setting that includes spontaneous facial expressions. We also report results on a dataset that contains 75 feigned macro-expressions and 37 feigned micro-expressions. We achieve over a 75% true positive rate with a 1% false positive rate for macro-expressions, and a nearly 80% true positive rate for spotting micro-expressions with a .3% false positive rate.

Event-Related Alpha Suppression in Response to Facial Motion

While biological motion refers to both face and body movements, little is known about the visual perception of facial motion. We therefore examined alpha wave suppression as a reduction in power is thought to reflect visual activity, in addition to attentional reorienting and memory processes. Nineteen neurologically healthy adults were tested on their ability to discriminate between successive facial motion captures. These animations exhibited both rigid and non-rigid facial motion, as well as speech expressions. The structural and surface appearance of these facial animations did not differ, thus participants decisions were based solely on differences in facial movements. Upright, orientation-inverted and luminance-inverted facial stimuli were compared. At occipital and parieto-occipital regions, upright facial motion evoked a transient increase in alpha which was then followed by a significant reduction. This finding is discussed in terms of neural efficiency, gating mechanisms and neural synchronization. Moreover, there was no difference in the amount of alpha suppression evoked by each facial stimulus at occipital regions, suggesting early visual processing remains unaffected by manipulation paradigms. However, upright facial motion evoked greater suppression at parieto-occipital sites, and did so in the shortest latency. Increased activity within this region may reflect higher attentional reorienting to natural facial motion but also involvement of areas associated with the visual control of body effectors.

Event-related alpha suppression in response to facial motion.

While biological motion refers to both face and body movements, little is known about the visual perception of facial motion. We therefore examined alpha wave suppression as a reduction in power is thought to reflect visual activity, in addition to attentional reorienting and memory processes. Nineteen neurologically healthy adults were tested on their ability to discriminate between successive facial motion captures. These animations exhibited both rigid and non-rigid facial motion, as well as speech expressions. The structural and surface appearance of these facial animations did not differ, thus participants decisions were based solely on differences in facial movements. Upright, orientation-inverted and luminance-inverted facial stimuli were compared. At occipital and parieto-occipital regions, upright facial motion evoked a transient increase in alpha which was then followed by a significant reduction. This finding is discussed in terms of neural efficiency, gating mechanisms and neural synchronization. Moreover, there was no difference in the amount of alpha suppression evoked by each facial stimulus at occipital regions, suggesting early visual processing remains unaffected by manipulation paradigms. However, upright facial motion evoked greater suppression at parieto-occipital sites, and did so in the shortest latency. Increased activity within this region may reflect higher attentional reorienting to natural facial motion but also involvement of areas associated with the visual control of body effectors.

Smooth adaptive fitting of 3D face model for the estimation of rigid and nonrigid facial motion in video sequences

We propose a 3D wireframe face model alignment for the task of simultaneously tracking of rigid head motion and nonrigid facial expressions in video sequences. The system integrates two levels: (i) at the low level, automatic and accurate location of facial features are obtained via a cascaded optimization algorithm of a 2D shape model, (ii) at the high level, we recover, via minimizing an energy function, the optimal motion parameters of the 3D model, namely the 3D rigid motion parameters and seven nonrigid animation (Action Unit) parameters. In this latter inference, a 3D face shape model (Candide) is automatically fitted to the image sequence via a least squares minimization of the energy, defined as the residual between the projected 3D wireframe model and the 2D shape model, meanwhile imposing temporal and spatial motion-smoothness constraints over the 3D model points. Our proposed system tackles many disadvantages of the optimization and training associated with active appearance models. Extensive fitting and tracking experiments demonstrate the feasibility, accuracy and effectiveness of the developed methods. Qualitative and quantitative performance of the proposed system on several facial sequences, indicate its potential usefulness for multimedia applications, as well as facial expression analysis.

A Unified Probabilistic Framework for Spontaneous Facial Action Modeling and Understanding

Facial expression is a natural and powerful means of human communication. Recognizing spontaneous facial actions, however, is very challenging due to subtle facial deformation, frequent head movements, and ambiguous and uncertain facial motion measurements. Because of these challenges, current research in facial expression recognition is limited to posed expressions and often in frontal view. A spontaneous facial expression is characterized by rigid head movements and nonrigid facial muscular movements. More importantly, it is the coherent and consistent spatiotemporal interactions among rigid and nonrigid facial motions that produce a meaningful facial expression. Recognizing this fact, we introduce a unified probabilistic facial action model based on the Dynamic Bayesian network (DBN) to simultaneously and coherently represent rigid and nonrigid facial motions, their spatiotemporal dependencies, and their image measurements. Advanced machine learning methods are introduced to learn the model based on both training data and subjective prior knowledge. Given the model and the measurements of facial motions, facial action recognition is accomplished through probabilistic inference by systematically integrating visual measurements with the facial action model. Experiments show that compared to the state-of-the-art techniques, the proposed system yields significant improvements in recognizing both rigid and nonrigid facial motions, especially for spontaneous facial expressions.

Non-rigid face tracking with enforced convexity and local appearance consistency constraint

Convex quadratic fitting (CQF) has demonstrated great success recently in the task of non-rigidly registering a face in a still image using a constrained local model (CLM). A CLM is a commonly used model for non-rigid object registration and contains two components: (i) local patch-experts that model the appearance of each landmark in the object, and (ii) a global shape prior describing how each of these landmarks can vary non-rigidly. Conventional CLMs can be used in non-rigid facial tracking applications through a track-by-detection strategy. However, the registration performance of such a strategy is susceptible to local appearance ambiguity. Since there is no motion continuity constraint between neighboring frames of the same sequence, the resultant object alignment might not be consistent from frame to frame and the motion field is not temporally smooth. In this paper, we extend the CQF fitting method into the spatio-temporal domain by enforcing the appearance consistency constraint of each local patch between neighboring frames. More importantly, we show, as in the original CQF formulation, that the global warp update can be optimized jointly in an efficient manner. Finally, we demonstrate that our approach receives improved performance for the task of non-rigid facial motion tracking on the videos of clinical patients.

Natural facial motion enhances cortical responses to faces

The ability to perceive facial motion is important to successfully interact in social environments. Previously, imaging studies have investigated neural correlates of facial motion primarily using abstract motion stimuli. Here, we studied how the brain processes natural non-rigid facial motion in direct comparison to static stimuli and matched phase-scrambled controls. As predicted from previous studies, dynamic faces elicit higher responses than static faces in lateral temporal areas corresponding to hMT+/V5 and STS. Interestingly, individually defined, static-face-sensitive regions in bilateral fusiform gyrus and left inferior occipital gyrus also respond more to dynamic than static faces. These results suggest integration of form and motion information during the processing of dynamic faces even in ventral temporal and inferior lateral occipital areas. In addition, our results show that dynamic stimuli are a robust tool to localize areas related to the processing of static and dynamic face information.

A Factorization-Based Approach for Articulated Nonrigid Shape, Motion and Kinematic Chain Recovery From Video

Recovering articulated shape and motion, especially human body motion, from video is a challenging problem with a wide range of applications in medical study, sport analysis and animation, etc. Previous work on articulated motion recovery generally requires prior knowledge of the kinematic chain and usually does not concern the recovery of the articulated shape. The non-rigidity of some articulated part, e.g. human body motion with nonrigid facial motion, is completely ignored. We propose a factorization-based approach to recover the shape, motion and kinematic chain of an articulated object with nonrigid parts altogether directly from video sequences under a unified framework. The proposed approach is based on our modeling of the articulated non-rigid motion as a set of intersecting motion subspaces. A motion subspace is the linear subspace of the trajectories of an object. It can model a rigid or non-rigid motion. The intersection of two motion subspaces of linked parts models the motion of an articulated joint or axis. Our approach consists of algorithms for motion segmentation, kinematic chain building, and shape recovery. It handles outliers and can be automated. We test our approach through synthetic and real experiments and demonstrate how to recover articulated structure with non-rigid parts via a single-view camera without prior knowledge of its kinematic chain.

A search advantage for faces learned in motion

Recently there has been growing interest in the role that motion might play in the perception and representation of facial identity. Most studies have considered old/new recognition as a task. However, especially for non-rigid motion, these studies have often produced contradictory results. Here, we used a delayed visual search paradigm to explore how learning is affected by non-rigid facial motion. In the current studies we trained observers on two frontal view faces, one moving non-rigidly, the other a static picture. After a delay, observers were asked to identify the targets in static search arrays containing 2, 4 or 6 faces. On a given trial target and distractor faces could be shown in one of five viewpoints, frontal, 22 degrees or 45 degrees to the left or right. We found that familiarizing observers with dynamic faces led to a constant reaction time advantage across all setsizes and viewpoints compared to static familiarization. This suggests that non-rigid motion affects identity decisions even across extended periods of time and changes in viewpoint. Furthermore, it seems as if such effects may be difficult to observe using more traditional old/new recognition tasks.

The use of facial motion and facial form during the processing of identity

Previous research has shown that facial motion can carry information about age, gender, emotion and, at least to some extent, identity. By combining recent computer animation techniques with psychophysical methods, we show that during the computation of identity the human face recognition system integrates both types of information: individual non-rigid facial motion and individual facial form. This has important implications for cognitive and neural models of face perception, which currently emphasize a separation between the processing of invariant aspects (facial form) and changeable aspects (facial motion) of faces.

A matching advantage for dynamic human faces.

In a series of three experiments, we used a sequential matching task to explore the impact of non-rigid facial motion on the perception of human faces. Dynamic prime images, in the form of short video sequences, facilitated matching responses relative to a single static prime image. This advantage was observed whenever the prime and target showed the same face but an identity match was required across expression (experiment 1) or view (experiment 2). No facilitation was observed for identical dynamic prime sequences when the matching dimension was shifted from identity to expression (experiment 3). We suggest that the observed dynamic advantage, the first reported for non-degraded facial images, arises because the matching task places more emphasis on visual working memory than typical face recognition tasks. More specifically, we believe that representational mechanisms optimised for the processing of motion and/or change-over-time are established and maintained in working memory and that such 'dynamic representations' (Freyd, 1987 Psychological Review 94 427-438) capitalise on the increased information content of the dynamic primes to enhance performance.

Categorizing sex and identity from the biological motion of faces

Head and facial movements can provide valuable cues to identity in addition to their primary roles in communicating speech and expression [1–8]. Here we report experiments in which we have used recent motion capture and animation techniques to animate an average head [9]. These techniques have allowed the isolation of motion from other cues and have enabled us to separate rigid translations and rotations of the head from nonrigid facial motion. In particular, we tested whether human observers can judge sex and identity on the basis of this information. Results show that people can discriminate both between individuals and between males and females from motion-based information alone. Rigid head movements appear particularly useful for categorization on the basis of identity, while nonrigid motion is more useful for categorization on the basis of sex. Accuracy for both sex and identity judgements is reduced when faces are presented upside down, and this finding shows that performance is not based on low-level motion cues alone and suggests that the information is represented in an object-based motion-encoding system specialized for upright faces. Playing animations backward also reduced performance for sex judgements and emphasized the importance of direction specificity in admitting access to stored representations of characteristic male and female movements.