Facial Reenactment Research Articles

Simultaneous control of head pose and expressions in 3D facial keypoint-based GAN

In this work, we present a novel method for simultaneously controlling the head pose and the facial expressions of a given input image using a 3D keypoint-based GAN. Existing methods for controlling head pose and expressions simultaneously are not suitable for real images, or they generate unnatural results because it is not trivial to capture head pose (large changes) and expressions (small changes) simultaneously. In this work, we achieve simultaneous control of head pose and facial expressions by introducing 3D facial keypoints for GAN-based facial image synthesis, unlike the existing 2D landmark-based approach. As a result, our method can handle both large variations due to different head poses and subtle variations due to changing facial expressions faithfully. Furthermore, our model takes audio input as an additional modality for further enhancing the quality of generated images. Our model was evaluated on the VoxCeleb2 dataset to demonstrate its state-of-the-art performance for both facial reenactment and facial image manipulation tasks, and our model tends not to be affected by the driving images.

Self-Information Forgery Mining for Face Forgery Detection

Abstract In the face of rapid advances in face forgery technology, effective detection methods have become crucial to maintain the authenticity of digital media. Deep learning technology has provided new strategies for recognizing and preventing face forgery in recent years. In this study, a new face forgery detection technique is proposed by utilizing self-information theory, which improves the accuracy and robustness of detection by mining forgery traces, especially in diverse forgery scenarios. The study extracts face features through an improved high-resolution network HRNet and optimizes identity information extraction by combining facial reenactment techniques to detect forged faces efficiently. Experiments have been conducted on several mainstream forged face datasets, and the method presented in this paper can effectively improve the detection performance with an average accuracy of 74.75% on C40 recompressed images. Comparison experiments show that this research method’s frame-level and video-level detection accuracy on the Celeb-DF dataset are 0.9846 and 0.9985, respectively, which are higher than those of existing techniques. Cross-library tests validate the method’s generalization performance, and the AUC metric remains at 0.7305 even in low-quality video environments, which shows good resistance to environmental interference. This study proposes a self-information forgery mining technique that enhances forgery detection accuracy while demonstrating superior generalization ability.

HEADSET: Human Emotion Awareness under Partial Occlusions Multimodal DataSET.

The volumetric representation of human interactions is one of the fundamental domains in the development of immersive media productions and telecommunication applications. Particularly in the context of the rapid advancement of Extended Reality (XR) applications, this volumetric data has proven to be an essential technology for future XR elaboration. In this work, we present a new multimodal database to help advance the development of immersive technologies. Our proposed database provides ethically compliant and diverse volumetric data, in particular 27 participants displaying posed facial expressions and subtle body movements while speaking, plus 11 participants wearing head-mounted displays (HMDs). The recording system consists of a volumetric capture (VoCap) studio, including 31 synchronized modules with 62 RGB cameras and 31 depth cameras. In addition to textured meshes, point clouds, and multi-view RGB-D data, we use one Lytro Illum camera for providing light field (LF) data simultaneously. Finally, we also provide an evaluation of our dataset employment with regard to the tasks of facial expression classification, HMDs removal, and point cloud reconstruction. The dataset can be helpful in the evaluation and performance testing of various XR algorithms, including but not limited to facial expression recognition and reconstruction, facial reenactment, and volumetric video. HEADSET and its all associated raw data and license agreement will be publicly available for research purposes.

Digital Face Manipulation Creation and Detection: A Systematic Review

The introduction of publicly available large-scale datasets and advances in generative adversarial networks (GANs) have revolutionized the generation of hyper-realistic facial images, which are difficult to detect and can rapidly reach millions of people, with adverse impacts on the community. Research on manipulated facial image detection and generation remains scattered and in development. This survey aimed to address this gap by providing a comprehensive analysis of the methods used to produce manipulated face images, with a focus on deepfake technology and emerging techniques for detecting fake images. The review examined four key groups of manipulated face generation techniques: (1) attributes manipulation, (2) facial re-enactment, (3) face swapping, and (4) face synthesis. Through an in-depth investigation, this study sheds light on commonly used datasets, standard manipulated face generation/detection approaches, and benchmarking methods for each manipulation group. Particular emphasis is placed on the advancements and detection techniques related to deepfake technology. Furthermore, the paper explores the benefits of analyzing deepfake while also highlighting the potential threats posed by this technology. Existing challenges in the field are discussed, and several directions for future research are proposed to tackle these challenges effectively. By offering insights into the state of the art for manipulated face image detection and generation, this survey contributes to the advancement of understanding and combating the misuse of deepfake technology.

Steganography-based facial re-enactment using generative adversarial networks

Steganography-based facial re-enactment using generative adversarial networks

Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait

With the continuous development of cross-modality generation, audio-driven talking face generation has made substantial advances in terms of speech content and mouth shape, but existing research on talking face emotion generation is still relatively unsophisticated. In this work, we present Emotionally Controllable Talking Face Generation from an Arbitrary Emotional Portrait to synthesize lip-sync and an emotionally controllable high-quality talking face. Specifically, we take a facial reenactment perspective, using facial landmarks as an intermediate representation driving the expression generation of talking faces through the landmark features of an arbitrary emotional portrait. Meanwhile, decoupled design ideas are used to divide the model into three sub-networks to improve emotion control. They are the lip-sync landmark animation generation network, the emotional landmark animation generation network, and the landmark-to-animation translation network. The two landmark animation generation networks are responsible for generating content-related lip area landmarks and facial expression landmarks to correct the landmark sequences of the target portrait. Following this, the corrected landmark sequences and the target portrait are fed into the translation network to generate an emotionally controllable talking face. Our method controls the expressions of talking faces by driving the emotional portrait images while ensuring the generation of animated lip-sync, and can handle new audio and portraits not seen during training. A multi-perspective user study and extensive quantitative and qualitative evaluations demonstrate the superiority of the system in terms of visual emotion representation and video authenticity.

One-shot many-to-many facial reenactment using Bi-Layer Graph Convolutional Networks

Facial reenactment is aimed at animating a source face image into a new place using a driving facial picture. In a few shot scenarios, the present strategies are designed with one or more identities or identity-sustained suffering protection challenges. These current solutions are either developed with one or more identities in mind, or face identity protection issues in one or more shot situations. Multiple pictures from the same entity have been used in previous research to model facial reenactment. In contrast, this paper presents a novel model of one-shot many-to-many facial reenactments that uses only one facial image of a face. The proposed model produces a face that represents the objective representation of the same source identity. The proposed technique can simulate motion from a single image by decomposing an object into two layers. Using bi-layer with Convolutional Neural Network (CNN), we named our model Bi-Layer Graph Convolutional Layers (BGCLN) which utilized to create the latent vector’s optical flow representation. This yields the precise structure and shape of the optical stream. Comprehensive studies suggest that our technique can produce high-quality results and outperform most recent techniques in both qualitative and quantitative data comparisons. Our proposed system can perform facial reenactment at 15 fps, which is approximately real time. Our code is publicly available at https://github.com/usaeed786/BGCLN.

Identity Manipulation: Responding to Advances in Artificial Intelligence and Robotics

Advances in artificial intelligence (AI) and robotics technologies have destabilized our ability to control our identity. Today, it is increasingly accessible for the average person to appropriate the voice, image, and body of another individual through the use of technology. Deepfake videos swap new faces into existing videos, facial re-enactment allows for the face of one person to be superimposed on the face of someone else in a real time video, artificial speech synthesis can clone another person’s voice, and 3D printing and modern robotics can reproduce life-size copies of living people. These are all examples of the ways technology is being used to replicate aspects of people’s identity, sometimes without their consent. Once limited to the hands of computer-generated image specialists and robotics professionals, the barriers to accessing these technologies are quickly being removed. By scrapping audio-visual data from digitally available sources or purchasing robotics parts on the global marketplace, even hobbyists and amateurs are able to experiment with many of these technologies and create content representing another person. While many of these technologies are in their infancy and are not yet successfully producing perfect replicas, there have already been examples wherein this technology has produced convincing copies of aspects of the targeted individual that have effectively fooled other people.10 As these technologies become more user friendly, less expensive, and produce more realistic content, it is imperative that we explore the law’s role in protecting an individual’s identity from these types of identity manipulation. To explore these issues, this paper will first discuss why an individual’s identity, particularly representations of their image and voice, are increasingly relevant in our current society. Second, it will cover some existing technologies to illustrate the ways that identities are being manipulated through artificial intelligence and robotics, and highlight some aspects of their production that may be relevant when applying legal protections. Third, it will explore the economically focused torts of appropriation of personality and publicity rights in common law Canada and the United States, respectively, as well as dignity focused personality and image rights in the civil law systems of Germany and Quebec. Fourth, it will argue that legal solutions to identity violations should include more of a dignity focused approach that takes a broad understanding of identity to protect people’s identities from being harmed by misuses of this technology.

Deferred neural rendering

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering , a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures , which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

Face2Face: Real-time facial reenactment

Abstract This article summarizes the dissertation “Face2Face: Realtime Facial Reenactment” by Justus Thies (Eurographics Graphics Dissertation Online, 2017). It shows advances in the field of 3D reconstruction of human faces using commodity hardware. Besides the reconstruction of the facial geometry and texture, real-time face tracking is demonstrated. The developed algorithms are based on the principle of analysis-by-synthesis. To apply this principle, a mathematical model that represents a face virtually is defined. Utilizing this model to synthesize facial imagery, the model parameters are adjusted, such that the synthesized image fits the input image as good as possible. Thus, in reverse, this process transfers the input image to a virtual representation of the face. The achieved quality allows many new applications that require a good reconstruction of the face. One of these applications is the so-called “Facial Reenactment”. Our developed methods show that such an application does not need any special hardware. The generated results are nearly photo-realistic videos that show the transfer of the expressions of one person to another person. These techniques can for example be used to bring movie dubbing to a new level. Instead of adapting the audio to the video, which might also include changes of the text, the video can be post-processed to match the mouth movements of the dubber. Since the approaches that are shown in the dissertation run in real-time, one can also think of a live dubber in a video teleconferencing system that simultaneously translates the speech of a person to another language. The published videos of the projects in this dissertation led to a broad discussion in the media. On the one hand this is due to the fact that our methods are designed such that they run in real-time and on the other hand that we reduced the hardware requirements to a minimum while improving the resulting quality. In fact, after some preprocessing, we are able to edit ordinary videos from the Internet in real-time. Amongst others, we impose a different mimic to faces of prominent persons like former presidents of the United States of America. This led inevitably to a discussion about trustworthiness of video material, especially from unknown source. Most people did not expect that such manipulations are possible, neglecting existing methods that are already able to edit videos (e. g. special effects in movie productions). Thus, besides the advances in real-time face tracking, our projects raised the awareness of video manipulation.

Face2Face

Face2Face is an approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time. This live setup has also been shown at SIGGRAPH Emerging Technologies 2016, by Thies et al. where it won the Best in Show Award.

Anchor Cascade for Efficient Face Detection.

Face detection is essential to facial analysis tasks such as facial reenactment and face recognition. Both cascade face detectors and anchor-based face detectors have translated shining demos into practice and received intensive attention from the community. However, cascade face detectors often suffer from a low detection accuracy, while anchor-based face detectors rely heavily on very large neural networks pre-trained on large scale image classification datasets such as ImageNet [1], which is not computationally efficient for both training and deployment. In this paper, we devise an efficient anchor-based cascade framework called anchor cascade. To improve the detection accuracy by exploring contextual information, we further propose a context pyramid maxout mechanism for anchor cascade. As a result, anchor cascade can train very efficient face detection models with a high detection accuracy. Specifically, comparing with a popular CNN-based cascade face detector MTCNN [2], our anchor cascade face detector greatly improves the detection accuracy, e.g., from 0.9435 to 0.9704 at 1k false positives on FDDB, while it still runs in comparable speed. Experimental results on two widely used face detection benchmarks, FDDB and WIDER FACE, demonstrate the effectiveness of the proposed framework.

State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

AbstractThe computer graphics and vision communities have dedicated long standing efforts in building computerized tools for reconstructing, tracking, and analyzing human faces based on visual input. Over the past years rapid progress has been made, which led to novel and powerful algorithms that obtain impressive results even in the very challenging case of reconstruction from a single RGB or RGB‐D camera. The range of applications is vast and steadily growing as these technologies are further improving in speed, accuracy, and ease of use.Motivated by this rapid progress, this state‐of‐the‐art report summarizes recent trends in monocular facial performance capture and discusses its applications, which range from performance‐based animation to real‐time facial reenactment. We focus our discussion on methods where the central task is to recover and track a three dimensional model of the human face using optimization‐based reconstruction algorithms. We provide an in‐depth overview of the underlying concepts of real‐world image formation, and we discuss common assumptions and simplifications that make these algorithms practical. In addition, we extensively cover the priors that are used to better constrain the under‐constrained monocular reconstruction problem, and discuss the optimization techniques that are employed to recover dense, photo‐geometric 3D face models from monocular 2D data. Finally, we discuss a variety of use cases for the reviewed algorithms in the context of motion capture, facial animation, as well as image and video editing.

Real-time expression transfer for facial reenactment

We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor. The novelty of our approach lies in the transfer and photorealistic re-rendering of facial deformations and detail into the target video in a way that the newly-synthesized expressions are virtually indistinguishable from a real video. To achieve this, we accurately capture the facial performances of the source and target subjects in real-time using a commodity RGB-D sensor. For each frame, we jointly fit a parametric model for identity, expression, and skin reflectance to the input color and depth data, and also reconstruct the scene lighting. For expression transfer, we compute the difference between the source and target expressions in parameter space, and modify the target parameters to match the source expressions. A major challenge is the convincing re-rendering of the synthesized target face into the corresponding video stream. This requires a careful consideration of the lighting and shading design, which both must correspond to the real-world environment. We demonstrate our method in a live setup, where we modify a video conference feed such that the facial expressions of a different person (e.g., translator) are matched in real-time.