Facial Editing Research Articles

Deep Realistic Facial Editing via Label-restricted Mask Disentanglement.

With the rapid development of GAN (generative adversarial network), recent years have witnessed an increasing number of tasks on reference-guided facial attributes transfer. Most state-of-the-art methods consist of facial information extraction, latent space disentanglement, and target attribute manipulation. However, they either adopt reference-guided translation methods for manipulation or monolithic modules for diverse attribute exchange, which cannot accurately disentangle the exact facial attributes with specific styles from the reference image. In this paper, we propose a deep realistic facial editing method (termed LMGAN) based on target region focusing and dual label constraint. The proposed method, manipulating target attributes by latent space exchange, consists of subnetworks for every individual attribute. Each subnetwork exerts label-restrictions on both the target attributes exchanging stage and the training process aimed at optimizing generative quality and reference-style correlation. Our method performs greatly on disentangled representation and transferring the target attribute's style accurately. A global discriminator is introduced to combine the generated editing regional image with other nonediting areas of the source image. Both qualitative and quantitative results on the CelebA dataset verify the ability of the proposed LMGAN.

IA-FaceS: A bidirectional method for semantic face editing

Semantic face editing has achieved substantial progress in recent years. However, existing face editing methods, which often encode the entire image into a single code, still have difficulty in enabling flexible editing while keeping high-fidelity reconstruction. The one-code scheme also brings entangled face manipulations and limited flexibility in editing face components. In this paper, we present IA-FaceS, a bidirectional method for disentangled face attribute manipulation as well as flexible, controllable component editing. We propose to embed images onto two branches: one branch computes high-dimensional component-invariant content embedding for capturing face details, and the other provides low-dimensional component-specific embeddings for component manipulations. The two-branch scheme naturally enables high-quality facial component-level editing while keeping faithful reconstruction with details. Moreover, we devise a component adaptive modulation (CAM) module, which integrates component-specific guidance into the decoder and successfully disentangles highly-correlated face components. The single-eye editing is developed for the first time without editing face masks or sketches. According to the experimental results, IA-FaceS establishes a good balance between maintaining image details and performing flexible face manipulation. Both quantitative and qualitative results indicate that the proposed method outperforms the existing methods in reconstruction, face attribute manipulation, and component transfer. We release the code and weights at: https://github.com/CMACH508/IA-FaceS.

Disentangled face editing via individual walk in personalized facial semantic field

Disentangled face editing via individual walk in personalized facial semantic field

Controllable face editing for video reconstruction in human digital twins

High fidelity and controllable manipulation are critical to facial video reconstruction in human digital twins. Current generative adversarial networks (GANs) have achieved impressive performance in realistic face generation with high resolution, motivating several recent works to perform face editing via pretrained GANs. However, existing works suffer identity loss and semantic entanglement while editing real faces. To tackle these limitations, we propose a framework to perform controllable facial editing in video reconstruction. First, we propose to train a semantic inversion network to embed the target attribute change into the latent space of GANs. Disentangled semantic manipulation is performed during the semantic inversion by changing only the target attribute with the other unrelated attributes kept. Furthermore, we propose a novel personalized GAN inversion for the real face cropped from videos via retraining the generator of GANs, which can embed the real face into the latent space of GANs and preserve identity details for the real face. Finally, the realistic edited face is fused back into the original video. We use the identity preservation rate and disentanglement rate to evaluate the performance of our controllable face editing. Both qualitative and quantitative evaluations show that our method achieves prominent identity preservation and semantic disentanglement in controllable face editing, outperforming recent state-of-the-art methods. • Controllable face editing to videos with high resolution at 1920*1080. • Semantic inversion networks trained for disentangled facial editing. • Personalized latent spaces trained for identity preservation. • Outperforming current state-of-the-art works in extensive experiments.

Pivotal Tuning for Latent-based Editing of Real Images

Recently, numerous facial editing techniques have been proposed that leverage the generative power of a pretrained StyleGAN. To successfully edit an image this way, one must first project (or invert) the image into the pretrained generator’s domain. As it turns out, StyleGAN’s latent space induces an inherent tradeoff between distortion and editability, i.e., between maintaining the original appearance and convincingly altering its attributes. Hence, it remains challenging to apply ID-preserving edits to real facial images. In this article, we present an approach to bridge this gap. The idea is Pivotal Tuning —a brief training process that preserves editing quality, while surgically changing the portrayed identity and appearance. In Pivotal Tuning Inversion, an initial inverted latent code serves as a pivot, around which the generator is fine-tuned. At the same time, a regularization term keeps nearby identities intact, to locally contain the effect. We further show that Pivotal Tuning also applies to accommodating for a multitude of faces, while introducing negligible distortion on the rest of the domain. We validate our technique through inversion and editing metrics and show preferable scores to state-of-the-art methods. Last, we present successful editing for harder cases, including elaborate make-up or headwear.

SwapInpaint: Identity-Specific Face Inpainting With Identity Swapping

As face editing scenarios have become popular, the face inpainting technique has become a hot topic. Although some existing methods can inpaint faces with preserved identity information, they fail to solve a more flexible inpainting problem that fills the “holes” with identity-specific content from other faces. In this work, we propose a disentangle and subject-agnostic framework that affects both full and partial-face inpainting with the guidance of a reference face image. The framework consists of an identity encoding module, a content inference module and a generative module. The identity encoding module extracts the identity embedding from the reference image, the content inference module learns to predict the content image, and the generative module integrates the content image and the reference identity embedding to generate the identity-specific inpainted result. To minimize the structure and style gap between the incomplete image and inpainted image, we use a double attribute loss to the generative module and a postprocess of blending operation to the swapped result. We compare our method with state-of-the-art works and demonstrate that our method achieves higher identity similarity and better structural correctness.

Research Progress of Face Editing Based on Deep Generative Model

Research Progress of Face Editing Based on Deep Generative Model

A Survey of Anti-forensic for Face Image Forgery

Deep learning related technologies, especially generative adversarial network, are widely used in the fields of face image tampering and forgery. Forensics researchers have proposed a variety of passive forensic and related anti-forensic methods for image tampering and forgery, especially face images, but there is still a lack of overview of anti-forensic methods at this stage. Therefore, this paper will systematically discuss the anti-forensic methods for face image tampering and forgery. Firstly, this paper expounds the relevant background, including the relevant tampering and forgery methods and forensic schemes of face images. The former mainly includes four aspects: conventional processing, fake face generation, face editing and face swapping; The latter is mainly the relevant forensic means based on spatial domain and frequency domain using deep learning technology. Then, this paper divides the existing anti-forensic works into three categories according to their method characteristics, namely hiding operation traces, forgery reconstruction and adversarial attack. Finally, this paper summarizes the limitations and prospects of the existing anti-forensic technologies.

Transforming the Latent Space of Stylegan for Real Face Editing

Despite recent advances in semantic manipulation using StyleGAN, semantic editing of real faces remains challenging. The gap between the $W$ space and the $W$+ space demands an undesirable trade-off between reconstruction quality and editing quality. To solve this problem, we propose to expand the latent space by replacing fully-connected layers in the StyleGAN's mapping network with attention-based transformers. This simple and effective technique integrates the aforementioned two spaces and transforms them into one new latent space called $W$++. Our modified StyleGAN maintains the state-of-the-art generation quality of the original StyleGAN with moderately better diversity. But more importantly, the proposed $W$++ space achieves superior performance in both reconstruction quality and editing quality. Despite these significant advantages, our $W$++ space supports existing inversion algorithms and editing methods with only negligible modifications thanks to its structural similarity with the $W/W$+ space. Extensive experiments on the FFHQ dataset prove that our proposed $W$++ space is evidently more preferable than the previous $W/W$+ space for real face editing. The code is publicly available for research purposes at https://github.com/AnonSubm2021/TransStyleGAN.

Multi-Density Sketch-to-Image Translation Network

Sketch-to-image (S2I) translation plays an important role in image synthesis and manipulation tasks, such as photo editing and colorization. Some specific S2I translations, including sketch-to-photo and sketch-to-painting, can be used as powerful tools in the art design industry. However, previous methods only support S2I translation with a single level of density, which gives less flexibility to users for controlling the input sketches. In this work, we propose the first multi-level density sketch-to-image translation framework, which allows the input sketch to cover a wide range from rough object outlines to microstructures. Moreover, to tackle the problem of noncontinuous representation of multi-level density input sketches, we project the density level into a continuous latent space, which can then be linearly controlled by a parameter. This allows users to conveniently control the densities of input sketches and the generation of images. Moreover, our method has been successfully verified on various datasets for different applications, including face editing, multi-modal sketch-to-photo translation, and anime colorization, providing coarse-to-fine levels of controls to these applications.

Towards Disentangling Latent Space for Unsupervised Semantic Face Editing.

Facial attributes in StyleGAN generated images are entangled in the latent space which makes it very difficult to independently control a specific attribute without affecting the others. Supervised attribute editing requires annotated training data which is difficult to obtain and limits the editable attributes to those with labels. Therefore, unsupervised attribute editing in an disentangled latent space is key to performing neat and versatile semantic face editing. In this paper, we present a new technique termed Structure-Texture Independent Architecture with Weight Decomposition and Orthogonal Regularization (STIA-WO) to disentangle the latent space for unsupervised semantic face editing. By applying STIA-WO to GAN, we have developed a StyleGAN termed STGAN-WO which performs weight decomposition through utilizing the style vector to construct a fully controllable weight matrix to regulate image synthesis, and employs orthogonal regularization to ensure each entry of the style vector only controls one independent feature matrix. To further disentangle the facial attributes, STGAN-WO introduces a structure-texture independent architecture which utilizes two independently and identically distributed (i.i.d.) latent vectors to control the synthesis of the texture and structure components in a disentangled way. Unsupervised semantic editing is achieved by moving the latent code in the coarse layers along its orthogonal directions to change texture related attributes or changing the latent code in the fine layers to manipulate structure related ones. We present experimental results which show that our new STGAN-WO can achieve better attribute editing than state of the art methods.

PattGAN: Pluralistic Facial Attribute Editing

Facial attribute editing has witnessed substantial progress due to the increasing power of generative adversarial networks. However, existing methods mainly focus on improving the quality of facial attribute editing but ignore the diversity of facial attribute editing, which can only generate a single editing result by binary attribute labels and cannot reveal the diversity of attribute styles. To overcome this limitation, we propose a novel pluralistic facial attribute editing approach termed PattGAN (pluralistic attribute GAN). Instead of directly using the binary attribute labels to guide facial attribute editing, PattGAN first learns the disentangled representation of facial attributes by the binary attribute labels and then uses the disentangled representation of attributes to guide facial attribute editing. To this end, an independent encoder called the “attribute encoder” is introduced to extract distributions of specific attributes from face images. Furthermore, a novel swapping strategy is designed to assist the attribute encoder in modeling the disentangled representation of facial attributes by enhancing the model’s ability to learn pluralistic attributes. Coupled with the classification loss, the attribute encoder can accurately separate attribute-related information from face images. Both qualitative and quantitative experiments on the CelebA datasets show that PattGAN can achieve diverse face editing results by different exemplars. In summary, rather than other state-of-the-art methods, PattGAN performs better in diversity facial attribute editing.

A Survey on Various Image Inpainting T arious Image Inpainting Techniques echniques

Over the decades' researchers have studied the image inpainting problem intensively due to its high significance and effectiveness in various image processing applications such as people and object security, object removal, face editing applications. Image inpainting is defined as the process of completing or removing a missing region in images. It is considered one of the most challenging topics in the image processing field, although, it requires a deep understanding of the image details in terms of texture and structure. In this paper, a survey of most image inpainting techniques is presented and summarized with comparisons including the merits and demerits of each technique which could help researchers in evaluating their proposed techniques against.

GuidedStyle: Attribute knowledge guided style manipulation for semantic face editing

Although significant progress has been made in synthesizing high-quality and visually realistic face images by unconditional Generative Adversarial Networks (GANs), there is still a lack of control over the generation process in order to achieve semantic face editing. In this paper, we propose a novel learning framework, called GuidedStyle, to achieve semantic face editing on pretrained StyleGAN by guiding the image generation process with a knowledge network. Furthermore, we allow an attention mechanism in StyleGAN generator to adaptively select a single layer for style manipulation. As a result, our method is able to perform disentangled and controllable edits along various attributes, including smiling, eyeglasses, gender, mustache, hair color and attractive. Both qualitative and quantitative results demonstrate the superiority of our method over other competing methods for semantic face editing. Moreover, we show that our model can be also applied to different types of real and artistic face editing, demonstrating strong generalization ability.

DeepFaceEditing

Recent facial image synthesis methods have been mainly based on conditional generative models. Sketch-based conditions can effectively describe the geometry of faces, including the contours of facial components, hair structures, as well as salient edges (e.g., wrinkles) on face surfaces but lack effective control of appearance, which is influenced by color, material, lighting condition, etc. To have more control of generated results, one possible approach is to apply existing disentangling works to disentangle face images into geometry and appearance representations. However, existing disentangling methods are not optimized for human face editing, and cannot achieve fine control of facial details such as wrinkles. To address this issue, we propose DeepFaceEditing, a structured disentanglement framework specifically designed for face images to support face generation and editing with disentangled control of geometry and appearance. We adopt a local-to-global approach to incorporate the face domain knowledge: local component images are decomposed into geometry and appearance representations, which are fused consistently using a global fusion module to improve generation quality. We exploit sketches to assist in extracting a better geometry representation, which also supports intuitive geometry editing via sketching. The resulting method can either extract the geometry and appearance representations from face images, or directly extract the geometry representation from face sketches. Such representations allow users to easily edit and synthesize face images, with decoupled control of their geometry and appearance. Both qualitative and quantitative evaluations show the superior detail and appearance control abilities of our method compared to state-of-the-art methods.

UniFaceGAN: A Unified Framework for Temporally Consistent Facial Video Editing.

Recent research has witnessed advances in facial image editing tasks including face swapping and face reenactment. However, these methods are confined to dealing with one specific task at a time. In addition, for video facial editing, previous methods either simply apply transformations frame by frame or utilize multiple frames in a concatenated or iterative fashion, which leads to noticeable visual flickers. In this paper, we propose a unified temporally consistent facial video editing framework termed UniFaceGAN. Based on a 3D reconstruction model and a simple yet efficient dynamic training sample selection mechanism, our framework is designed to handle face swapping and face reenactment simultaneously. To enforce the temporal consistency, a novel 3D temporal loss constraint is introduced based on the barycentric coordinate interpolation. Besides, we propose a region-aware conditional normalization layer to replace the traditional AdaIN or SPADE to synthesize more context-harmonious results. Compared with the state-of-the-art facial image editing methods, our framework generates video portraits that are more photo-realistic and temporally smooth.

F³A-GAN: Facial Flow for Face Animation With Generative Adversarial Networks.

Formulated as a conditional generation problem, face animation aims at synthesizing continuous face images from a single source image driven by a set of conditional face motion. Previous works mainly model the face motion as conditions with 1D or 2D representation (e.g., action units, emotion codes, landmark), which often leads to low-quality results in some complicated scenarios such as continuous generation and large-pose transformation. To tackle this problem, the conditions are supposed to meet two requirements, i.e., motion information preserving and geometric continuity. To this end, we propose a novel representation based on a 3D geometric flow, termed facial flow, to represent the natural motion of the human face at any pose. Compared with other previous conditions, the proposed facial flow well controls the continuous changes to the face. After that, in order to utilize the facial flow for face editing, we build a synthesis framework generating continuous images with conditional facial flows. To fully take advantage of the motion information of facial flows, a hierarchical conditional framework is designed to combine the extracted multi-scale appearance features from images and motion features from flows in a hierarchical manner. The framework then decodes multiple fused features back to images progressively. Experimental results demonstrate the effectiveness of our method compared to other state-of-the-art methods.

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs.

Although generative adversarial networks (GANs) have made significant progress in face synthesis, there lacks enough understanding of what GANs have learned in the latent representation to map a random code to a photo-realistic image. In this work, we propose a framework called InterFaceGAN to interpret the disentangled face representation learned by the state-of-the-art GAN models and study the properties of the facial semantics encoded in the latent space. We first find that GANs learn various semantics in some linear subspaces of the latent space. After identifying these subspaces, we can realistically manipulate the corresponding facial attributes without retraining the model. We then conduct a detailed study on the correlation between different semantics and manage to better disentangle them via subspace projection, resulting in more precise control of the attribute manipulation. Besides manipulating the gender, age, expression, and presence of eyeglasses, we can even alter the face pose and fix the artifacts accidentally made by GANs. Furthermore, we perform an in-depth face identity analysis and a layer-wise analysis to evaluate the editing results quantitatively. Finally, we apply our approach to real face editing by employing GAN inversion approaches and explicitly training feed-forward models based on the synthetic data established by InterFaceGAN. Extensive experimental results suggest that learning to synthesize faces spontaneously brings a disentangled and controllable face representation.

Controllable Multi-Attribute Editing of High-Resolution Face Images

In recent years, significant progress has been achieved in face image editing due to the success of Generative Adversarial Network (GAN). However, state-of-the-art face editing methods mainly suffer from the following two limitations: 1) they are only applicable to face images with relative low-resolutions and 2) multi-attribute face editing may generate uncontrollable changes in non-target face attribute categories. To solve these problems, we propose a novel High-Quality Generative Adversarial Network (HQ-GAN) for controllable editing of multiple face attributes in high-resolution images. HQ-GAN has two novel ideas to break the limitations of resolution and controllability correspondingly: 1) fine-grained textures and realistic details of high-resolution face images are better preserved with the aid of textural features extracted by the wavelet transform module and 2) desired multi-attribute targets of face editing are emphasized using a weighted binary cross-entropy (BCE) loss so that the influence on non-target attributes is greatly reduced. To the best of our knowledge, HQ-GAN is the first attempt to achieve continuous editing of multiple face attributes on high-resolution images of the CelebA-HQ using only 28 000 training samples. Extensive qualitative results demonstrate the superiority of the proposed method in rendering realistic high-resolution face images with accurate attribute modification, and comprehensive quantitative results show that the proposed method significantly outperforms state-of-the-art face editing methods.

SliderGAN: Synthesizing Expressive Face Images by Sliding 3D Blendshape Parameters

Image-to-image (i2i) translation is the dense regression problem of learning how to transform an input image into an output using aligned image pairs. Remarkable progress has been made in i2i translation with the advent of deep convolutional neural networks and particular using the learning paradigm of generative adversarial networks (GANs). In the absence of paired images, i2i translation is tackled with one or multiple domain transformations (i.e., CycleGAN, StarGAN etc.). In this paper, we study the problem of image-to-image translation, under a set of continuous parameters that correspond to a model describing a physical process. In particular, we propose the SliderGAN which transforms an input face image into a new one according to the continuous values of a statistical blendshape model of facial motion. We show that it is possible to edit a facial image according to expression and speech blendshapes, using sliders that control the continuous values of the blendshape model. This provides much more flexibility in various tasks, including but not limited to face editing, expression transfer and face neutralisation, comparing to models based on discrete expressions or action units.