Generate Face Images Research Articles

Prompt-Based Modality Bridging for Unified Text-to-Face Generation and Manipulation

Text-driven face image generation and manipulation are significant tasks. However, such tasks are quite challenging due to the gap between text and image modalities. It is difficult to utilize current methods to deal with both of the two problems because these methods are usually designed for one certain task, limiting their application in real scenarios. To address the two problems in one framework, we propose a U nified P rompt-based C ross- M odal Frame work (UPCM-Frame) to bridge the gap between the text modality and image modality with CLIP and StyleGAN, which are two large-scale pre-trained models. The proposed framework is combined with two main modules: a Text Embedding-to-Image Embedding projection module based on a special prompt embedding pair, and a projection module mapping Image Embeddings to semantically aligned StyleGAN Embeddings which can be used in both image generation and manipulation. The proposed framework is able to handle complicated descriptions and generate impressive results with high quality due to the utilization of large-scale pre-trained models. In order to demonstrate the effectiveness of the proposed method in the two tasks, we conduct experiments to evaluate the results of our method both quantitatively and qualitatively.

Refining CycleGAN with attention mechanisms and age-Aware training for realistic Deepfakes

In the evolving landscape of deep learning technologies, the emergence of Deepfakes and synthetic media is becoming increasingly prominent within digital media production. This research addresses the limitations inherent in existing face image generation algorithms based on Generative Adversarial Networks (GAN), particularly the challenges of domain irrelevancy and inadequate facial detail representation. The study introduces an enhanced face image generation algorithm, aiming to refine the CycleGAN framework. The enhancement involves a two-fold strategy: firstly, the generator's architecture is refined through the integration of an attention mechanism and adaptive residual blocks, enabling the extraction of more nuanced facial features. Secondly, the discriminator's accuracy in distinguishing real from synthetic images is improved by incorporating a relative loss concept into the loss function. Additionally, this study presents a novel model training approach that incorporates age constraints, thereby mitigating the effects of age variations on the synthesized images. The effectiveness of the proposed algorithm is empirically validated through comparative analysis with existing methodologies, utilizing the CelebA dataset. The results demonstrate that the proposed algorithm significantly enhances the realism of generated face images, outperforming current methods in terms of Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM), while also achieving notable improvements in subjective visual quality. The implementation of this advanced method is anticipated to substantially elevate the efficiency and quality of digital media production, contributing positively to the broader field of digital media creation.

An Improved Detail-Enhancement CycleGAN Using AdaLIN for Facial Style Transfer

The rise of comics and games has led to increased artistic processing of portrait photos. With growing commercial demand and advancements in deep learning, neural networks for rapid facial style transfer have become a key research area in computer vision. This involves converting face photos into different styles while preserving content. Face images are more complex than regular images, requiring extensive modification. However, current methods often face issues such as unnatural color transitions, loss of detail in highlighted areas, and noticeable artifacts along edges, resulting in low-quality stylized images. In this study, an enhanced generative adversarial network (GAN) is proposed, which is based on Adaptive Layer Instance Normalization (AdaLIN) + Laplacian. This approach incorporates the AdaLIN normalization method, allowing for the dynamic adjustment of Instance Normalization (IN) and Layer Normalization (LN) parameters’ weights during training. By combining the strengths of both normalization techniques, the model selectively preserves and alters content information to some extent, aiming to strike a balance between style and content. This helps address problems such as unnatural color transitions and loss of details in highlights that lead to color inconsistencies. Furthermore, the introduction of a Laplacian regularization term aids in denoising the image, preventing noise features from interfering with the color transfer process. This regularization also helps reduce color artifacts along the face’s edges caused by noise while maintaining the image’s contour information. These enhancements significantly enhance the quality of the generated face images. To compare our method with traditional CycleGAN and recent algorithms such as XGAN and CariGAN, both subjective and objective evaluations were conducted. Subjectively, our method demonstrates more natural color transitions and superior artifact elimination, achieving higher scores in Mean Opinion Score (MOS) evaluations. Objectively, experiments using our method yielded better scores across three metrics: FID, SSIM, and MS-SSIM. The effectiveness of the proposed methods is validated through both objective and subjective evaluations.

PMSGAN: Parallel Multistage GANs for Face Image Translation.

In this article, we address the face image translation task, which aims to translate a face image of a source domain to a target domain. Although significant progress has been made by recent studies, face image translation is still a challenging task because it has more strict requirements for texture details: even a few artifacts will greatly affect the impression of generated face images. Targeting to synthesize high-quality face images with admirable visual appearance, we revisit the coarse-to-fine strategy and propose a novel p arallel m ultistage architecture on the basis of g enerative a dversarial n etworks (PMSGAN). More specifically, PMSGAN progressively learns the translation function by disintegrating the general synthesis process into multiple parallel stages that take images with gradually decreasing spatial resolution as inputs. To prompt the information exchange between various stages, a cross-stage atrous spatial pyramid (CSASP) structure is specially designed to receive and fuse the contextual information from other stages. At the end of the parallel model, we introduce a novel attention-based module that leverages multistage decoded outputs as in situ supervised attention to refine the final activations and yield the target image. Extensive experiments on several face image translation benchmarks show that PMSGAN performs considerably better than state-of-the-art approaches.

GAN-Generated Face Detection Based on Multiple Attention Mechanism and Relational Embedding

The rapid development of the Generative Adversarial Network (GAN) makes generated face images more and more visually indistinguishable, and the detection performance of previous methods will degrade seriously when the testing samples are out-of-sample datasets or have been post-processed. To address the above problems, we propose a new relational embedding network based on “what to observe” and “where to attend” from a relational perspective for the task of generated face detection. In addition, we designed two attention modules to effectively utilize global and local features. Specifically, the dual-self attention module selectively enhances the representation of local features through both image space and channel dimensions. The cross-correlation attention module computes similarity between images to capture the global information of the output in the image. We conducted extensive experiments to validate our method, and the proposed algorithm can effectively extract the correlations between features and achieve satisfactory generalization and robustness in generating face detection. In addition, we also explored the design of the model structure and the inspection performance on more categories of generated images (not limited to faces). The results show that RENet also has good detection performance on datasets other than faces.

Personalized hairstyle and hair color editing based on multi-feature fusion

In the metaverse era, virtual design of hairstyle becomes very popular for personalized aesthetics. As hair design tasks can be decomposed into hair attribute editing and generation, the development of generative adversarial networks (GANs) has significantly prompted its development. The majority of the existing algorithms focus on transferring the overall hair region from one face to another, which ignore fine control over the color and geometric features. Furthermore, these algorithms may result in unnatural generation results. In this paper, we propose a hair modification framework that learns hairstyle information from a reference face mask and color information from a guidance face image. Firstly, the features of the input face image and reference images are extracted through a group of encoders, and then divided into feature vectors of coarse, medium, and fine levels. Secondly, multi-level feature vectors are fused in the latent space using attention-based modulation modules. Finally, the fused feature vector is passed through a StyleGAN generator to generate face images with specified hairstyle and hair color. Experimental results show that the proposed method can finely simulate the hairstyle transition between long and short hair under the constraint of the reference mask, and can produce realistic fusion effects in the hair-covered regions, such as ears, neck, and forehead. Various hair dyeing effects that adapt to personalized characteristics are demonstrated, as facial features including skin color and hair texture are preserved when transferring the hair color.

Generating Accurate Human Face Sketches from Text Descriptions

Drawing a face for a suspect just based on the descriptions of the eyewitnesses is a difficult task. There are some state-of-the-art methods in generating images from text, but there are only a few research in generating face images from text and close to none in generating sketches from text. As a result, there is no dataset available to tackle this task. In this paper, we generated a new text-to-sketch dataset for our novel task, and provide two attention based SOTA GAN end-to-end models, Attn_LSTM_256 and Attn_GRU_512, trained on the dataset resulting in Inception score of 1.868 and 1.902, and FID of 175.46 and 176.98. We further propose possible future improvements by applying different model architectures or preserving performance with simplified architectures for real-world applications.

Conditioning diffusion models via attributes and semantic masks for face generation

Deep generative models have shown impressive results in generating realistic images of faces. GANs managed to generate high-quality, high-fidelity images when conditioned on semantic masks, but they still lack the ability to diversify their output. Diffusion models partially solve this problem and are able to generate diverse samples given the same condition. This paper introduces a novel strategy for enhancing diffusion models through multi-conditioning, harnessing cross-attention mechanisms to utilize multiple feature sets, ultimately enabling the generation of high-quality and controllable images. The proposed method extends previous approaches by introducing conditioning on both attributes and semantic masks, ensuring finer control over the generated face images. In order to improve the training time and the generation quality, the impact of applying perceptual-focused loss weighting into the latent space instead of the pixel space is also investigated. The proposed solution has been evaluated on the CelebA-HQ dataset, and it can generate realistic and diverse samples while allowing for fine-grained control over multiple attributes and semantic regions. Experiments on the DeepFashion dataset have also been performed in order to analyze the capability of the proposed model to generalize to different domains. In addition, an ablation study has been conducted to evaluate the impact of different conditioning strategies on the quality and diversity of the generated images.

Color Face Image Generation with Improved Generative Adversarial Networks

This paper focuses on the development of an improved Generative Adversarial Network (GAN) specifically designed for generating color portraits from sketches. The construction of the system involves using a GPU (Graphics Processing Unit) computing host as the primary unit for model training. The tasks that require high-performance calculations are handed over to the GPU host, while the user host only needs to perform simple image processing and use the model trained by the GPU host to generate images. This arrangement reduces the computer specification requirements for the user. This paper will conduct a comparative analysis of various types of generative networks which will serve as a reference point for the development of the proposed Generative Adversarial Network. The application part of the paper focuses on the practical implementation and utilization of the developed Generative Adversarial Network for the generation of multi-skin tone portraits. By constructing a face dataset specifically designed to incorporate information about ethnicity and skin color, this approach can overcome a limitation associated with traditional generation networks, which typically generate only a single skin color.

Limitations of Face Image Generation

Text-to-image diffusion models have achieved widespread popularity due to their unprecedented image generation capability. In particular, their ability to synthesize and modify human faces has spurred research into using generated face images in both training data augmentation and model performance assessments. In this paper, we study the efficacy and shortcomings of generative models in the context of face generation. Utilizing a combination of qualitative and quantitative measures, including embedding-based metrics and user studies, we present a framework to audit the characteristics of generated faces conditioned on a set of social attributes. We applied our framework on faces generated through state-of-the-art text-to-image diffusion models. We identify several limitations of face image generation that include faithfulness to the text prompt, demographic disparities, and distributional shifts. Furthermore, we present an analytical model that provides insights into how training data selection contributes to the performance of generative models. Our survey data and analytics code can be found online at https://github.com/wi-pi/Limitations_of_Face_Generation

Adv-Diffusion: Imperceptible Adversarial Face Identity Attack via Latent Diffusion Model

Adversarial attacks involve adding perturbations to the source image to cause misclassification by the target model, which demonstrates the potential of attacking face recognition models. Existing adversarial face image generation methods still can’t achieve satisfactory performance because of low transferability and high detectability. In this paper, we propose a unified framework Adv-Diffusion that can generate imperceptible adversarial identity perturbations in the latent space but not the raw pixel space, which utilizes strong inpainting capabilities of the latent diffusion model to generate realistic adversarial images. Specifically, we propose the identity-sensitive conditioned diffusion generative model to generate semantic perturbations in the surroundings. The designed adaptive strength-based adversarial perturbation algorithm can ensure both attack transferability and stealthiness. Extensive qualitative and quantitative experiments on the public FFHQ and CelebA-HQ datasets prove the proposed method achieves superior performance compared with the state-of-the-art methods without an extra generative model training process. The source code is available at https://github.com/kopper-xdu/Adv-Diffusion.

Investigation the influence related to parameters configuration of Generative Adversarial Networks in face image generation

Due to the excellent performance of Generative Adversarial Networks (GAN) for age regression on face images, it is particularly important to explore the effect of different parameters on model training. In this study, the origin and development of Artificial Intelligence (AI) is first discussed, from which the concept and principles of GAN are derived. This is followed by a brief introduction of the UTKface dataset used in this research, and the Conditional Adversarial Autoencoder (CAAE) framework based on the GAN technique. The division of labor and roles of the encoder, generator, and the two discriminators in the model are described. The various learning rates as well as batch size combinations attempted in this study are then illustrated, and the training results of the model are shown in the form of graphs and plots of the loss value function. A situation where the model stops learning is highlighted in the results, which is similar to pattern descent in GAN, and is shown to be characterized by the inability of the discriminator to successfully recognize it. Ultimately, drawing from the acquired outcomes, it can be deduced that employing a larger batch size serves to enhance the pace of model training. It is advisable to concurrently elevate the learning rate by an equivalent factor when augmenting the batch size, thereby ensuring a consistent trajectory for model convergence.

A siamese-based verification system for open-set architecture attribution of synthetic images

Despite the wide variety of methods developed for synthetic image attribution, most of them can only attribute images generated by models or architectures included in the training set and do not work with unknown architectures, hindering their applicability in real-world scenarios. In this paper, we propose a verification framework that relies on a Siamese Network to address the problem of open-set attribution of synthetic images to the architecture that generated them. We consider two different settings. In the first setting, the system determines whether two images have been produced by the same generative architecture or not. In the second setting, the system verifies a claim about the architecture used to generate a synthetic image, utilizing one or multiple reference images generated by the claimed architecture. The main strength of the proposed system is its ability to operate in both closed and open-set scenarios so that the input images, either the query and reference images, can belong to the architectures considered during training or not. Experimental evaluations encompassing various generative architectures such as GANs, diffusion models, and transformers, focusing on synthetic face image generation, confirm the excellent performance of our method in both closed and open-set settings, as well as its strong generalization capabilities.

Diversified realistic face image generation GAN for human subjects in multimedia content creation

AbstractFace image generation plays an important role in generating innovative and unique multimedia content using the GAN model. With these qualities of the GAN model, they have numerous challenges in the human face image generation. The problems encountered in the generation of facial images are like blurriness in images, incomplete details in the generated facial images, high computational power requirements, and so forth. In this manuscript, we proposed a GAN model that utilizes the composite strength of VGG‐16 and ResNet‐50's models to overcome those difficulties. It uses VGG‐16 to build a discriminator model to discriminate between real and fake images. The generator model utilizes a combination of components from the ResNet‐50 and VGG‐16 models to enhance the image generation process at each iteration, resulting in the creation of realistic face images. The proposed DRFI GAN (Diversified and Realistic Face Image Generation GAN) model's generator achieves an impressive low FID score of 20.50, which is less than existing state‐of‐the‐art approaches. Furthermore, our findings indicate that the images generated by the DRFI GAN model exhibit 10%–15% greater efficiency and realism with reduced training time compared to existing state‐of‐the‐art methods with lower FID scores.

CGOF++: Controllable 3D Face Synthesis With Conditional Generative Occupancy Fields.

Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images. However, previous methods focus on controllable 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes. In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors. At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh, built on top of EG3D (Chan et al. 2022), a recent tri-plane-based generative model. To achieve accurate control over fine-grained 3D face shapes of the synthesized images, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis framework. Experiments validate the effectiveness of the proposed method, which is able to generate high-fidelity face images and shows more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.

An interactive augmented reality software for facial reconstructive surgeries

Background and ObjectiveSurgical trainees need a lot of training and practice before being able to operate independently. The current approach of surgical education mainly involves didactic teaching and psychomotor training through physical models or real tissue. Due to the unavailability of physical resources and lack of objective ways of evaluation, there is a demand for developing alternative training methods for surgeons. In this paper, we present an application that provides additional training opportunities to surgical trainees in the field of facial reconstructive surgeries. MethodsWe built a mobile augmented reality application that helps the user to visualize important concepts and experiment with different surgical plans for facial reconstructive surgeries. The application can overlay relaxed skin tension lines on a live video input or a patient's photo, which serve as bases for aligning a skin flap. A surgical trainee can interactively compare different skin flap design choices with estimated final scars on a photo of a patient. Data collection capability is also added to the application, and we performed a Monte Carlo experiment with simulated users (five classes of 100 users each) as an example of objectively measuring user performance. ResultsThe application can overlay relaxed skin tension lines on a patient's face in real time on a modern mobile device. Accurate overlays were achieved in over 91% as well as 84% and 88% out of 263 generated face images, depending on the method. Visual comparisons of the three overlay methods are presented on sample faces from different population groups. From the Monte Carlo experiment, we see that user actions in each class follow a normal distribution with a distinct set of parameters. ConclusionsThis application can serve as a basis for teaching surgical trainees the fundamentals of different facial reconstructive procedures, especially concepts related to relaxed skin tension lines and skin flaps. It can objectively evaluate the performance of surgical trainees in a course. This setup focuses on illustrating the relationship between the orientation of skin flaps and relaxed skin tension lines, which is a prerequisite of minimizing scarring in patients in addition to good surgical techniques.

Face image generation and feature visualization using deep convolutional generative adversarial networks

Generative Neural Networks (GAN) aims to generate realistic and recognizable images, including portraits, cartoons and other modalities. Image generation has broad application prospects and important research value in the fields of public security and digital entertainment, and has become one of the current research hotspots. This article will introduce and apply an important image generation model called GAN, which stands for Generative Adversarial Network. Unlike recent image processing models such as Variational Autoencoders (VAE), The discriminative network evaluates potential candidates while the GAN generates candidates. As a result, the discriminative network distinguishes created and real candidates, while the generative network learns to map from a latent space to an interest data distribution. In this article, the GAN model and some of its extensions will be thoroughly applied and implemented based on the dataset of CelebA, and details will be discussed through the images and graphs generated by the model. Specific training methods for various models and optimization algorithms can be produced by the GAN framework. The experiment’s findings in this article will show how the framework’s potential may be quantified and qualitatively assessed using the samples that were produced.

Toward High-Resolution Face Image Generation From Coded Aperture Camera

Toward High-Resolution Face Image Generation From Coded Aperture Camera

Digital Face Manipulation Creation and Detection: A Systematic Review

The introduction of publicly available large-scale datasets and advances in generative adversarial networks (GANs) have revolutionized the generation of hyper-realistic facial images, which are difficult to detect and can rapidly reach millions of people, with adverse impacts on the community. Research on manipulated facial image detection and generation remains scattered and in development. This survey aimed to address this gap by providing a comprehensive analysis of the methods used to produce manipulated face images, with a focus on deepfake technology and emerging techniques for detecting fake images. The review examined four key groups of manipulated face generation techniques: (1) attributes manipulation, (2) facial re-enactment, (3) face swapping, and (4) face synthesis. Through an in-depth investigation, this study sheds light on commonly used datasets, standard manipulated face generation/detection approaches, and benchmarking methods for each manipulation group. Particular emphasis is placed on the advancements and detection techniques related to deepfake technology. Furthermore, the paper explores the benefits of analyzing deepfake while also highlighting the potential threats posed by this technology. Existing challenges in the field are discussed, and several directions for future research are proposed to tackle these challenges effectively. By offering insights into the state of the art for manipulated face image detection and generation, this survey contributes to the advancement of understanding and combating the misuse of deepfake technology.

Attention guided domain alignment for conditional face image generation

Despite the great success of Generative Adversarial Network (GAN) in face image generation, it is still a challenge to generate a faithful yet high-fidelity face image given an exemplar image and a conditional input from a distinct domain (e.g., a semantic segmentation mask or an edge map). Existing methods learn image-level features to align distinct domains in an intermediate domain, which ignore the spatial relationship of the facial semantic parts and therefore lead to semantic mismatching problems. In addition, it is computationally expensive to establish dense correspondences in the intermediate domain, especially for high-resolution face images. To address these problems, we propose a novel attention guided domain alignment method for conditional face image generation, which aligns two domains directly under the guidance of the local attention learned from semantically similar face parts. In particular, we assign a dedicated index for each feature block and adopt a top-k ranking operation to correspond block-wise features in two distinct domains, which exploits the spatial relationship of facial parts and preserves the texture structure during alignment. The local attention is then learned from the retrieved blocks, which reduces computation complexity substantially and therefore enables to build alignment in a high resolution. The aligned features are finally fused with adaptive weights learned from their long-range correlation coefficients, which capture the semantic coherence of the style features between the two domains. Extensive experimental results on the CelebAMask-HQ dataset demonstrate that the proposed method is superior to the state-of-the-art methods.