High-resolution Face Image Research Articles

An improved face attributes editing method based on DDIM

The main advantage of DDIM is that it guarantees the quality of the generated images while increasing the efficiency of the generation by modifying the sampling strategy in the diffusion process. DiffusionRig, which addresses the problem of maintaining identity consistency by learning the person-specific facial prior in a tiny personalized dataset, is a successful representation of the DDIM strategy. Based on DiffusionRig, in this article, we propose an improved face attributes editing method based on DDIM to improve naturalness and accuracy of editing results in complex face attribute editing tasks and the generalization ability. Our method combines DDIM and DECA together and use two-stage training strategy. To reduce DiffusionRig’s limitations in handling face attribute editing tasks that require nonlinear understanding and fine-tuning, our method also introduces a channel attention mechanism and a depth-separable convolution technique in the training model. In the first stage we trained our model on the open FFHQ dataset, which consists of 30,000 high-resolution face image. In the second stage, the model was refined by using a tiny personalized dataset. Face attribute editing experiments,comparative experiment with DiffusionRig, and a series of ablation experiments have been performed. Then, the validity of our improved method is verified by qualitative and quantitative analysis.

Learning face super-resolution through identity features and distilling facial prior knowledge

Deep learning techniques in electronic surveillance have shown impressive performance for super-resolution (SR) of captured low-quality face images. Most of these techniques adopt facial priors to improve the feature details in the resultant super-resolved images. However, the estimation of facial priors from the captured low-quality images is often inaccurate in real-life situations because of their tiny, noisy, and blurry nature. Thus, the fusion of such priors badly affects the performance of these models. Therefore, this work presents a teacher–student-based face SR framework that efficiently preserves the personal facial structure information in the super-resolved faces. In the proposed framework, the teacher network exploits the facial heatmap-based ground-truth-prior to learn the facial structure that is utilized by the student network. The student network is trained with the identity feature loss for maintaining the identity and facial structure information in reconstructed high-resolution (HR) face images. The performance of the proposed framework is evaluated by conducting the experimental study on standard datasets namely CelebA-HQ and LFW face. The experimental results reveal that the proposed technique conquers the existing methods for the face SR task.

Personalized Super Resolution with Face Prior

Face Super-Resolution (FSR) is a critical technology in computer vision that aims to reconstruct high-resolution facial images from low-resolution inputs. Despite recent advancements, current FSR methods struggle to accurately reconstruct personalized and detailed features. This paper proposes a novel FSR approach that addresses these challenges through a personalized feature extraction and fusion framework. Our method integrates a U-Net based downsampling mechanism to extract individual- specific features from high-resolution reference images, which are then fused with a pre-trained Generative Adversarial Network (GAN) for enhanced reconstruction. We introduce a comprehensive loss function that combines reconstruction, adversarial, facial component, and identity preservation losses to guide the learning process. Extensive experiments on the augmented FFHQ dataset demonstrate that our approach significantly improves the reconstruction of rich facial features, particularly for older individuals, outperforming existing state-of-the-art methods in both quantitative metrics and qualitative visual assessments.

2D/3D Expression Generation Using Advanced Learning Techniques and the Emotion Wheel

Facial expression analysis is a critical component in numerous applications, ranging from human- computer interaction to digital character animation. Despite the availability of extensive datasets, most focus predominantly on basic emotions, limiting the expressiveness and applicability of models trained on them. This article introduces a novel approach to generating compound emotional expressions by leveraging the Emotion Wheel, a principle that captures the complex interrelations between basic emotions. Our method integrates the EMOCA (Emotion Driven Monocular Face Capture and Animation) [1] framework, which enhances 3D facial reconstruction by incorporating emotion recognition, stacking models [2, 54] with a sophisticated expression blending algorithm to synthesize nuanced 2D and 3D facial animations. Utilizing the VKIST dataset, which includes high-resolution facial images of Vietnamese individuals, we build a comprehensive database of emotion parameters. Through principal component analysis (PCA) and correlation-driven blending, our approach not only enhances the realism of generated facial expressions but also preserves the subtle nuances that are characteristic of human emotions. Experimental results demonstrate that our method consistently outperforms traditional linear interpolation techniques, producing more distinct and recognizable blended expressions. A user study further validates the naturalness and quality of the generated expressions, with average ratings indicating a strong preference for the proposed method over existing approaches. These findings suggest significant potential for improving emotional expressiveness in both static and dynamic digital character representations.

Cross-Modal Face Super-Resolution Based on Quasi-Siamese Domain Transfer Fusion Network

In this paper, we propose a Cross-Modal Face Super-Resolution (CMFSR) method to construct high-resolution (HR) facial images from low-resolution (LR) cross-modal facial images captured respectively by disjoint visible light (VIS) and near-infrared (NIR) cameras. Due to the coupling of modality transformation and information fusion, CMFSR is more difficult to obtain HR reconstructed results compared with traditional super-resolution. To solve this problem, a Quasi-Siamese Domain Transfer Fusion Network (QSDTFN) for CMFSR is proposed in this paper, whose two branches transfer two LR face modality to HR face modality by domain transfer respectively. Different from two completely independent branches in the traditional pseudo-siamese network, only the HR-to-LR face transfer processes of the two branches in our quasi-siamese network are independent, while the LR-to-HR face transfer processes are coupled. This coupled module called the Adaptive Weighted Domain Transfer Fusion Module (AWDTFM) disentangles the modality and identity information in the two LR faces, thus achieving modality transformation and identity information fusion simultaneously. In order to strengthen the optimization on the process of CMFSR, this method further introduces the backward QSDTFN to form a higher-level bidirectional structure with the forward QSDTFN, and specifically designs two types of losses: intra-network loss and inter-network loss, to constrain the modality and identity consistencies within one QSDTFN and between two QSDTFNs respectively. The experimental results on the challenging LR cross-modal face datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods.

High Fidelity Makeup via 2D and 3D Identity Preservation Net

In this article, we address the challenging makeup transfer task, aiming to transfer makeup from a reference image to a source image while preserving facial geometry and background consistency. Existing deep neural network-based methods have shown promising results in aligning facial parts and transferring makeup textures. However, they often neglect the facial geometry of the source image, leading to two adverse effects: (1) alterations in geometrically relevant facial features, causing face flattening and loss of personality, and (2) difficulties in maintaining background consistency, as networks cannot clearly determine the face-background boundary. To jointly tackle these issues, we propose the High Fidelity Makeup via two-dimensional (2D) and 3D Identity Preservation Network (IP23-Net), to the best of our knowledge, a novel framework that leverages facial geometry information to generate more realistic results. Our method comprises a 3D Shape Identity Encoder, which extracts identity and 3D shape features. We incorporate a 3D face reconstruction model to ensure the three-dimensional effect of face makeup, thereby preserving the characters’ depth and natural appearance. To preserve background consistency, our Background Correction Decoder automatically predicts an adaptive mask for the source image, distinguishing the foreground and background. In addition to popular benchmarks, we introduce a new large-scale High Resolution Synthetic Makeup Dataset containing 335,230 diverse high-resolution face images to evaluate our method’s generalization ability. Experiments demonstrate that IP23-Net achieves high-fidelity makeup transfer while effectively preserving background consistency. The code will be made publicly available.

Locality-constraint Representation with Minkowski distance metric for an effective Face Hallucination

Face hallucination (FaceH), the domain specific super resolution technique has significant importance in the field of image processing and computer vision. These methods provide visually convincing high-resolution face images using the low-resolution input images. To do this, all the state-of-the-art methods make use of distance measure algorithms. The performance of distance measure is one of the core components that impact the performance of FaceH. However, there are many k-NN distance measure algorithms, to the best of our knowledge, the most commonly used metric is the Euclidean distance, though it is the best, a little deviation may result in large elevation on estimating the reconstructed weight of the pixel. To overcome this, it is necessary to find an alternative method and this motivated to develop Minkowski Locality-constrained Representation (MinLcR) FaceH that makes use of Minkowski distance and after which it is integrated with Locality-constrained Representation (LcR) that incorporates the contextual information of the neighboring pixel for better hallucination. The main idea of MinLcR is to make a difference in the distance metric and compare it with methods that make use of Euclidean distance. Experimental analysis carried out using FEI database shows that the proposed method has an effective improvement over other methods.

Rethinking Prior-Guided Face Super-Resolution: A New Paradigm With Facial Component Prior.

Recently, facial priors (e.g., facial parsing maps and facial landmarks) have been widely employed in prior-guided face super-resolution (FSR) because it provides the location of facial components and facial structure information, and helps predict the missing high-frequency (HF) information. However, most existing approaches suffer from two shortcomings: 1) the extracted facial priors are inaccurate since they are extracted from low-resolution (LR) or low-quality super-resolved (SR) face images and 2) they only consider embedding facial priors into the reconstruction process from LR to SR face images, thus failing to explore facial priors to generate LR face image. In this article, we propose a novel pre-prior guided approach that extracts facial prior information from original high-resolution (HR) face images and embeds them into LR ones to obtain HF information-rich LR face images, thereby improving the performance of face reconstruction. Specifically, a novel component hybrid method is proposed, which fuses HR facial components and LR facial background to generate new LR face images (namely, LRmix) via facial parsing maps extracted from HR face images. Furthermore, we design a component hybrid network (CHNet) that learns the LR to LRmix mapping function to ensure that the LRmix can be obtained from LR face images in testing and real-world datasets. Experimental results show that our proposed scheme significantly improves the reconstruction performance for FSR.

Toward High-Resolution Face Image Generation From Coded Aperture Camera

Toward High-Resolution Face Image Generation From Coded Aperture Camera

Coupled discriminative manifold alignment for low-resolution face recognition

In practical applications, due to a long distance between the monitored population and monitoring equipment, the face images or human pose captured by the cameras often incur low-resolution (LR), small size, and poor quality, which leads to extreme difficulty in directly matching an LR face with the high-resolution (HR) ones in the gallery. In this paper, we propose a novel coupled discriminative manifold alignment (CDMA) method for LR face recognition. Specifically, in the training stage the principal component analysis (PCA) is first used to reduce the dimensional gap between LR and HR facial features. Next the LR face images and the corresponding HR face images are converted into a common shared feature subspace by learning two linear mappings in a supervised manner, where the neighborhood samples within the same class and from different classes are jointly exploited to align the manifold structures of LR and HR faces. In the test stage, for a given LR face in the probe set, two learned coupled mappings (CMs) are applied to match the HR images in the gallery set through the correlative metric. Thorough experimental results on three representative face databases verify the effectiveness of the proposed method in comparing with other state-of-the-art competitors. In particular, the proposed method is capable of yielding more competitive recognition performance than other predecessors when lower dimensional feature subspaces are applied to match the expected HR faces.

Generation of High-Resolution Facial Expression Images Using a Super-Resolution Technique and Self-Supervised Guidance

The recent spread of smartphones and social networking services has increased the means of seeing images of human faces. Particularly, in the face image field, the generation of face images using facial expression transformation has already been realized using deep learning–based approaches. However, in the existing deep learning–based models, only low-resolution images can be generated due to limited computational resources. Consequently, the generated images are blurry or aliasing. To address this problem, we proposed a two-step method to enhance the resolution of the generated facial images by combining a super-resolution network following the generative model, which can be considered a serial model, in our previous work. We further proposed a parallel model that trains a generative adversarial network and a superresolution network through multitask learning. In this paper, we propose a new model that integrates self-supervised guidance encoders into the parallel model to further improve the accuracy of the generated results. Using the peak signalto- noise ratio as an evaluation index, image quality was improved by 0.25 dB for the male test data and 0.28 dB for the female test data compared with our previous multitaskbased parallel model.

Attention guided domain alignment for conditional face image generation

Despite the great success of Generative Adversarial Network (GAN) in face image generation, it is still a challenge to generate a faithful yet high-fidelity face image given an exemplar image and a conditional input from a distinct domain (e.g., a semantic segmentation mask or an edge map). Existing methods learn image-level features to align distinct domains in an intermediate domain, which ignore the spatial relationship of the facial semantic parts and therefore lead to semantic mismatching problems. In addition, it is computationally expensive to establish dense correspondences in the intermediate domain, especially for high-resolution face images. To address these problems, we propose a novel attention guided domain alignment method for conditional face image generation, which aligns two domains directly under the guidance of the local attention learned from semantically similar face parts. In particular, we assign a dedicated index for each feature block and adopt a top-k ranking operation to correspond block-wise features in two distinct domains, which exploits the spatial relationship of facial parts and preserves the texture structure during alignment. The local attention is then learned from the retrieved blocks, which reduces computation complexity substantially and therefore enables to build alignment in a high resolution. The aligned features are finally fused with adaptive weights learned from their long-range correlation coefficients, which capture the semantic coherence of the style features between the two domains. Extensive experimental results on the CelebAMask-HQ dataset demonstrate that the proposed method is superior to the state-of-the-art methods.

Demographic Bias in Low-Resolution Deep Face Recognition in the Wild

Face biometrics play a primary role in smart cities, from consumer- to organizational-level applications. This class of technologies has been recently shown to emphasize performance disparities across gender and ethnic groups. Prior work on demographic bias in deep face recognition has tended to focus exclusively on high-resolution images, leaving out low-resolution images captured, for example, by a certain surveillance camera at a distance. Notable reasons for this focus include the lack of low-resolution face image data sets that report user's demographic attributes. Therefore, demographic bias in low-resolution deep face recognition in the wild still remains under-explored. In this paper, we propose a framework for exploring demographic disparities in low-resolution face recognition at a distance. To this end, we devised a deep generative approach to turn high-resolution face images to their low-resolution counterpart. We then trained state-of-the-art face recognition models on different combinations of high- and low-resolution images. Finally, we assessed the model demographic disparities on artificially degraded images from five data sets. Our results show that face images artificially degraded through our generative approach are more realistic than those obtained with other statistical changes. Key disparities across gender and ethnic groups exist and urge timely interventions. Source code: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://cutt.ly/FChLfOC</uri>

Towards Spatially Disentangled Manipulation of Face Images With Pre-Trained StyleGANs

Generative Adversarial Networks with style-based generators could successfully synthesize realistic images from input latent code. Moreover, recent studies have revealed that interpretable translations of generated images could be obtained by linearly traversing in the latent space. However, in most existing latent spaces, linear interpolation often leads to ‘spatially entangled modification’ in the manipulation result, which is undesirable in many real-world applications where local editing is required. To solve this problem, we propose to manipulate the latent code in the ‘style space’ and analyze its advantage in achieving spatial disentanglement. Furthermore, we point out the weakness of simply interpolating in the style space and propose ‘Style Intervention’, a lightweight optimization-based algorithm, to further improve the visual fidelity of manipulation results. The performance of our method is verified with the task of attribute editing on high-resolution face images. Both qualitative and quantitative results demonstrate the advantage of image translation in the style space and the effectiveness of our method on both real and synthetic images.

Joint face completion and super-resolution using multi-scale feature relation learning

Previous research on face restoration often focused on repairing specific types of low-quality facial images such as low-resolution (LR) or occluded facial images. However, in the real world, both the above-mentioned forms of image degradation often coexist. Therefore, it is important to design a model that can repair images that are LR and occluded simultaneously. This paper proposes a multi-scale feature graph generative adversarial network (MFG-GAN) to carry out face restoration in contexts in which both LR and occluded degradation modes coexist, and also to repair images with a single type of degradation. Based on the GAN, the MFG-GAN integrates the graph convolution and feature pyramid networks to restore occluded low-resolution face images to non-occluded high-resolution face images. The MFG-GAN uses a set of customized losses to ensure that high-quality images are generated. In addition, we designed the network in an end-to-end format. We conduct experiments on general face image restoration and facial expression restoration. Experimental results on the public-domain databases show that the proposed approach outperforms state-of-the-art methods in performing face super resolution (up to 4× or 8×) and face completion simultaneously and can recover better facial expression details.

A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors

Face super-resolution (FSR) aims to reconstruct high-resolution face images from low-resolution (LR) ones. Despite the progress made by deep convolutional neural networks (DCNNs) on FSR, convolutions struggle to relate spatially distant concepts and what is more, all image pixels and prior information (e.g., landmarks and facial component heatmaps) are treated equally regardless of importance, causing inaccuracy and decreasing the quality of face image recovery. To address these issues, in this paper we propose a composite network model for FSR with multi-order head attention facial priors. The proposed model contains a face hallucination transformer (FHT)-based network and a multi-order head attention (MOHA)-based DCNN. The FHT-based network can capture long-range dependencies and gradually increase resolution to achieve efficient and effective inference, while the MOHA-based DCNN exploits detailed and two-dimensional information of LR face images. Moreover, the novel generic submodule of the MOHA-based DCNN, namely Multi-Order Head Attention Network, can accurately model the relationship of facial components between spatial and channel dimensions. The proposed composite network model seamlessly integrates the advantages of DCNNs and transformers to super-resolve LR face images. When compared with state-of-the-art FSR methods on public benchmark datasets, the proposed model shows competitive recovery performance.

Deep generative image priors for semantic face manipulation

Previous works on generative adversarial networks (GANs) mainly focus on how to synthesize high-fidelity images. In this paper, we present a framework to leverage the knowledge learned by GANs for semantic face manipulation. In particular, we propose to control the semantics of synthesized faces by adapting the latent codes with an attribute prediction model. Moreover, in order to achieve a more accurate estimation of different facial attributes, we propose to pretrain the attribute prediction model by inverting the synthesized face images back to the GAN latent space. As a result, our method explicitly considers the semantics encoded in the latent space of a pretrained GAN and is able to faithfully edit various attributes like eyeglasses, smiling, bald, age, mustache and gender for high-resolution face images. Extensive experiments show that our method has superior performance compared to state of the art for both face attribute prediction and semantic face manipulation.

Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.

Compressing Face Images Using Genetic and Gray Wolf Meta-heuristic Algorithms Based on Variable Bit Allocation

In image processing, compression plays an important role in monitoring, controlling, and securing the process. The spatial resolution is one of the most effective factors in improving the quality of an image; but, it increases the amount of storage memory required. Based on meta-heuristic algorithms, this article presents a compression model for face images with block division and variable bit allocation. Wavelet transform is used to reduce the dimensions of high spatial resolution face images. In order to identify important and similar areas of identical macroblocks, genetic algorithms and gray wolves are used. A bit rate allocation is calculated for each block to achieve the best recognition accuracy, average PSNR, and SSIM. The CIE and FEI databases have been used as case studies. The proposed method has been tested and compared with the accuracy of image recognition under uncompressed conditions and using the common SPIHT and JPEG coding methods. Recognition accuracy increased from 0.18% for 16×16 blocks to 1.97% for 32×32 blocks. Additionally, the gray wolf algorithm is much faster than the genetic algorithm in reaching the optimal answer. Depending on the application type of the problem, the genetic algorithm or the gray wolf may be preferred to achieve the maximum average PSNR or SSIM. At the bit rate of 0.9, the maximum average PSNR for the gray wolf algorithm is 34.92 and the maximum average SSIM for the genetic algorithm is 0.936. Simulation results indicate that the mentioned algorithms increase PSNR and SSIM by stabilizing or increasing recognition accuracy.

Deep HyFeat Based Attention in Attention Model for Face Super-Resolution

Face super-resolution is the task of generating high-resolution (HR) face images from the low-resolution (LR) inputs. Recently, deep learning-based methods have shown remarkable progress in the super-resolution (SR) field. Most of the methods perform additional tasks such as face parsing, landmark, and attention to generate the HR images. However, parsing maps and landmark-guided models require the supplementary labeled dataset, which is difficult to obtain in real life. The attention mechanism does not require the dataset’s extra labeling and is also beneficial for face SR. These methods focus on a few critical features and ignore the remaining ones, which sometime causes to ignore the valuable features. Therefore, this paper proposes a novel deep HyFeat-based Attention in Attention model for face SR. Moreover, the proposed model uses the coarse SR network and deep convolutional neural network (CNN) to generate the HR image. A coarse SR network is applied to upsample the LR image and generate the coarse super-resolved image, which is further sent to the deep CNN model. The proposed work incorporates the Hybrid Feature Attention in Attention unit (HyFA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> U), which consists of Hybrid Feature block (HyFeat) and Attention in Attention block (A <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> B) in the deep CNN model to improve the visual quality of the output face images. HyFeat block assists the model in extracting the coarse features and learning the enriched contextual information to enhance the details of coarse features. Attention in Attention block preserves the attentive and non-attentive beneficial features while suppressing the unwanted features. The attention branch focuses on specific facial features and ignores the rest of the features. The non-attention branch aims to learn the informative features that the attention branch ignores. The proposed model repeats the HyFA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> U units to focus on different facial components and enhance the features to improve the quality of resultant faces. Experimental outcomes exhibit that the proposed model gains state-of-the-art performance on the standard datasets, namely CelebAHQ, Helen, FFHQ, and LFW face. The proposed method achieves an improvement of more than 0.35 dB in PSNR and 0.012 in SSIM on different datasets over the best models available in the literature.