Face Super-resolution Research Articles

A dual branch attention network based on practical degradation model for face super resolution.

In recent years, the field of face super-resolution (FSR) has advanced rapidly. However, complex degradation factors in real-world scenarios can severely deteriorate image quality, significantly affecting the reconstruction performance of FSR methods. Currently, there is a lack of research on degradation modeling for real-world facial images, which impacts the generalization ability of existing FSR methods. In this paper, a practical degradation model based on hybrid degradation processes is proposed to select multiple degradation processes including Gaussian noise, Rayleigh noise, Motion blur, Salt-and-Pepper noise, and Mean blur through a stochastic strategy to more realistically simulate the effect of image distortion in real scenarios. We also design a dual-branch attention network called DBANet for face super-resolution and conduct experiments on the SCUT_FBP, Helen and PFHQ datasets, achieving satisfactory results. Our proposed model is effective in handling image distortion under different degradation modalities, which improves the robustness of super-resolution reconstruction. This study introduces an innovative approach to the field of face image super-resolution, which has the potential for a wide range of practical applications. The code of DBANet will be available at https://github.com/bxzha/DBANet .

Scalable multimodal approach for face generation and super-resolution using a conditional diffusion model

Multimodal Conditioned face image generation and face super-resolution are significant areas of research. To achieve optimal results, this paper utilizes diffusion models as the primary engine for these tasks. This paper presents two main contributions: (1) “Speaking the Language of Faces” (SLF): a flexible, modular, fusion-less and architecturally simple multimodal system. (2) A Scalability scheme and a sensitivity analysis which can assist practitioners in system parameter estimation and feature selection. SLF consists of two main components: a feature vector generator (encoder), and an image generator (decoder) utilizing a conditional diffusion model. SLF can accept various inputs, including low-resolution images, speech signals, person attributes (age, gender, ethnicity), or any combination of these. Moreover, Scalability based on conditional scale values is utilized. The implementation of SLF has confirmed its versatility (e.g., speech to face image generation, conditioned face super-resolution). We trained multiple system versions to conduct a sensitivity analysis and to determine the influence of each individual feature on the output image. Consequently, speaker embeddings have proven to be sufficient audio features for our task. It was also found that the effects of audio signals are profound and are more pronounced than those of the low resolution images (8 × 8), whose effects are still significant. The effect of gender, ethnicity and age were found to be moderate. On another note, conditional scale values significantly impact the system’s behavior and performance.

Learning face super-resolution through identity features and distilling facial prior knowledge

Deep learning techniques in electronic surveillance have shown impressive performance for super-resolution (SR) of captured low-quality face images. Most of these techniques adopt facial priors to improve the feature details in the resultant super-resolved images. However, the estimation of facial priors from the captured low-quality images is often inaccurate in real-life situations because of their tiny, noisy, and blurry nature. Thus, the fusion of such priors badly affects the performance of these models. Therefore, this work presents a teacher–student-based face SR framework that efficiently preserves the personal facial structure information in the super-resolved faces. In the proposed framework, the teacher network exploits the facial heatmap-based ground-truth-prior to learn the facial structure that is utilized by the student network. The student network is trained with the identity feature loss for maintaining the identity and facial structure information in reconstructed high-resolution (HR) face images. The performance of the proposed framework is evaluated by conducting the experimental study on standard datasets namely CelebA-HQ and LFW face. The experimental results reveal that the proposed technique conquers the existing methods for the face SR task.

Personalized Super Resolution with Face Prior

Face Super-Resolution (FSR) is a critical technology in computer vision that aims to reconstruct high-resolution facial images from low-resolution inputs. Despite recent advancements, current FSR methods struggle to accurately reconstruct personalized and detailed features. This paper proposes a novel FSR approach that addresses these challenges through a personalized feature extraction and fusion framework. Our method integrates a U-Net based downsampling mechanism to extract individual- specific features from high-resolution reference images, which are then fused with a pre-trained Generative Adversarial Network (GAN) for enhanced reconstruction. We introduce a comprehensive loss function that combines reconstruction, adversarial, facial component, and identity preservation losses to guide the learning process. Extensive experiments on the augmented FFHQ dataset demonstrate that our approach significantly improves the reconstruction of rich facial features, particularly for older individuals, outperforming existing state-of-the-art methods in both quantitative metrics and qualitative visual assessments.

PLGNet: Prior-Guided Local and Global Interactive Hybrid Network for Face Super-Resolution

PLGNet: Prior-Guided Local and Global Interactive Hybrid Network for Face Super-Resolution

CCIR: high fidelity face super-resolution with controllable conditions in diffusion models

CCIR: high fidelity face super-resolution with controllable conditions in diffusion models

Cross-Modal Face Super-Resolution Based on Quasi-Siamese Domain Transfer Fusion Network

In this paper, we propose a Cross-Modal Face Super-Resolution (CMFSR) method to construct high-resolution (HR) facial images from low-resolution (LR) cross-modal facial images captured respectively by disjoint visible light (VIS) and near-infrared (NIR) cameras. Due to the coupling of modality transformation and information fusion, CMFSR is more difficult to obtain HR reconstructed results compared with traditional super-resolution. To solve this problem, a Quasi-Siamese Domain Transfer Fusion Network (QSDTFN) for CMFSR is proposed in this paper, whose two branches transfer two LR face modality to HR face modality by domain transfer respectively. Different from two completely independent branches in the traditional pseudo-siamese network, only the HR-to-LR face transfer processes of the two branches in our quasi-siamese network are independent, while the LR-to-HR face transfer processes are coupled. This coupled module called the Adaptive Weighted Domain Transfer Fusion Module (AWDTFM) disentangles the modality and identity information in the two LR faces, thus achieving modality transformation and identity information fusion simultaneously. In order to strengthen the optimization on the process of CMFSR, this method further introduces the backward QSDTFN to form a higher-level bidirectional structure with the forward QSDTFN, and specifically designs two types of losses: intra-network loss and inter-network loss, to constrain the modality and identity consistencies within one QSDTFN and between two QSDTFNs respectively. The experimental results on the challenging LR cross-modal face datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods.

Face super resolution with a high frequency highway

AbstractFace shape priors such as landmarks, heatmaps, and parsing maps are widely used to improve face super resolution (SR). It is observed that face priors provide locations of high‐frequency details in key facial areas such as the eyes and mouth. However, existing methods fail to effectively exploit the high‐frequency information by using the priors as either constraints or inputs. This paper proposes a novel high frequency highway () framework to better utilize prior information for face SR, which dynamically decomposes the final SR face into a coarse SR face and a high frequency (HF) face. The coarse SR face is reconstructed from a low‐resolution face via a texture branch, using only pixel‐wise reconstruction loss. Meanwhile, the HF face is directly generated from face priors via an HF branch that employs the proposed inception–hourglass model. As a result, allows the face priors to have a direct impact on the SR face by adding the outputs of both branches as the final result and provides an extra face editing function. Extensive experiments show that significantly outperforms state‐of‐the‐art face SR methods, is general for different texture branch models and face priors, and is robust to dataset mismatch and pose variations.

DAFNet: A Dual Attention Fusion-Based Face Super-Resolution Network

With the development of convolutional neural networks (CNNs), deep learning-based face super-resolution (FSR) approaches have achieved remarkable results in recent years. However, existing FSR methods often rely on face prior knowledge, which greatly increases network complexity and computation. In this paper, we propose DAFNet, a dual attention fusion-based FSR network comprising numerous face attention fusion modules (FAFMs). FAFM is a residual structure containing attention mechanism, which is divided into feature branches and attention branches. The feature branch introduces an hourglass block to extract multi-scale information, while the attention branch incorporates channel attention and spatial attention in series. This design ensures that the network prioritizes important information and effectively recovers detailed facial features. Luminance-chrominance error loss and gradient loss are introduced to guide the training process. Additionally, adversarial loss and perceptual loss are incorporated to enhance the recovery of visually realistic face images. Notably, our method can produce clear faces at a high-scale factor of eight times without relying on any facial prior information, effectively reducing network complexity. Quantitative and qualitative experiments conducted on the CelebA and Helen datasets underscore the effectiveness of the proposed model.

Progressive reconstruction-decoupled face super-resolution framework with controllable knowledge guidance

Large-factor face super-resolution is a severely ill-posed problem that requires additional information to assist in image reconstruction. Existing methods improve the super-resolution performance by introducing estimated face or pre-trained generative adversarial network (GAN) priors. However, they ignore the interference between contour and detail reconstruction as well as the errors in pretrained GAN and estimated face priors. To address these problems, we propose a progressive reconstruction-decoupled face super-resolution (PDFSR) framework with controllable knowledge guidance. First, a three-level index (pose, semantics, geometry) is proposed to utilize the high-resolution face texture to construct a lightweight knowledge base robust to bias. Then, the face contours are reconstructed by a basic super-resolution network. Finally, a constraint module is proposed to constrain the output of the knowledge base to provide accurate and controllable explicit knowledge for the reconstruction of facial details. The progressive framework decouples contour and detail reconstruction during training for the reconstructed images to consider both fidelity and perceptual quality. The knowledge base provides accurate and controllable low-dimensional explicit knowledge without estimation or pre-training, which reduces prior errors. Experiments demonstrate that our PDFSR outperforms state-of-the-art face super-resolution methods. Moreover, during the testing phase, faces can be edited by adjusting the explicit knowledge output by the knowledge base to obtain diverse reconstruction results.

Feature Maps Need More Attention: A Spatial-Channel Mutual Attention-Guided Transformer Network for Face Super-Resolution

Recently, transformer-based face super-resolution (FSR) approaches have achieved promising success in restoring degraded facial details due to their high capability for capturing both local and global dependencies. However, while existing methods focus on introducing sophisticated structures, they neglect the potential feature map information, limiting FSR performance. To circumvent this problem, we carefully design a pair of guiding blocks to dig for possible feature map information to enhance features before feeding them to transformer blocks. Relying on the guiding blocks, we propose a spatial-channel mutual attention-guided transformer network for FSR, for which the backbone architecture is a multi-scale connected encoder–decoder. Specifically, we devise a novel Spatial-Channel Mutual Attention-guided Transformer Module (SCATM), which is composed of a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). SCATM on the top layer (SCATM-T) aims to promote both local facial details and global facial structures, while SCATM on the bottom layer (SCATM-B) seeks to optimize the encoded features. Considering that different scale features are complementary, we further develop a Multi-scale Feature Fusion Module (MFFM), which fuses features from different scales for better restoration performance. Quantitative and qualitative experimental results on various datasets indicate that the proposed method outperforms other state-of-the-art FSR methods.

Why Not Both? An Attention-Guided Transformer with Pixel-Related Deconvolution Network for Face Super-Resolution

Transformer-based encoder-decoder networks for face super-resolution (FSR) have achieved promising success in delivering stunningly clear and detailed facial images by capturing local and global dependencies. However, these methods have certain limitations. Specifically, the deconvolution in upsampling layers neglects the relationship between adjacent pixels, which is crucial in facial structure reconstruction. Additionally, raw feature maps are fed to the transformer blocks directly without mining their potential feature information, resulting in suboptimal face images. To circumvent these problems, we propose an attention-guided transformer with pixel-related deconvolution network for FSR. Firstly, we devise a novel Attention-Guided Transformer Module (AGTM), which is composed of an Attention-Guiding Block (AGB) and a Channel-wise Multi-head Transformer Block (CMTB). AGTM at the top of the encoder-decoder network (AGTM-T) promotes both local facial details and global facial structures, while AGTM at the bottleneck side (AGTM-B) optimizes the encoded features. Secondly, a Pixel-Related Deconvolution (PRD) layer is specially designed to establish direct relationships among adjacent pixels in the upsampling process. Lastly, we develop a Multi-scale Feature Fusion Module (MFFM) to fuse multi-scale features for better network flexibility and reconstruction results. Quantitative and qualitative experimental results on various datasets demonstrate that the proposed method outperforms other state-of-the-art FSR methods.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

Edge fusion back projection GAN for large scale face super resolution

Face super-resolution is an important low-level vision task that has wide applications. Existing deep-learning-based face super-resolution (SR) methods often optimize the image super-resolution network by directly minimizing the pixel or feature level distance between the synthetic low-resolution face and the ground truth face image. These methods usually generate blur or over smooth results and lack of high-frequency face details. These are especially true for making high super-resolution of faces. Say for example, a super-resolution of 16x, only 0.4 % of the reference data points are available. To address the problem, we propose a novel network with edge fusion, back projection, and GAN prior (EFBPGAN) which can significantly improve the visual quality and generate realistic faces. To further make use of the spatial information and keep the structural consistency, we have developed new edge fusion and spatial fusion modules. We also propose a back projection extensive based coarse to fine SR pipeline to suppress the distortion and artifacts caused by GAN. Much experimental work has been done, results of which show that our proposed EFBPGAN can outperform the state-of-the-art approaches not only on numerical metrics but also on subjective visual evaluations.

Implicit Mutual Learning With Dual-Branch Networks for Face Super-Resolution

Implicit Mutual Learning With Dual-Branch Networks for Face Super-Resolution

Low-Light Face Super-resolution via Illumination, Structure, and Texture Associated Representation

Human face captured at night or in dimly lit environments has become a common practice, accompanied by complex low-light and low-resolution degradations. However, the existing face super-resolution (FSR) technologies and derived cascaded schemes are inadequate to recover credible textures. In this paper, we propose a novel approach that decomposes the restoration task into face structural fidelity maintaining and texture consistency learning. The former aims to enhance the quality of face images while improving the structural fidelity, while the latter focuses on eliminating perturbations and artifacts caused by low-light degradation and reconstruction. Based on this, we develop a novel low-light low-resolution face super-resolution framework. Our method consists of two steps: an illumination correction face super-resolution network (IC-FSRNet) for lighting the face and recovering the structural information, and a detail enhancement model (DENet) for improving facial details, thus making them more visually appealing and easier to analyze. As the relighted regions could provide complementary information to boost face super-resolution and vice versa, we introduce the mutual learning to harness the informative components from relighted regions and reconstruction, and achieve the iterative refinement. In addition, DENet equipped with diffusion probabilistic model is built to further improve face image visual quality. Experiments demonstrate that the proposed joint optimization framework achieves significant improvements in reconstruction quality and perceptual quality over existing two-stage sequential solutions. Code is available at https://github.com/wcy-cs/IC-FSRDENet.

CFGPFSR: A Generative Method Combining Facial and GAN Priors for Face Super-Resolution

In recent years, facial prior has been widely applied to enhance the quality of super-resolution (SR) facial images in face super-resolution (FSR) methods based on deep learning. However, most of the existing facial prior-based FSR methods have insufficient attention to local texture details, which can cause the generated SR facial images with overly smooth and unrealistic texture details, and show obvious artifacts under large magnification. With the help of GAN prior, recent advances can produce excellent results in terms of fidelity and realness. A generative framework for FSR is proposed in this work, which combines GAN and facial prior, termed CFGPFSR. Firstly, we pre-train a face StyleGAN2 and a face parsing network (FPN) that can generate decent parsing maps, in which the proposed CFGPFSR exploits rich and varied priors encapsulated in the face StyleGAN2 (GAN prior) and face parsing maps extracted from the FPN (facial prior) for FSR. Moreover, we introduce the Channel-Split Spatial Feature Transform (CS-SFT) method to further improve FSR performance. GAN and facial priors are introduced into the FSR process through the designed CS-SFT layers so that SR facial images obtain a promising balance between fidelity and realness. Unlike GAN inversion methods which necessitate costly image optimization at runtime, the proposed CFGPFSR can jointly recover facial details by only utilizing one forward pass. Experimental results on synthetic and real images indicate that the proposed CFGPFSR obtains remarkable performance in 16 × SR task, and some of its metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM) are higher than that of the comparison methods. Meanwhile, it shows impressive results in reconstructing high-quality facial images.

Edge-attention network for preserving structure in face super-resolution

Edge-attention network for preserving structure in face super-resolution

Rethinking Prior-Guided Face Super-Resolution: A New Paradigm With Facial Component Prior.

Recently, facial priors (e.g., facial parsing maps and facial landmarks) have been widely employed in prior-guided face super-resolution (FSR) because it provides the location of facial components and facial structure information, and helps predict the missing high-frequency (HF) information. However, most existing approaches suffer from two shortcomings: 1) the extracted facial priors are inaccurate since they are extracted from low-resolution (LR) or low-quality super-resolved (SR) face images and 2) they only consider embedding facial priors into the reconstruction process from LR to SR face images, thus failing to explore facial priors to generate LR face image. In this article, we propose a novel pre-prior guided approach that extracts facial prior information from original high-resolution (HR) face images and embeds them into LR ones to obtain HF information-rich LR face images, thereby improving the performance of face reconstruction. Specifically, a novel component hybrid method is proposed, which fuses HR facial components and LR facial background to generate new LR face images (namely, LRmix) via facial parsing maps extracted from HR face images. Furthermore, we design a component hybrid network (CHNet) that learns the LR to LRmix mapping function to ensure that the LRmix can be obtained from LR face images in testing and real-world datasets. Experimental results show that our proposed scheme significantly improves the reconstruction performance for FSR.

An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution

In the Face Super-Resolution (FSR) task, it is important to precisely recover facial textures while maintaining facial contours for realistic high resolution faces. Although several CNN-based FSR methods have achieved great performance, they fail in restoring the facial contours due to the limitation of local convolutions. In contrast, Transformer-based methods which use self-attention as the basic component, are expert in modeling long-range dependencies between image patches. However, learning long-range dependencies often deteriorates facial textures due to the lack of locality. Therefore, a question is naturally raised: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">how to effectively combine the superiority of CNN and Transformer for better reconstructing faces?</i> To address this issue, we propose an Efficient Latent Style guided Transformer-CNN framework for FSR called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ELSFace</b> , which can sufficiently integrate the advantages of CNN and Transformer. The framework consists of a Feature Preparation Stage and a Feature Carving Stage. Basic facial contours and textures are generated in the Feature Preparation Stage, and separately guided by latent styles, so that facial details are better represented in reconstruction. CNN and Transformer streams in the Feature Carving Stage are used to individually restore facial textures and facial contours, respectively in a parallel recursive way. Considering the negligence of high-frequency features when learning the long-range dependencies, we design the High-Frequency Enhancement Block (HFEB) in the Transformer stream. The Sharp Loss is also proposed for better perceptual quality in optimization. Extensive experimental results demonstrate that our ELSFace can achieve the best results among all metrics compared to the state-of-the-art CNN and Transformer-based methods on commonly used datasets and real-world tasks. Meanwhile, our ELSFace method has the least model parameters and running time. The codes are released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FVL2020/ELSFace</uri> .