Face Super-resolution Methods Research Articles

Cross-Modal Face Super-Resolution Based on Quasi-Siamese Domain Transfer Fusion Network

In this paper, we propose a Cross-Modal Face Super-Resolution (CMFSR) method to construct high-resolution (HR) facial images from low-resolution (LR) cross-modal facial images captured respectively by disjoint visible light (VIS) and near-infrared (NIR) cameras. Due to the coupling of modality transformation and information fusion, CMFSR is more difficult to obtain HR reconstructed results compared with traditional super-resolution. To solve this problem, a Quasi-Siamese Domain Transfer Fusion Network (QSDTFN) for CMFSR is proposed in this paper, whose two branches transfer two LR face modality to HR face modality by domain transfer respectively. Different from two completely independent branches in the traditional pseudo-siamese network, only the HR-to-LR face transfer processes of the two branches in our quasi-siamese network are independent, while the LR-to-HR face transfer processes are coupled. This coupled module called the Adaptive Weighted Domain Transfer Fusion Module (AWDTFM) disentangles the modality and identity information in the two LR faces, thus achieving modality transformation and identity information fusion simultaneously. In order to strengthen the optimization on the process of CMFSR, this method further introduces the backward QSDTFN to form a higher-level bidirectional structure with the forward QSDTFN, and specifically designs two types of losses: intra-network loss and inter-network loss, to constrain the modality and identity consistencies within one QSDTFN and between two QSDTFNs respectively. The experimental results on the challenging LR cross-modal face datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods.

Face super resolution with a high frequency highway

AbstractFace shape priors such as landmarks, heatmaps, and parsing maps are widely used to improve face super resolution (SR). It is observed that face priors provide locations of high‐frequency details in key facial areas such as the eyes and mouth. However, existing methods fail to effectively exploit the high‐frequency information by using the priors as either constraints or inputs. This paper proposes a novel high frequency highway () framework to better utilize prior information for face SR, which dynamically decomposes the final SR face into a coarse SR face and a high frequency (HF) face. The coarse SR face is reconstructed from a low‐resolution face via a texture branch, using only pixel‐wise reconstruction loss. Meanwhile, the HF face is directly generated from face priors via an HF branch that employs the proposed inception–hourglass model. As a result, allows the face priors to have a direct impact on the SR face by adding the outputs of both branches as the final result and provides an extra face editing function. Extensive experiments show that significantly outperforms state‐of‐the‐art face SR methods, is general for different texture branch models and face priors, and is robust to dataset mismatch and pose variations.

DAFNet: A Dual Attention Fusion-Based Face Super-Resolution Network

With the development of convolutional neural networks (CNNs), deep learning-based face super-resolution (FSR) approaches have achieved remarkable results in recent years. However, existing FSR methods often rely on face prior knowledge, which greatly increases network complexity and computation. In this paper, we propose DAFNet, a dual attention fusion-based FSR network comprising numerous face attention fusion modules (FAFMs). FAFM is a residual structure containing attention mechanism, which is divided into feature branches and attention branches. The feature branch introduces an hourglass block to extract multi-scale information, while the attention branch incorporates channel attention and spatial attention in series. This design ensures that the network prioritizes important information and effectively recovers detailed facial features. Luminance-chrominance error loss and gradient loss are introduced to guide the training process. Additionally, adversarial loss and perceptual loss are incorporated to enhance the recovery of visually realistic face images. Notably, our method can produce clear faces at a high-scale factor of eight times without relying on any facial prior information, effectively reducing network complexity. Quantitative and qualitative experiments conducted on the CelebA and Helen datasets underscore the effectiveness of the proposed model.

Progressive reconstruction-decoupled face super-resolution framework with controllable knowledge guidance

Large-factor face super-resolution is a severely ill-posed problem that requires additional information to assist in image reconstruction. Existing methods improve the super-resolution performance by introducing estimated face or pre-trained generative adversarial network (GAN) priors. However, they ignore the interference between contour and detail reconstruction as well as the errors in pretrained GAN and estimated face priors. To address these problems, we propose a progressive reconstruction-decoupled face super-resolution (PDFSR) framework with controllable knowledge guidance. First, a three-level index (pose, semantics, geometry) is proposed to utilize the high-resolution face texture to construct a lightweight knowledge base robust to bias. Then, the face contours are reconstructed by a basic super-resolution network. Finally, a constraint module is proposed to constrain the output of the knowledge base to provide accurate and controllable explicit knowledge for the reconstruction of facial details. The progressive framework decouples contour and detail reconstruction during training for the reconstructed images to consider both fidelity and perceptual quality. The knowledge base provides accurate and controllable low-dimensional explicit knowledge without estimation or pre-training, which reduces prior errors. Experiments demonstrate that our PDFSR outperforms state-of-the-art face super-resolution methods. Moreover, during the testing phase, faces can be edited by adjusting the explicit knowledge output by the knowledge base to obtain diverse reconstruction results.

Feature Maps Need More Attention: A Spatial-Channel Mutual Attention-Guided Transformer Network for Face Super-Resolution

Recently, transformer-based face super-resolution (FSR) approaches have achieved promising success in restoring degraded facial details due to their high capability for capturing both local and global dependencies. However, while existing methods focus on introducing sophisticated structures, they neglect the potential feature map information, limiting FSR performance. To circumvent this problem, we carefully design a pair of guiding blocks to dig for possible feature map information to enhance features before feeding them to transformer blocks. Relying on the guiding blocks, we propose a spatial-channel mutual attention-guided transformer network for FSR, for which the backbone architecture is a multi-scale connected encoder–decoder. Specifically, we devise a novel Spatial-Channel Mutual Attention-guided Transformer Module (SCATM), which is composed of a Spatial-Channel Mutual Attention Guiding Block (SCAGB) and a Channel-wise Multi-head Transformer Block (CMTB). SCATM on the top layer (SCATM-T) aims to promote both local facial details and global facial structures, while SCATM on the bottom layer (SCATM-B) seeks to optimize the encoded features. Considering that different scale features are complementary, we further develop a Multi-scale Feature Fusion Module (MFFM), which fuses features from different scales for better restoration performance. Quantitative and qualitative experimental results on various datasets indicate that the proposed method outperforms other state-of-the-art FSR methods.

SwinDPSR: Dual-Path Face Super-Resolution Network Integrating Swin Transformer

Whether to use face priors in the face super-resolution (FSR) methods is a symmetry problem.Various face priors are used to describe the overall and local face features, making the generation of super-resolution face images expensive and laborious. FSR methods that do not require any prior information tend to focus too much on the local features of the face, ignoring the modeling of global information. To solve this problem, we propose a dual-path facial image super-resolution network (SwinDPSR) fused with Swin Transformer. The network does not require additional face priors, and it learns global face shape and local face components through two independent branches. In addition, the channel attention ECA module is used to aggregate the global and local face information in the above dual-path sub-networks, which can generate corresponding high-quality face images. The results of face super-resolution reconstruction experiments on public face datasets and a real-scene face dataset show that SwinDPSR is superior to previous advanced methods both in terms of visual effects and objective indicators. The reconstruction results are evaluated with four evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity (SSIM), learned perceptual image patch similarity (LPIPS), and mean perceptual score (MPS).

Edge fusion back projection GAN for large scale face super resolution

Face super-resolution is an important low-level vision task that has wide applications. Existing deep-learning-based face super-resolution (SR) methods often optimize the image super-resolution network by directly minimizing the pixel or feature level distance between the synthetic low-resolution face and the ground truth face image. These methods usually generate blur or over smooth results and lack of high-frequency face details. These are especially true for making high super-resolution of faces. Say for example, a super-resolution of 16x, only 0.4 % of the reference data points are available. To address the problem, we propose a novel network with edge fusion, back projection, and GAN prior (EFBPGAN) which can significantly improve the visual quality and generate realistic faces. To further make use of the spatial information and keep the structural consistency, we have developed new edge fusion and spatial fusion modules. We also propose a back projection extensive based coarse to fine SR pipeline to suppress the distortion and artifacts caused by GAN. Much experimental work has been done, results of which show that our proposed EFBPGAN can outperform the state-of-the-art approaches not only on numerical metrics but also on subjective visual evaluations.

CFGPFSR: A Generative Method Combining Facial and GAN Priors for Face Super-Resolution

In recent years, facial prior has been widely applied to enhance the quality of super-resolution (SR) facial images in face super-resolution (FSR) methods based on deep learning. However, most of the existing facial prior-based FSR methods have insufficient attention to local texture details, which can cause the generated SR facial images with overly smooth and unrealistic texture details, and show obvious artifacts under large magnification. With the help of GAN prior, recent advances can produce excellent results in terms of fidelity and realness. A generative framework for FSR is proposed in this work, which combines GAN and facial prior, termed CFGPFSR. Firstly, we pre-train a face StyleGAN2 and a face parsing network (FPN) that can generate decent parsing maps, in which the proposed CFGPFSR exploits rich and varied priors encapsulated in the face StyleGAN2 (GAN prior) and face parsing maps extracted from the FPN (facial prior) for FSR. Moreover, we introduce the Channel-Split Spatial Feature Transform (CS-SFT) method to further improve FSR performance. GAN and facial priors are introduced into the FSR process through the designed CS-SFT layers so that SR facial images obtain a promising balance between fidelity and realness. Unlike GAN inversion methods which necessitate costly image optimization at runtime, the proposed CFGPFSR can jointly recover facial details by only utilizing one forward pass. Experimental results on synthetic and real images indicate that the proposed CFGPFSR obtains remarkable performance in 16 × SR task, and some of its metrics such as peak signal to noise ratio (PSNR) and structural similarity (SSIM) are higher than that of the comparison methods. Meanwhile, it shows impressive results in reconstructing high-quality facial images.

Noisy face super-resolution method based on three-level information representation constraints

Noisy face super-resolution method based on three-level information representation constraints

An Efficient Latent Style Guided Transformer-CNN Framework for Face Super-Resolution

In the Face Super-Resolution (FSR) task, it is important to precisely recover facial textures while maintaining facial contours for realistic high resolution faces. Although several CNN-based FSR methods have achieved great performance, they fail in restoring the facial contours due to the limitation of local convolutions. In contrast, Transformer-based methods which use self-attention as the basic component, are expert in modeling long-range dependencies between image patches. However, learning long-range dependencies often deteriorates facial textures due to the lack of locality. Therefore, a question is naturally raised: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">how to effectively combine the superiority of CNN and Transformer for better reconstructing faces?</i> To address this issue, we propose an Efficient Latent Style guided Transformer-CNN framework for FSR called <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ELSFace</b> , which can sufficiently integrate the advantages of CNN and Transformer. The framework consists of a Feature Preparation Stage and a Feature Carving Stage. Basic facial contours and textures are generated in the Feature Preparation Stage, and separately guided by latent styles, so that facial details are better represented in reconstruction. CNN and Transformer streams in the Feature Carving Stage are used to individually restore facial textures and facial contours, respectively in a parallel recursive way. Considering the negligence of high-frequency features when learning the long-range dependencies, we design the High-Frequency Enhancement Block (HFEB) in the Transformer stream. The Sharp Loss is also proposed for better perceptual quality in optimization. Extensive experimental results demonstrate that our ELSFace can achieve the best results among all metrics compared to the state-of-the-art CNN and Transformer-based methods on commonly used datasets and real-world tasks. Meanwhile, our ELSFace method has the least model parameters and running time. The codes are released at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FVL2020/ELSFace</uri> .

Reconstruction with robustness: A semantic prior guided face super-resolution framework for multiple degradations

Despite the rapid advancements made with the support of deep learning, Face Super-Resolution (FSR) methods still suffer from challenges under multiple degradations. These challenges significantly impede the practical applications of FSR methods in real-world scenarios. Incorporating facial priors could potentially relieve this issue. However, ground truth priors are not feasible in real-world applications, meanwhile the accuracy of predicted priors is difficult to guarantee, especially for low-resolution faces under multiple degradations. Hence, it is worth exploring how to effectively leverage facial priors for improving the robustness of FSR under multiple degradations. To tackle these problems, we propose RSemFace, a robust semantic prior guided FSR framework to reconstruct multiple degraded faces. In RSemFace, we design the Degradation Stage to synthesize multiple degraded low-resolution faces with a variety of interpolations, noise levels, blurring kernels, and even the real-world interference. The Generation Stage generates Coarse-SR faces, and extracts semantic features from the Coarse-SR as priors, which are used to the reconstruction of Fine-SR faces with the support of Semantic Feature Attention Blocks (SFABs) and Semantic Loss. Both quantitative and qualitative results show the better robustness of our RSemFace for content recovery and perceptual quality in simultaneously handling multiple degraded faces compared with other state-of-the-art methods. Lastly, faces reconstructed by RSemFace are proven to improve the high-level vision task due to better recovered identities.

Face Super-Resolution via Joint Edge Information and Attention Aggregation Network

Face super-resolution (SR) is a specialized image super-resolution problem that is particularly relevant in various intelligent transportation scenes such as video surveillance and identification systems. Deep learning-steered solutions for face images often leverage facial priors (i.e., face parsing and landmark) to restore intricate facial components and have demonstrated promising performance. However, these methods typically rely on abundant manually labeled data and require lengthy training times. In this work, we introduce a meaningful approach called the Joint Edge Information and Attention Aggregation Network (JEANet) for the face SR problem. JEANet is established on our carefully designed Face Attention Aggregation Module (FAAM), which incorporates parallel connections of both channel-wise and spatial-wise features. Specifically, we integrate an attention fusion strategy into the residual blocks and elaborately employ edge blocks to extract edge information. Additionally, we introduce adaptive shortcuts that interpolate reconstruction parts at multiple scales. With the aid of adaptive shortcuts, JEANet effectively restores detailed images textures by leveraging contextual features optimally. These considerations enable the convolutional operations to flexibly distill information related to primary facial structures and progressively incorporate edge information to assemble local and global meaningful structures. Qualitative and quantitative evaluations demonstrate that our proposed method excels in recovering highly realistic face images compared to several competitive face SR methods.

A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors

Face super-resolution (FSR) aims to reconstruct high-resolution face images from low-resolution (LR) ones. Despite the progress made by deep convolutional neural networks (DCNNs) on FSR, convolutions struggle to relate spatially distant concepts and what is more, all image pixels and prior information (e.g., landmarks and facial component heatmaps) are treated equally regardless of importance, causing inaccuracy and decreasing the quality of face image recovery. To address these issues, in this paper we propose a composite network model for FSR with multi-order head attention facial priors. The proposed model contains a face hallucination transformer (FHT)-based network and a multi-order head attention (MOHA)-based DCNN. The FHT-based network can capture long-range dependencies and gradually increase resolution to achieve efficient and effective inference, while the MOHA-based DCNN exploits detailed and two-dimensional information of LR face images. Moreover, the novel generic submodule of the MOHA-based DCNN, namely Multi-Order Head Attention Network, can accurately model the relationship of facial components between spatial and channel dimensions. The proposed composite network model seamlessly integrates the advantages of DCNNs and transformers to super-resolve LR face images. When compared with state-of-the-art FSR methods on public benchmark datasets, the proposed model shows competitive recovery performance.

Multi-phase attention network for face super-resolution.

Previous general super-resolution methods do not perform well in restoring the details structure information of face images. Prior and attribute-based face super-resolution methods have improved performance with extra trained results. However, they need an additional network and extra training data are challenging to obtain. To address these issues, we propose a Multi-phase Attention Network (MPAN). Specifically, our proposed MPAN builds on integrated residual attention groups (IRAG) and a concatenated attention module (CAM). The IRAG consists of residual channel attention blocks (RCAB) and an integrated attention module (IAM). Meanwhile, we use IRAG to bootstrap the face structures. We utilize the CAM to concentrate on informative layers, hence improving the network's ability to reconstruct facial texture features. We use the IAM to focus on important positions and channels, which makes the network more effective at restoring key face structures like eyes and mouths. The above two attention modules form the multi-phase attention mechanism. Extensive experiments show that our MPAN has a significant competitive advantage over other state-of-the-art networks on various scale factors using various metrics, including PSNR and SSIM. Overall, our proposed Multi-phase Attention mechanism significantly improves the network for recovering face HR images without using additional information.

Self-attention learning network for face super-resolution

Existing face super-resolution methods depend on deep convolutional networks (DCN) to recover high-quality reconstructed images. They either acquire information in a single space by designing complex models for direct reconstruction, or employ additional networks to extract multiple prior information to enhance the representation of features. However, existing methods are still challenging to perform well due to the inability to learn complete and uniform representations. To this end, we propose a self-attention learning network (SLNet) for three-stage face super-resolution, which fully explores the interdependence of low- and high-level spaces to achieve compensation of the information used for reconstruction. Firstly, SLNet uses a hierarchical feature learning framework to obtain shallow information in the low-level space. Then, the shallow information with cumulative errors due to DCN is improved under high-resolution (HR) supervision, while bringing an intermediate reconstruction result and a powerful intermediate benchmark. Finally, the improved feature representation is further enhanced in high-level space by a multi-scale context-aware encoder–decoder for facial reconstruction. The features in both spaces are explored progressively from coarse to fine reconstruction information. The experimental results show that SLNet has a competitive performance compared to the state-of-the-art methods.

Low-Light Robust Face Image Super-Resolution via Neuro-Fuzzy Inferencing-Based Locality Constrained Representation

Face super-resolution (FSR) has recently become an interesting topic of study in the fields of image processing and computer vision. However, current FSR methods are still incapable of dealing with those faces suffering from low-light issues, which are frequently caused by uneven lighting conditions in the environment. Therefore, a novel neuro-fuzzy inferencing-based locality-constrained representation (NFILcR) method for low-light robust FSR is proposed in this paper. In NFILcR, the adaptive neuro-fuzzy inferencing system and locality-constrained representation approach are integrated into one unified framework to super resolve the captured low-light and low-resolution faces. Specifically, an adaptive neuro-fuzzy inferencing-based low-light factor estimator (ANFILLE) model is introduced that estimates the low-light factor in the test images. Further, the estimated low-light factor is incorporated into the objective function of the proposed NFILcR method. These contributions make the proposed method robust against the low-light issue. The experimental results on benchmark face datasets and real-world images show that the proposed NFILcR method outperforms the other FSR methods that have been compared.

CTCNet: A CNN-Transformer Cooperation Network for Face Image Super-Resolution.

Recently, deep convolution neural networks (CNNs) steered face super-resolution methods have achieved great progress in restoring degraded facial details by joint training with facial priors. However, these methods have some obvious limitations. On the one hand, multi-task joint learning requires additional marking on the dataset, and the introduced prior network will significantly increase the computational cost of the model. On the other hand, the limited receptive field of CNN will reduce the fidelity and naturalness of the reconstructed facial images, resulting in suboptimal reconstructed images. In this work, we propose an efficient CNN-Transformer Cooperation Network (CTCNet) for face super-resolution tasks, which uses the multi-scale connected encoder-decoder architecture as the backbone. Specifically, we first devise a novel Local-Global Feature Cooperation Module (LGCM), which is composed of a Facial Structure Attention Unit (FSAU) and a Transformer block, to promote the consistency of local facial detail and global facial structure restoration simultaneously. Then, we design an efficient Feature Refinement Module (FRM) to enhance the encoded features. Finally, to further improve the restoration of fine facial details, we present a Multi-scale Feature Fusion Unit (MFFU) to adaptively fuse the features from different stages in the encoder procedure. Extensive evaluations on various datasets have assessed that the proposed CTCNet can outperform other state-of-the-art methods significantly. Source code will be available at https://github.com/IVIPLab/CTCNet.

Propagating Facial Prior Knowledge for Multitask Learning in Face Super-Resolution

Existing face hallucination methods always achieve improved performance through regularizing the model with facial prior. Most of them always estimate facial prior information first and then leverage it to help the prediction of the target high-resolution face image. However, the accuracy of prior estimation is difficult to guarantee, especially for the low-resolution face image. Once the estimated prior is inaccurate or wrong, the following face super-resolution performance is unavoidably influenced. A natural question that arises: how to incorporate facial prior effectively and efficiently without prior estimation? To achieve this goal, we propose to learn facial prior knowledge at training stage, but test only with low-resolution face image, which can overcome the difficulty of estimating accurate prior. In addition, instead of estimating facial prior, we directly explore the potential of high-quality facial prior in the training phase and progressively propagate the facial prior knowledge from the teacher network (trained with the low-resolution face/high-quality facial prior and high-resolution face image pairs) to the student network (trained with the low-resolution face and high-resolution face image pairs). Quantitative and qualitative comparisons on benchmark face datasets demonstrate that our method outperforms the state-of-the-art face super-resolution methods. The source codes of the proposed method will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/wcy-cs/KDFSRNet</uri> .

Deep Illumination-Enhanced Face Super-Resolution Network for Low-Light Images

Face images are typically a key component in the fields of security and criminal investigation. However, due to lighting and shooting angles, faces taken under low-light conditions are often difficult to recognize. Face super-resolution (FSR) technology can restore high-resolution faces based on low-resolution inputs. However, existing face super-resolution methods typically rely on prior knowledge of inaccurate faces estimated from low-resolution images. Faces restored by low-light inputs may suffer from problems such as low brightness and many missing details. In this article, we proposed an Illumination-Enhanced Face Super-Resolution (IEFSR) model that can progressively super-resolve low-light faces of 32 × 32 pixels by an upscaling factor of 8. While reconstructing the low-light low-resolution face into a clear and high-quality face, we introduce a coarse low-resolution (LR) restoration network to recover the LR face details hidden in the dark. In the generator, we use a series of style blocks with noise to make the generated faces appear to have a more realistic visual aesthetic. Additionally, we introduce spectrum normalization in the discriminator to improve training stability. Extensive experimental evaluations show that the proposed IEFSR yields visually and metrically more attractive results than existing state-of-the-art FSR methods.

Attention-Driven Graph Neural Network for Deep Face Super-Resolution.

With the help of convolutional neural networks (CNNs), deep learning-based methods have achieved remarkable performance in face super-resolution (FSR) task. Despite their success, most of the existing methods neglect non-local correlations of face images, leaving much room for improvement. In this paper, we introduce a novel end-to-end trainable attention-driven graph neural network (AD-GNN) for more discriminative feature extraction and feature relation modeling. This is achieved by two major components. The first component is a cross-scale dynamic graph (CDG) block. The CDG block considers cross-scale relationships of patches in distant areas and employs two dynamic graphs to construct enhanced features. The second component is a series of channel attention and spatial dynamic graph (CASDG) blocks. A CASDG block has a channel-wise attention unit and a spatial-aware dynamic graph (SDG) unit. The SDG unit extracts informative features by exploring spatial non-local self-similarity information of the patches using dynamic graph convolution. Using these two components, facial details can be effectively reconstructed with the help of information supplemented by similar but spatially remote patches and structural information of faces. Extensive experiments on two public benchmarks demonstrate the superiority of AD-GNN over the state-of-the-art FSR methods.