Low-resolution Face Research Articles

Cross-Modal Face Super-Resolution Based on Quasi-Siamese Domain Transfer Fusion Network

In this paper, we propose a Cross-Modal Face Super-Resolution (CMFSR) method to construct high-resolution (HR) facial images from low-resolution (LR) cross-modal facial images captured respectively by disjoint visible light (VIS) and near-infrared (NIR) cameras. Due to the coupling of modality transformation and information fusion, CMFSR is more difficult to obtain HR reconstructed results compared with traditional super-resolution. To solve this problem, a Quasi-Siamese Domain Transfer Fusion Network (QSDTFN) for CMFSR is proposed in this paper, whose two branches transfer two LR face modality to HR face modality by domain transfer respectively. Different from two completely independent branches in the traditional pseudo-siamese network, only the HR-to-LR face transfer processes of the two branches in our quasi-siamese network are independent, while the LR-to-HR face transfer processes are coupled. This coupled module called the Adaptive Weighted Domain Transfer Fusion Module (AWDTFM) disentangles the modality and identity information in the two LR faces, thus achieving modality transformation and identity information fusion simultaneously. In order to strengthen the optimization on the process of CMFSR, this method further introduces the backward QSDTFN to form a higher-level bidirectional structure with the forward QSDTFN, and specifically designs two types of losses: intra-network loss and inter-network loss, to constrain the modality and identity consistencies within one QSDTFN and between two QSDTFNs respectively. The experimental results on the challenging LR cross-modal face datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods.

Edge fusion back projection GAN for large scale face super resolution

Face super-resolution is an important low-level vision task that has wide applications. Existing deep-learning-based face super-resolution (SR) methods often optimize the image super-resolution network by directly minimizing the pixel or feature level distance between the synthetic low-resolution face and the ground truth face image. These methods usually generate blur or over smooth results and lack of high-frequency face details. These are especially true for making high super-resolution of faces. Say for example, a super-resolution of 16x, only 0.4 % of the reference data points are available. To address the problem, we propose a novel network with edge fusion, back projection, and GAN prior (EFBPGAN) which can significantly improve the visual quality and generate realistic faces. To further make use of the spatial information and keep the structural consistency, we have developed new edge fusion and spatial fusion modules. We also propose a back projection extensive based coarse to fine SR pipeline to suppress the distortion and artifacts caused by GAN. Much experimental work has been done, results of which show that our proposed EFBPGAN can outperform the state-of-the-art approaches not only on numerical metrics but also on subjective visual evaluations.

Low-Light Face Super-resolution via Illumination, Structure, and Texture Associated Representation

Human face captured at night or in dimly lit environments has become a common practice, accompanied by complex low-light and low-resolution degradations. However, the existing face super-resolution (FSR) technologies and derived cascaded schemes are inadequate to recover credible textures. In this paper, we propose a novel approach that decomposes the restoration task into face structural fidelity maintaining and texture consistency learning. The former aims to enhance the quality of face images while improving the structural fidelity, while the latter focuses on eliminating perturbations and artifacts caused by low-light degradation and reconstruction. Based on this, we develop a novel low-light low-resolution face super-resolution framework. Our method consists of two steps: an illumination correction face super-resolution network (IC-FSRNet) for lighting the face and recovering the structural information, and a detail enhancement model (DENet) for improving facial details, thus making them more visually appealing and easier to analyze. As the relighted regions could provide complementary information to boost face super-resolution and vice versa, we introduce the mutual learning to harness the informative components from relighted regions and reconstruction, and achieve the iterative refinement. In addition, DENet equipped with diffusion probabilistic model is built to further improve face image visual quality. Experiments demonstrate that the proposed joint optimization framework achieves significant improvements in reconstruction quality and perceptual quality over existing two-stage sequential solutions. Code is available at https://github.com/wcy-cs/IC-FSRDENet.

Rethinking Prior-Guided Face Super-Resolution: A New Paradigm With Facial Component Prior.

Recently, facial priors (e.g., facial parsing maps and facial landmarks) have been widely employed in prior-guided face super-resolution (FSR) because it provides the location of facial components and facial structure information, and helps predict the missing high-frequency (HF) information. However, most existing approaches suffer from two shortcomings: 1) the extracted facial priors are inaccurate since they are extracted from low-resolution (LR) or low-quality super-resolved (SR) face images and 2) they only consider embedding facial priors into the reconstruction process from LR to SR face images, thus failing to explore facial priors to generate LR face image. In this article, we propose a novel pre-prior guided approach that extracts facial prior information from original high-resolution (HR) face images and embeds them into LR ones to obtain HF information-rich LR face images, thereby improving the performance of face reconstruction. Specifically, a novel component hybrid method is proposed, which fuses HR facial components and LR facial background to generate new LR face images (namely, LRmix) via facial parsing maps extracted from HR face images. Furthermore, we design a component hybrid network (CHNet) that learns the LR to LRmix mapping function to ensure that the LRmix can be obtained from LR face images in testing and real-world datasets. Experimental results show that our proposed scheme significantly improves the reconstruction performance for FSR.

Research on Face Attribute Recognition Technology Based on Fine-Grained Features

Abstract This study explores fine-grained feature-based face attribute recognition techniques to enhance the accuracy of face recognition in low-resolution and complex environments. The article proposes a global feature extraction method and a local texture feature extraction method to extract global and regional features by enhancing feature reuse and information flow through dense connectivity and ShuffleNet V2 framework. Then, a multiscale feature exchange method is used to fuse different scale features to enhance the capture of detail information. Finally, efficient feature integration is achieved by the multiscale feature fusion method. Experimental results on the CK+ and FER2013 datasets show that the accuracy of this method on face expression recognition reaches 97.24% and 95.93%, respectively, and the average recognition accuracy in the face attribute recognition experiments on the CelebA dataset is 97.11%, which is significantly better than the comparison algorithm. In addition, the analysis of the recognition effect on low-resolution faces shows that this paper’s method achieves a recognition accuracy of 54.03% at a resolution of 15 × 15 and a high accuracy of over 99% at resolutions of 70 × 70 and above. These results show that the face attribute recognition technique based on fine-grained features proposed in this paper significantly improves recognition accuracy.

Degradation model and attention guided distillation approach for low resolution face recognition

Deep convolution neural networks (CNN) have shown their efficacy in face recognition tasks due to their ability to extract highly discriminant face representations from face images. On high-resolution benchmark datasets, outstanding identification and verification results have been achieved. However, the performance of these algorithms is significantly degraded when tested on low-resolution (LR) images such as those captured from surveillance cameras. A straightforward solution to this problem is to use both high-resolution (HR) images and corresponding down-sampled LR images during training. Although this strategy improves the performance for LR images, it has some limitations. First, there is a significant difference between down-sampled LR images and LR images from surveillance cameras, leading to performance saturation at an earlier stage. Another limitation is the deterioration in the performance of HR images. In this work, solutions to both these limitations are proposed. A degradation model is proposed that synthesizes LR images from corresponding HR, emulating the real-world degradation effects in synthetic data, thus enabling the face recognition system to tolerate various blurry and noisy effects. To address the deterioration in the performance of HR images, an attention-guided distillation is proposed, which utilizes attention maps from convolutional layers in combination with deep features to transfer informative HR features from teacher to student network. The attention maps from the teacher network guide the student network to a better optimum and produce resolution robust face representations. The results of the proposed approach on the popular LR datasets like SCface, Coxface, and PaSC show that it outperforms the recent state-of-the-art (SOTA) techniques by a significant margin demonstrating its effectiveness for different cross-resolution scenarios.

Seg-DGDNet: Segmentation Based Disguise Guided Dropout Network for Low Resolution Face Recognition

Seg-DGDNet: Segmentation Based Disguise Guided Dropout Network for Low Resolution Face Recognition

Reconstruction with robustness: A semantic prior guided face super-resolution framework for multiple degradations

Despite the rapid advancements made with the support of deep learning, Face Super-Resolution (FSR) methods still suffer from challenges under multiple degradations. These challenges significantly impede the practical applications of FSR methods in real-world scenarios. Incorporating facial priors could potentially relieve this issue. However, ground truth priors are not feasible in real-world applications, meanwhile the accuracy of predicted priors is difficult to guarantee, especially for low-resolution faces under multiple degradations. Hence, it is worth exploring how to effectively leverage facial priors for improving the robustness of FSR under multiple degradations. To tackle these problems, we propose RSemFace, a robust semantic prior guided FSR framework to reconstruct multiple degraded faces. In RSemFace, we design the Degradation Stage to synthesize multiple degraded low-resolution faces with a variety of interpolations, noise levels, blurring kernels, and even the real-world interference. The Generation Stage generates Coarse-SR faces, and extracts semantic features from the Coarse-SR as priors, which are used to the reconstruction of Fine-SR faces with the support of Semantic Feature Attention Blocks (SFABs) and Semantic Loss. Both quantitative and qualitative results show the better robustness of our RSemFace for content recovery and perceptual quality in simultaneously handling multiple degraded faces compared with other state-of-the-art methods. Lastly, faces reconstructed by RSemFace are proven to improve the high-level vision task due to better recovered identities.

Coupled discriminative manifold alignment for low-resolution face recognition

In practical applications, due to a long distance between the monitored population and monitoring equipment, the face images or human pose captured by the cameras often incur low-resolution (LR), small size, and poor quality, which leads to extreme difficulty in directly matching an LR face with the high-resolution (HR) ones in the gallery. In this paper, we propose a novel coupled discriminative manifold alignment (CDMA) method for LR face recognition. Specifically, in the training stage the principal component analysis (PCA) is first used to reduce the dimensional gap between LR and HR facial features. Next the LR face images and the corresponding HR face images are converted into a common shared feature subspace by learning two linear mappings in a supervised manner, where the neighborhood samples within the same class and from different classes are jointly exploited to align the manifold structures of LR and HR faces. In the test stage, for a given LR face in the probe set, two learned coupled mappings (CMs) are applied to match the HR images in the gallery set through the correlative metric. Thorough experimental results on three representative face databases verify the effectiveness of the proposed method in comparing with other state-of-the-art competitors. In particular, the proposed method is capable of yielding more competitive recognition performance than other predecessors when lower dimensional feature subspaces are applied to match the expected HR faces.

Super-Resolving Face Image by Facial Parsing Information

Face super-resolution is a technology that transforms a low-resolution face image into the corresponding high-resolution one. In this paper, we build a novel parsing map guided face super-resolution network which extracts the face prior (i.e., parsing map) directly from low-resolution face image for the following utilization. To exploit the extracted prior fully, a parsing map attention fusion block is carefully designed, which can not only effectively explore the information of parsing map, but also combines powerful attention mechanism. Moreover, in light of that high-resolution features contain more precise spatial information while low-resolution features provide strong contextual information, we hope to maintain and utilize these complementary information. To achieve this goal, we develop a multi-scale refine block to maintain spatial and contextual information and take advantage of multi-scale features to refine the feature representations. Experimental results demonstrate that our method outperforms the state-of-the-arts in terms of quantitative metrics and visual quality. The source codes will be available at.

A Multi-Scale Deep Back-Projection Backbone for Face Super-Resolution with Diffusion Models

Face verification and recognition are important tasks that have made great progress in recent years. However, recognizing low-resolution faces from small images is still a difficult problem. In this paper, we advocate using diffusion models (DMs) to enhance face resolution and improve their quality for various downstream applications. Most existing DMs for super-resolution use U-Net as their backbone network, which only exploits multi-scale features along the spatial dimension. These approaches result in a slow convergence of corresponding DMs and the inability to capture complex details and fine textures. To address this issue, we propose a novel conditional generative model based on DMs called BPSR3, which replaces the U-Net in super-resolution via repeated refinement (SR3) with a multi-scale deep back-projection network structure. BPSR3 can extract richer features not only in depth but also in breadth. This helps to effectively refine the image quality at different scales. The experimental results on facial datasets show that BPSR3 significantly improved both convergence speed and reconstruction performance. BPSR3 has about 1/4 of the parameters of SR3 but achieves a 50.1% improvement in PSNR, a 19.8% improvement in SSIM, and a 15.4% reduction in FID. Our contribution lies in achieving less time and space consumption and better reconstruction results. In addition, we propose an idea of enhancing the performance of DMs by replacing the U-Net with a better network.

Cascaded face super‐resolution with shape and identity priors

AbstractDespite impressive progress in face super‐resolution (SR), it is an open challenge to reconstruct a reliable SR face that preserves authentic facial characteristics. Here, the problem of super‐resolving low‐resolution (LR) faces to high‐resolution (HR) ones is addressed. To tackle the ill‐posed nature of face SR, the cascaded super‐resolution network (CSRNet) is proposed to utilize shape and identity priors jointly and progressively, the first to explore multiple priors. Specifically, CSRNet adopts a cascaded structure to transform an LR face to HR face progressively via multiple stages. At each stage, CSRNet forces its output face image to match both the shape priors and identity priors extracted from the ground‐truth HR face. The shape priors estimated in one stage are merged into the inputs of its subsequent stage to provide rich information for the face SR. To generate realistic yet discriminative faces, the cascaded super‐resolution generative adversarial network (CSRGAN) is also proposed to incorporate the adversarial loss and identification loss into CSRNet. Extensive experiments on popular benchmarks show that the CSRNet and CSRGAN outperform existing face SR state‐of‐the‐art methods, both quantitatively and qualitatively, and detailed ablation studies show the advantage of this method.

Low-resolution Face Recognition and Sports Training Action Analysis based on Wireless Sensors

Low-resolution Face Recognition and Sports Training Action Analysis based on Wireless Sensors

Demographic Bias in Low-Resolution Deep Face Recognition in the Wild

Face biometrics play a primary role in smart cities, from consumer- to organizational-level applications. This class of technologies has been recently shown to emphasize performance disparities across gender and ethnic groups. Prior work on demographic bias in deep face recognition has tended to focus exclusively on high-resolution images, leaving out low-resolution images captured, for example, by a certain surveillance camera at a distance. Notable reasons for this focus include the lack of low-resolution face image data sets that report user's demographic attributes. Therefore, demographic bias in low-resolution deep face recognition in the wild still remains under-explored. In this paper, we propose a framework for exploring demographic disparities in low-resolution face recognition at a distance. To this end, we devised a deep generative approach to turn high-resolution face images to their low-resolution counterpart. We then trained state-of-the-art face recognition models on different combinations of high- and low-resolution images. Finally, we assessed the model demographic disparities on artificially degraded images from five data sets. Our results show that face images artificially degraded through our generative approach are more realistic than those obtained with other statistical changes. Key disparities across gender and ethnic groups exist and urge timely interventions. Source code: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://cutt.ly/FChLfOC</uri>

Reconstruction Low- Resolution Image Face Using Restricted Boltzmann Machine

Low-resolution (LR) face images are one of the most challenging problems in face recognition (FR) systems. Due to the difficulty of finding the specific features of faces, the accuracy of face recognition is low. To solve this problem, some researchers are using an image reconstruction approach to improve the resolution of their images. In this research, we are trying to use the restricted Boltzmann machine (RBM) to solve the problem. Furthermore, a labelled face in the wild (lfw) database has been used to validate the proposed method. The results of the experiment show that the PSNR and SSIM of the image result are 34.05 dB and 96.8%, respectively.

Joint face completion and super-resolution using multi-scale feature relation learning

Previous research on face restoration often focused on repairing specific types of low-quality facial images such as low-resolution (LR) or occluded facial images. However, in the real world, both the above-mentioned forms of image degradation often coexist. Therefore, it is important to design a model that can repair images that are LR and occluded simultaneously. This paper proposes a multi-scale feature graph generative adversarial network (MFG-GAN) to carry out face restoration in contexts in which both LR and occluded degradation modes coexist, and also to repair images with a single type of degradation. Based on the GAN, the MFG-GAN integrates the graph convolution and feature pyramid networks to restore occluded low-resolution face images to non-occluded high-resolution face images. The MFG-GAN uses a set of customized losses to ensure that high-quality images are generated. In addition, we designed the network in an end-to-end format. We conduct experiments on general face image restoration and facial expression restoration. Experimental results on the public-domain databases show that the proposed approach outperforms state-of-the-art methods in performing face super resolution (up to 4× or 8×) and face completion simultaneously and can recover better facial expression details.

A Composite Network Model for Face Super-Resolution with Multi-Order Head Attention Facial Priors

Face super-resolution (FSR) aims to reconstruct high-resolution face images from low-resolution (LR) ones. Despite the progress made by deep convolutional neural networks (DCNNs) on FSR, convolutions struggle to relate spatially distant concepts and what is more, all image pixels and prior information (e.g., landmarks and facial component heatmaps) are treated equally regardless of importance, causing inaccuracy and decreasing the quality of face image recovery. To address these issues, in this paper we propose a composite network model for FSR with multi-order head attention facial priors. The proposed model contains a face hallucination transformer (FHT)-based network and a multi-order head attention (MOHA)-based DCNN. The FHT-based network can capture long-range dependencies and gradually increase resolution to achieve efficient and effective inference, while the MOHA-based DCNN exploits detailed and two-dimensional information of LR face images. Moreover, the novel generic submodule of the MOHA-based DCNN, namely Multi-Order Head Attention Network, can accurately model the relationship of facial components between spatial and channel dimensions. The proposed composite network model seamlessly integrates the advantages of DCNNs and transformers to super-resolve LR face images. When compared with state-of-the-art FSR methods on public benchmark datasets, the proposed model shows competitive recovery performance.

A new face reconstruction technique for noisy low-resolution images using regression learning

Over the past few decades, there has been a lot of advancement in face hallucination (or super-resolution) technologies, with the position-patch-based locality constrained methods showing the most promising performance. However, these methods have not achieved up to the mark performance with noisy images as they reconstructed the high-resolution images without knowing the actual level of noise in the input LR faces. To this end, this paper proposes a Regression Learning based Locality constrained Representation (RLLcR) technique for noise robust face hallucination. In RLLcR, the Gaussian process regression learning approach and locality-constrained representation approach are integrated together to super resolve the noisy low-resolution faces. Specifically, a Gaussian process regression learning based noise level (GPRNL) prediction model is introduced that predicts the level of noise in the test images. Further, the predicted noise level is incorporated into the objective function of the proposed RLLcR technique to regularize the reconstruction error. These contributions make the proposed method robust to noise. The experiments are performed on FEI and CAS+PEAL publicly available face datasets and some real-world captured face images to demonstrate the performance of the proposed RLLcR technique. The results show improvement of 17%, 15%, 17%, 6%, 72%, 38%, 26%, 11%, and 35% on FEI dataset; 15%, 10%, 8%, 5%, 76%, 20%, 23%, 13%, and 26% on CAS+PEAL dataset in terms of PSNR, SSIM, MS-SSIM, FSIM, IFC, UQI, D-SSIM, NLPD, and BRISQUE respectively over the best compared method reported in the literature. Moreover the visual results on real-world images also reveal the superiority of the proposed RLLcR technique.

Learning Streamed Attention Network from Descriptor Images for Cross-Resolution 3D Face Recognition

In this article, we propose a hybrid framework for cross-resolution 3D face recognition which utilizes a Streamed Attention Network (SAN) that combines handcrafted features with Convolutional Neural Networks (CNNs). It consists of two main stages: first, we process the depth images to extract low-level surface descriptors and derive the corresponding Descriptor Images (DIs), represented as four-channel images. To build the DIs, we propose a variation of the 3D Local Binary Pattern (3DLBP) operator that encodes depth differences using a sigmoid function. Then, we design a CNN that learns from these DIs. The peculiarity of our solution consists in processing each channel of the input image separately, and fusing the contribution of each channel by means of both self- and cross-attention mechanisms. This strategy showed two main advantages over the direct application of Deep-CNN to depth images of the face; on the one hand, the DIs can reduce the diversity between high- and low-resolution data by encoding surface properties that are robust to resolution differences. On the other, it allows a better exploitation of the richer information provided by low-level features, resulting in improved recognition. We evaluated the proposed architecture in a challenging cross-dataset, cross-resolution scenario. To this aim, we first train the network on scanner-resolution 3D data. Next, we utilize the pre-trained network as feature extractor on low-resolution data, where the output of the last fully connected layer is used as face descriptor. Other than standard benchmarks, we also perform experiments on a newly collected dataset of paired high- and low-resolution 3D faces. We use the high-resolution data as gallery, while low-resolution faces are used as probe, allowing us to assess the real gap existing between these two types of data. Extensive experiments on low-resolution 3D face benchmarks show promising results with respect to state-of-the-art methods.

Analysis of Real-Time Face-Verification Methods for Surveillance Applications.

In the last decade, face-recognition and -verification methods based on deep learning have increasingly used deeper and more complex architectures to obtain state-of-the-art (SOTA) accuracy. Hence, these architectures are limited to powerful devices that can handle heavy computational resources. Conversely, lightweight and efficient methods have recently been proposed to achieve real-time performance on limited devices and embedded systems. However, real-time face-verification methods struggle with problems usually solved by their heavy counterparts-for example, illumination changes, occlusions, face rotation, and distance to the subject. These challenges are strongly related to surveillance applications that deal with low-resolution face images under unconstrained conditions. Therefore, this paper compares three SOTA real-time face-verification methods for coping with specific problems in surveillance applications. To this end, we created an evaluation subset from two available datasets consisting of 3000 face images presenting face rotation and low-resolution problems. We defined five groups of face rotation with five levels of resolutions that can appear in common surveillance scenarios. With our evaluation subset, we methodically evaluated the face-verification accuracy of MobileFaceNet, EfficientNet-B0, and GhostNet. Furthermore, we also evaluated them with conventional datasets, such as Cross-Pose LFW and QMUL-SurvFace. When examining the experimental results of the three mentioned datasets, we found that EfficientNet-B0 could deal with both surveillance problems, but MobileFaceNet was better at handling extreme face rotation over 80 degrees.