Face Image Quality Research Articles

Perceptual metric for face image quality with pixel-level interpretability

This paper tackles the shortcomings of image evaluation metrics in evaluating facial image quality. Conventional metrics do neither accurately reflect the unique attributes of facial images nor correspond with human visual perception. To address these issues, we introduce a novel metric designed specifically for faces, utilizing a learning-based adversarial framework. This framework comprises a generator for simulating face restoration and a discriminator for quality evaluation. Drawing inspiration from facial neuroscience studies, our metric emphasizes the importance of primary facial features, acknowledging that minor changes in the eyes, nose, and mouth can significantly impact perception. Another key limitation of existing image evaluation metrics is their focus on numerical values at the image level, without providing insight into how different areas of the image contribute to the overall assessment. Our proposed metric offers interpretability regarding how each region of the image is evaluated. Comprehensive experimental results confirm that our face-specific metric surpasses traditional general image quality assessment metrics for facial images, including both full-reference and no-reference methods. The code and models are available at https://github.com/AIM-SKKU/IFQA.

GAN-Based High-Quality Face-Swapping Composite Network

Face swapping or face replacement is a challenging task that involves transferring a source face to a target face while maintaining the target’s facial motion and expression. Although many studies have made a lot of encouraging progress, we have noticed that most of the current solutions have the problem of blurred images, abnormal features, and unnatural pictures after face swapping. To solve these problems, in this paper, we proposed a composite face-swapping generation network, which includes a face extraction module and a feature fusion generation module. This model retains the original facial expression features, as well as the background and lighting of the image while performing face swapping, making the image more realistic and natural. Compared with other excellent models, our model is more robust in terms of face identity, posture verification, and image quality.

Causal Representation Learning for GAN-Generated Face Image Quality Assessment

Causal Representation Learning for GAN-Generated Face Image Quality Assessment

Video Enhancement to Reduce Reflection and Darkness Caused by the Sunglasses using Fusion Algorithm

The enhancement of video of a person wearing sunglasses to reduce the reflection and darkness is a very challenging task in computer vision applications such as video surveillance. The existence of reflections and darkness caused by sunglasses results in intrusive images. The absence of a clear eye diminishes the visible quality of the complete face image. The eyes under sunglasses are not identified clearly if the reflection and darkness exist in the sunglasses. This paper demonstrates the reduction of adverse artifacts such as reflection and darkness in the eye region caused by sunglasses. The system is carried out with the fusion algorithm which consists of three modules: eyeglass tracking on face images, reduction of reflection, and reduction of darkness through the image enhancement method. The image enhancement method includes a color balance algorithm and a histogram stretching algorithm. Firstly, an automatic glasses presence detection model, based on a Robust Local Binary Pattern identifies the imaging process of the ocular region covered by the sunglasses. Secondly, a non-convex optimization scheme, guided by landmarks on the glasses, effectively reduces reflections through several iterations. The image enhancement method incorporating Color Balance, and Histogram Stretching is used to identify eye regions within sunglasses. The resulting regenerated eye regions within sunglasses exhibit increased brightness, subtle darkness, and minimized reflection. The objective evaluation metrics such as peak signal-to-noise ratio, structural similarity index measure, and logarithmic mean square error are used to measure the strength of the proposed system. Qualitative evaluations are conducted to demonstrate the good quality of eyeglass face images with reduced reflection and darkness.

Vision-language joint representation learning for sketch less facial image retrieval

The traditional sketch-based facial image retrieval (SBFIR) framework assumes that a high-quality facial sketch has been prepared prior to the retrieval task. However, drawing such a sketch requires considerable skills and is time consuming, resulting in limited applicability. Sketch less facial image retrieval (SLFIR) framework aims to break these barriers through human–computer interaction during the sketching process. The primary challenges for the SLFIR problem can be noted that initial sketches (at early sketching) contain only local details and exhibit significant differences among users, resulting in poor performance at early stages and weak generalization abilities in practical testing. In this study, we developed a vision–language pretraining model to align the representation of facial images and their associated semantics. Based on this framework, we proposed a method for learning joint representations by fusing sketches with prior semantics, thereby enriching the information of initial sketches. Specifically, (1) we developed a series of well-designed operations to improve the quality of facial image–text pairs in the LAION-Face dataset; we trained a facial vision–language pretraining (FVLP) model to align the facial image and its semantics at the feature level. (2) subsequently, using FVLP as the backbone, we designed a convolutional attention module to fuse the multiscale features extracted from image encoder. This facilitated the learning of a multimodal representation crucial for the final retrieval process. In experiments, our proposed method achieved state-of-the-art performance at early stages on two public datasets; moreover, it exhibited good generalization capabilities during practical testing. Thus, our method significantly outperforms other baselines in terms of early retrieval performance. Codes are available at: https://github.com/ddw2AIGROUP2CQUPT/FVLP.

Facial recognition for disaster victim identification

Mass disaster events can result in high levels of casualties that need to be identified. Whilst disaster victim identification (DVI) relies on primary identifiers of DNA, fingerprints, and dental, these require ante-mortem data that may not exist or be easily obtainable. Facial recognition technology may be able to assist. Automated facial recognition has advanced considerably and access to ante-mortem facial images are readily available. Facial recognition could therefore be used to expedite the DVI process by narrowing down leads before primary identifiers are made available. This research explores the feasibility of using automated facial recognition technology to support DVI. We evaluated the performance of a commercial-off-the-self facial recognition algorithm on post-mortem images (representing images taken after a mass disaster) against ante-mortem images (representing a database that may exist within agencies who hold face databases for identity documents (such as passports or driver's licenses). We explored facial recognition performance for different operational scenarios, with different levels of face image quality, and by cause of death. Our research is the largest facial recognition evaluation of post-mortem and ante-mortem images to date. We demonstrated that facial recognition technology would be valuable for DVI and that the performance varies by image quality and cause of death. We provide recommendations for future research.

Identity-preserving editing of multiple facial attributes by learning global edit directions and local adjustments

Semantic facial attribute editing using pre-trained Generative Adversarial Networks (GANs) has attracted a great deal of attention and effort from researchers in recent years. Due to the high quality of face images generated by StyleGANs, much work has focused on the StyleGANs’ latent space and the proposed methods for facial image editing. Although these methods have achieved satisfying results for manipulating user-intended attributes, they have not fulfilled the goal of preserving the identity, which is an important challenge. We present ID-Style, a new architecture capable of addressing the problem of identity loss during attribute manipulation. The key components of ID-Style include a Learnable Global Direction (LGD) module, which finds a shared and semi-sparse direction for each attribute, and an Instance-Aware Intensity Predictor (IAIP) network, which finetunes the global direction according to the input instance. Furthermore, we introduce two losses during training to enforce the LGD and IAIP to find semi-sparse semantic directions that preserve the identity of the input instance. Despite reducing the size of the network by roughly 95% as compared to similar state-of-the-art works, ID-Style outperforms baselines by 10% and 7% in identity preserving metric (FRS) and average accuracy of manipulation (mACC), respectively.

Assessing Face Image Quality: A Large-Scale Database and a Transformer Method.

The amount of face images has been witnessing an explosive increase in the last decade, where various distortions inevitably exist on transmitted or stored face images. The distortions lead to visible and undesirable degradation on face images, affecting their quality of experience (QoE). To address this issue, this paper proposes a novel Transformer-based method for quality assessment on face images (named as TransFQA). Specifically, we first establish a large-scale face image quality assessment (FIQA) database, which includes 42,125 face images with diversifying content at different distortion types. Through an extensive crowdsource study, we obtain 712,808 subjective scores, which to the best of our knowledge contribute to the largest database for assessing the quality of face images. Furthermore, by investigating the established database, we comprehensively analyze the impacts of distortion types and facial components (FCs) on the overall image quality. Accordingly, we propose the TransFQA method, in which the FC-guided Transformer network (FT-Net) is developed to integrate the global context, face region and FC detailed features via a new progressive attention mechanism. Then, a distortion-specific prediction network (DP-Net) is designed to weight different distortions and accurately predict final quality scores. Finally, the experiments comprehensively verify that our TransFQA method significantly outperforms other state-of-the-art methods for quality assessment on face images.

High-Quality Synthesized Face Sketch Using Generative Reference Prior

Face sketch synthesis (FSS) is considered as an image-to-image translation problem, where a face sketch is generated from an input face photo. FSS plays a vital role in video/image surveillance-based law enforcement. In this paper, motivated by the recent success of generative adversarial networks (GAN), we consider conditional GAN (cGAN) to approach the problem of face sketch synthesis. However, despite the powerful cGAN model’s ability to generate fine textures, low-quality inputs characterized by the facial sketches drawn by artists cannot offer realistic and faithful details and have unknown degradation due to the drawing process, while high-quality references are inaccessible or even unexistent. In this context, we propose an approach based on Generative Reference Prior (GRP) to improve the synthesized face sketch perception. Our proposed model, that we call cGAN-GRP, leverages diverse and rich priors encapsulated in a pre-trained face GAN for generating high-quality facial sketch synthesis. Extensive experiments on publicly available face databases using facial sketch recognition rate and image quality assessment metrics as criteria demonstrate the effectiveness of our proposed model compared to several state-of-the-art methods.

Low-Light Face Super-resolution via Illumination, Structure, and Texture Associated Representation

Human face captured at night or in dimly lit environments has become a common practice, accompanied by complex low-light and low-resolution degradations. However, the existing face super-resolution (FSR) technologies and derived cascaded schemes are inadequate to recover credible textures. In this paper, we propose a novel approach that decomposes the restoration task into face structural fidelity maintaining and texture consistency learning. The former aims to enhance the quality of face images while improving the structural fidelity, while the latter focuses on eliminating perturbations and artifacts caused by low-light degradation and reconstruction. Based on this, we develop a novel low-light low-resolution face super-resolution framework. Our method consists of two steps: an illumination correction face super-resolution network (IC-FSRNet) for lighting the face and recovering the structural information, and a detail enhancement model (DENet) for improving facial details, thus making them more visually appealing and easier to analyze. As the relighted regions could provide complementary information to boost face super-resolution and vice versa, we introduce the mutual learning to harness the informative components from relighted regions and reconstruction, and achieve the iterative refinement. In addition, DENet equipped with diffusion probabilistic model is built to further improve face image visual quality. Experiments demonstrate that the proposed joint optimization framework achieves significant improvements in reconstruction quality and perceptual quality over existing two-stage sequential solutions. Code is available at https://github.com/wcy-cs/IC-FSRDENet.

An Open-Source Face-Aware Capture System

Poor-quality facial images pose challenges in biometric authentication, especially in passport photo acquisition and recognition. This study proposes a novel and open-source solution to address these issues by introducing a real-time facial image quality analysis utilizing computer vision technology on a low-power single-board computer. We present an open-source complete hardware solution that consists of a Jetson processor, a 16 MP autofocus RGB camera, a custom enclosure, and a touch sensor LCD for user interaction. To ensure the integrity and confidentiality of captured facial data, Advanced Encryption Standard (AES) is used for secure image storage. Using the pilot data collection, the system demonstrated its ability to capture high-quality images, achieving 98.98% accuracy in storing images of acceptable quality. This open-source, readily deployable, secure system offers promising potential for diverse real-time applications such as passport verification, security systems, etc.

Enhancing Face Image Quality: Strategic Patch Selection with Deep Reinforcement Learning and Super-Resolution Boost via RRDB

Enhancing Face Image Quality: Strategic Patch Selection with Deep Reinforcement Learning and Super-Resolution Boost via RRDB

Retouched Face Image Quality Assessment Based on Differential Perception and Textual Prompt

Retouched Face Image Quality Assessment Based on Differential Perception and Textual Prompt

Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model

Going the Extra Mile in Face Image Quality Assessment: A Novel Database and Model

Facial attribute editing method combined with parallel GAN for attribute separation

Facial attribute editing encounters problems of incorrect changes to face regions and artifacts in generated images. We propose a facial attribute editing method combined with parallel GAN for attribute separation. First, the method integrates the U2-net encoder and Trans-GAN decoder as a model encoder to extract and generate facial spatial information effectively. Second, RGB images and semantic mask images are used to train a parallel generator and discriminator respectively. Semantic consistency loss is introduced to ensure that the two branches have consistent semantic output and achieve the effect of convergence in the same direction parallel generator and discriminator. The proposed model, trained on the CelebAMask-HQ original dataset and verified by the CelebA dataset, adopts the separation of the face mask image and the background mask image, to improve the correct rate of face attribute editing. Compared with existing facial attribute editing methods, the proposed method is capable of balancing attribute editing ability and details preservation ability. It can accurately edit the target attribute area and greatly improve the quality of facial images.

Benford’s Law and Perceptual Features for Face Image Quality Assessment

The rapid growth in multimedia, storage systems, and digital computers has resulted in huge repositories of multimedia content and large image datasets in recent years. For instance, biometric databases, which can be used to identify individuals based on fingerprints, facial features, or iris patterns, have gained a lot of attention both from academia and industry. Specifically, face image quality assessment (FIQA) has become a very important part of face recognition systems, since the performance of such systems strongly depends on the quality of input data, such as blur, focus, compression, pose, or illumination. The main contribution of this paper is an analysis of Benford’s law-inspired first digit distribution and perceptual features for FIQA. To be more specific, I investigate the first digit distributions in different domains, such as wavelet or singular values, as quality-aware features for FIQA. My analysis revealed that first digit distributions with perceptual features are able to reach a high performance in the task of FIQA.

Assessment of Face Image Quality using Machine Learning

Assessment of Face Image Quality using Machine Learning

Depth Map Denoising Network and Lightweight Fusion Network for Enhanced 3D Face Recognition

With the increasing availability of consumer depth sensors, 3D face recognition (FR) has attracted more and more attention. However, the data acquired by these sensors are often coarse and noisy, making them impractical to use directly. In this paper, we introduce an innovative Depth map denoising network (DMDNet) based on the Denoising Implicit Image Function (DIIF) to reduce noise and enhance the quality of facial depth images for low-quality 3D FR. After generating clean depth faces using DMDNet, we further design a powerful recognition network called Lightweight Depth and Normal Fusion network (LDNFNet), which incorporates a multi-branch fusion block to learn unique and complementary features between different modalities such as depth and normal images. Comprehensive experiments conducted on four distinct low-quality databases demonstrate the effectiveness and robustness of our proposed methods. Furthermore, when combining DMDNet and LDNFNet, we achieve state-of-the-art results on the Lock3DFace database.

Multi-Task Explainable Quality Networks for Large-Scale Forensic Facial Recognition

Identifying suspects from surveillance footage is a crucial task in forensic investigations, but it is often hindered by the variable conditions of observation and the large amounts of data. Face image quality (FIQ) is a metric that measures the usefulness of a face sample for facial recognition. Existing methods for automated FIQ assessment only provide a scalar value for quality, and do not indicate which factors are causing low quality. Additionally, these methods are computationally expensive, which makes current FIQ assessment methods unsuitable for large numbers of images. To address these issues, we introduce multi-task explainable quality networks (XQNets). XQNets provide both the quality value and the associated facial and environmental attributes, and automatically learn how each attribute contributes to the quality value during the training process. We also propose a dataset-agnostic quality pairing protocol to ensure that sample pairs are balanced across datasets and evaluations are fair. Our experiments on the LFW and SCface benchmarks show that our approach generalizes well across different datasets and outperforms state-of-the-art methods. Our method offers a fast, explainable approach to FIQ assessment, making it suitable for large-scale forensic applications.

Attentive ExFeat based deep generative adversarial network for noise robust face super-resolution

A new noise robust face super-resolution model using an attentive ExFeat-based generative adversarial network is proposed in this paper. The proposed model introduces the Exigent Feature Attention Unit (ExFAU) which consists of an Exigent Feature (ExFeat) block with a spatial attention unit to enhance the visual quality of the generated face images. The ExFAU block assists the model in reducing the noise and extracting the micro and high-level facial features. Further, the ExFeat block is followed by a spatial attention unit to focus on specific facial features. This allows us to give more attention to key face attributes related features and less to the remaining features. The proposed model repeats the ExFAU block to focus on different facial components and enhance them to improve the overall quality of the resultant face images. Experimental outcomes exhibit that the proposed model gains state-of-the-art performance on the standard datasets, namely CelebAHQ, Helen, and LFW face.