Real-world Face Images Research Articles

Masked contrastive graph representation learning for age estimation

Age estimation of face images is a crucial task with various practical applications in areas such as video surveillance and Internet access control. While deep learning-based age estimation methods like Convolutional Neural Networks (CNN), Multilayer Perceptrons (MLP), and Transformers have shown remarkable performance, they have limitations when modeling complex or irregular objects, and often contain redundant information. Although existing graph neural networks (GNNs) can alleviate this issue, they only considers the structural information of the image and ignores the semantic features, thus the feature representation capability of graph nodes is limited. To address these issues, this paper proposes a novel Masked Contrastive Graph Representation Learning (MCGRL) method for accurate age estimation of face images. Our approach leverages CNN to extract semantic features of the image, which are then partitioned into patches that serve as nodes in the graph. We use a masked graph convolutional network (GCN) to derive image-based node representations that capture rich structural information. Finally, we incorporate multiple losses to explore the complementary relationship between structural information and semantic features, which improves the feature representation capability of GCN. Experimental results on real-world face image datasets demonstrate the superiority of our proposed method over other state-of-the-art age estimation approaches. Our code is available at https://github.com/yuntaoshou/MCGRL

Degradation learning and Skip-Transformer for blind face restoration

Blindrestoration of low-quality faces in the real world has advanced rapidly in recent years. The rich and diverse priors encapsulated by pre-trained face GAN have demonstrated their effectiveness in reconstructing high-quality faces from low-quality observations in the real world. However, the modeling of degradation in real-world face images remains poorly understood, affecting the property of generalization of existing methods. Inspired by the success of pre-trained models and transformers in recent years, we propose to solve the problem of blind restoration by jointly exploiting their power for degradation and prior learning, respectively. On the one hand, we train a two-generator architecture for degradation learning to transfer the style of low-quality real-world faces to the high-resolution output of pre-trained StyleGAN. On the other hand, we present a hybrid architecture, called Skip-Transformer (ST), which combines transformer encoder modules with a pre-trained StyleGAN-based decoder using skip layers. Such a hybrid design is innovative in that it represents the first attempt to jointly exploit the global attention mechanism of the transformer and pre-trained StyleGAN-based generative facial priors. We have compared our DL-ST model with the latest three benchmarks for blind image restoration (DFDNet, PSFRGAN, and GFP-GAN). Our experimental results have shown that this work outperforms all other competing methods, both subjectively and objectively (as measured by the Fréchet Inception Distance and NIQE metrics).

Cross-view information interaction and feedback network for face hallucination

Hallucinating a photo-realistic frontal face image from a low-resolution (LR) non-frontal face image is beneficial for a series of face-related applications. However, previous efforts either focus on super-resolving high-resolution (HR) face images from nearly frontal LR counterparts or frontalizing non-frontal HR faces. It is necessary to address all these challenges jointly for real-world face images in unconstrained environment. In this paper, we develop a novel Cross-view Information Interaction and Feedback Network (CVIFNet), which simultaneously handles the non-frontal LR face image super-resolution (SR) and frontalization in a unified framework and interacts them with each other to further improve their performance. Specifically, the CVIFNet is composed of two feedback sub-networks for frontal and profile face images. Considering the reliable correspondence between frontal and non-frontal face images can be crucial and contribute to face hallucination in a different manner, we design a cross-view information interaction module (CVIM) to aggregate HR representations of different views produced by the SR and frontalization processes to generate finer face hallucination results. Besides, since 3D rendered facial priors contain rich hierarchical features, such as low-level (e.g., sharp edge and illumination) and perception level (e.g., identity) information, we design an identity-preserving consistency loss based on 3D rendered facial priors, which can ensure that the high-frequency details of frontal face hallucination result are consistent with the profile. Extensive experiments demonstrate the effectiveness and advancement of CVIFNet.

Clustering by centroid drift and boundary shrinkage

Locating the centers before assigning clustering labels is a traditional routine of clustering methods, which also limits the development of new clustering ideas. In this paper, we achieve the clustering task by firstly identifying the boundary points in the feature space, and then we shrink the boundary points to allocate the un-clustered points. Concretely, we propose a Centroid Drift (CD) metric and a Boundary Shrinkage (BS) strategy to detect boundary points in the feature space and allocate labels for un-clustered points, respectively. Both the CD and BS are closely related to the pre-computed k-nearest neighbor matrix, contributing to the decrease of algorithm parameters. Moreover, the common problems of noise points and non-uniform density distribution of data points in clustering task can also be alleviated with our proposed large value suppression and normalization of k-nearest neighbor distance techniques. The experiments on synthetic datasets, real-world face image datasets and hyperspectral images demonstrate the superiorities of our proposed clustering framework.

Multi-CartoonGAN with Conditional Adaptive Instance-Layer Normalization for Conditional Artistic Face Translation

In CycleGAN, an image-to-image translation architecture was established without the use of paired datasets by employing both adversarial and cycle consistency loss. The success of CycleGAN was followed by numerous studies that proposed new translation models. For example, StarGAN works as a multi-domain translation model based on a single generator–discriminator pair, while U-GAT-IT aims to close the large face-to-anime translation gap by adapting its original normalization to the process. However, constructing robust and conditional translation models requires tradeoffs when the computational costs of training on graphic processing units (GPUs) are considered. This is because, if designers attempt to implement conditional models with complex convolutional neural network (CNN) layers and normalization functions, the GPUs will need to secure large amounts of memory when the model begins training. This study aims to resolve this tradeoff issue via the development of Multi-CartoonGAN, which is an improved CartoonGAN architecture that can output conditional translated images and adapt to large feature gap translations between the source and target domains. To accomplish this, Multi-CartoonGAN reduces the computational cost by using a pretrained VGGNet to calculate the consistency loss instead of reusing the generator. Additionally, we report on the development of the conditional adaptive layer-instance normalization (CAdaLIN) process for use with our model to make it robust to unique feature translations. We performed extensive experiments using Multi-CartoonGAN to translate real-world face images into three different artistic styles: portrait, anime, and caricature. An analysis of the visualized translated images and GPU computation comparison shows that our model is capable of performing translations with unique style features that follow the conditional inputs and at a reduced GPU computational cost during training.

Robust face recognition for occluded real‐world images using constrained probabilistic sparse network

Aiming at the occluded real-world face images across illumination, pose, expression, and resolution variations, a robust face recognition for occluded real-world images using constrained probabilistic sparse network is presented. A constrained probabilistic sparse representation network is constructed to obtain the features of all the training images from a global perspective, and the new network nodes are generated through the random combination of the training images. In the probabilistic sparse representation network, the probabilities of each class of the sparse subspace that the occluded test images individually belong to are defined and calculated. The final classifications of the test images are determined by the joint maximum probability of the network nodes. Meanwhile the second-order gradient constraint is the first introduced in the probabilistic sparse representation network. It is found that the constraint uses the adjacent pixels of the face images to obtain the local texture similarity, and further use the local texture similarity to distinguish the occlusion and non-occlusion parts. Thus the constraint can reduce the influence of the occlusion part on face recognition. Extensive experiments with the 12 existing methods on the five face databases demonstrate that the recognition rate of the proposed method is the best than the non-deep learning methods compared, and the proposed method can obtain nearly the same recognition rate with an advantage of a very less time consumption compared to the state-of-the-art deep learning methods.

Blind face images deblurring with enhancement

Face images deblurring has achieved advanced development; however, existing methods involve high computational cost problems. Furthermore, the recovered face images by current methods have the problems of over-smooth textures, ringing artifacts, and poor details. We consider the problem of face images deblurring as a semantic generation task. In this paper, we propose a generative adversarial network (GAN), which includes a perception-inspired blurry removal generator and a discriminator. The proposed generator reconstructs the latent deblurred image by a U-net based network that contains an enhancement module. Face images are highly structured, and thus can be served as a class-specific prior. Considering this, we propose a perceptual loss function to regularize the recovery of face images, which introduces more clear details and reduces the effects of artifacts. The proposed method has a robust capability of generating realistic face images with pleasant visual effects. Extensive experiments on both synthetic and real-world face images demonstrate that the proposed method is comparable with state-of-the-art methods.

Mixed-noise robust face super-resolution through residual-learning based error suppressed nearest neighbor representation

The conventional face super-resolution (SR) methods are highly sensitive to noise because of signal conversion error, bit error in transmission, heat, and illumination effects in the camera sensor. This paper proposes a new position-patch based face SR method, namely RLENR, which effectively enhances the low-resolution (LR) facial images impaired with heavy mixed (Gaussian and impulse) noise. Firstly, it suppresses the impulse noise (IN) from the LR face image by utilizing the PCA oriented mate face and the matrix having pixel-wise noise details. Subsequently, it minimizes the effect of Gaussian noise (GN) by incorporating the residual-learning to update the LR training set. Meanwhile, it hallucinates the respective LR face using the updated LR training set. Here, the residual means reconstruction error occurred by the GN. It employs the sparse set of similar patches from the training samples to generate the optimal weights for LR patch representation. Thus, the method simultaneously achieves both the sparsity and locality to recover vital information of the face. The results of extensive experiments on standard facial image dataset, real-world face image dataset, and surveillance face images confirm that the RLENR method quantitatively and visually outperforms state-of-the-art face SR methods.

Multi-resolution dictionary learning for face recognition

In recent years, there has been a growing interest in the study of dictionary learning for face recognition. Most of the conventional dictionary learning methods focus only on a single resolution, which ignores the variability of resolutions of real-world face images. In order to address the above issue, this paper proposes a novel multi-resolution dictionary learning method that provides multiple dictionaries each being associated with a resolution. Especially, to enhance the robustness of the model, our method adds a relatively strong constraint to keep the similarity of representations obtained using different dictionaries in the training phase. We compare the proposed method to several state-of-the-art dictionary learning methods by applying this method to multi-resolution face recognition. The experimental results demonstrate that our method outperforms many recently proposed dictionary learning methods. The MATLAB codes of the proposed method will be available at http://www.yongxu.org/lunwen.html.

Face liveness detection using convolutional-features fusion of real and deep network generated face images

Conventionally, classifiers designed for face liveness detection are trained on real-world images, where real-face images and corresponding face presentation attacks (PA) are very much overlapped. However, a little research has been carried out in utilization of the combination of real-world face images and face images generated by deep convolutional neural networks (CNN) for face liveness detection. In this paper, we evaluate the adaptive fusion of convolutional-features learned by convolutional layers from real-world face images and deep CNN generated face images for face liveness detection. Additionally, we propose an adaptive convolutional-features fusion layer that adaptively balance the fusion of convolutional-features of real-world face images and face images generated by deep CNN during training. Our extensive experiments on the state-of-the-art face anti-spoofing databases, i.e., CASIA, OULU and Replay-Attack face anti-spoofing databases with both intra-database and cross-database scenarios indicate promising performance of the proposed method on face liveness detection compared to state-of-the-art methods.

Face Match for Family Reunification

Despite the many advances in face recognition technology, practical face detection and matching for unconstrained images remain challenging. A real-world Face Image Retrieval (FIR) system is described in this paper. It is based on optimally weighted image descriptor ensemble utilized in single-image-per-person (SIPP) approach that works with large unconstrained digital photo collections. The described visual search can be deployed in many applications, e.g. person location in post-disaster scenarios, helping families reunite quicker. It provides efficient means for face detection, matching and annotation, working with images of variable quality, requiring no time-consuming training, yet showing commercial performance levels.

Adaptive Cascade Regression Model For Robust Face Alignment.

Cascade regression is a popular face alignment approach, and it has achieved good performances on the wild databases. However, it depends heavily on local features in estimating reliable landmark locations and therefore suffers from corrupted images, such as images with occlusion, which often exists in real-world face images. In this paper, we present a new adaptive cascade regression model for robust face alignment. In each iteration, the shape-indexed appearance is introduced to estimate the occlusion level of each landmark, and each landmark is then weighted according to its estimated occlusion level. Also, the occlusion levels of the landmarks act as adaptive weights on the shape-indexed features to decrease the noise on the shape-indexed features. At the same time, an exemplar-based shape prior is designed to suppress the influence of local image corruption. Extensive experiments are conducted on the challenging benchmarks, and the experimental results demonstrate that the proposed method achieves better results than the state-of-the-art methods for facial landmark localization and occlusion detection.

Sharpness estimation in facial images by spectrum approximation

This paper presents a novel approach to image sharpness assessment designed primarily for facial images. The approach can be described as holistic analysis of the frequency–amplitude spectrum by means of fitting an approximation model and obtaining the estimate based on the model’s parameters. The proposed method shows better correlation with perceived sharpness than other existing methods both on synthetic tests and on a set of real-world face images. We demonstrate an application of the resulting estimate to enhance the accuracy of a face gender classifier.

Exploring deep gradient information for biometric image feature representation

This paper presents a novel but simple biometric image feature representation method, called exploring deep gradient information (DGI). DGI first captures the local structure of an image by computing the histogram of gradient orientation of each macro-pixel (local patch around the reference pixel). Thus, one image can be decomposed into L sub-images (sub-orientation images) according to the gradient information of each macro-pixel since there are L bins in the local histogram. To enrich the gradient information, we also consider the gradient orientation and magnitude of original image as sub-images. For each sub-image, histogram of oriented gradient (HOG) is used to further explore the gradient orientation information. All HOG features are concatenated into one augmented super-vector. Finally, fisher linear discriminate analysis (FLDA) is applied to obtain the low-dimensional and discriminative feature vector. We evaluated the proposed method on the real-world face image datasets NUST-RWFR, Pubfig and LFW, the PolyU Finger-Knuckle-Print database and the PolyU Palmprint database. Experimental results clearly demonstrate the effectiveness of the proposed DGI compared with state-of-the-art algorithms, e.g., SIFT, HOG, LBP, POEM, LARK and IDLS.

Facial expression recognition on real world face images using intelligent techniques: A survey

Most of the research in facial expression recognition system has been done on the datasets which are acquired in predetermined lab environment. Acquiring real world data is crucial for the benchmarking of algorithms because facial expression recognition system will eventually have to work on the data from real world in an uncertain environment. In the recent years, the local descriptor-based facial expression recognition has been shown to be more robust to pose changes, misalignment and illumination effects. Local Binary Pattern (LBP) and Weber Local Descriptor (WLD) are commonly used local descriptors used to effectively represent local micro texture information of the face. Classification of facial expression images became more challenging due to different norms and gestures used among different races and ethnic groups and the change between different expression classes is very subtle. It necessitates storing not only micro but also macro texture information in case of real world face images to acquire better accuracy results. Traditional local descriptors are not capable to extract the macro texture information. The purpose of the proposed work is to present a novel framework to recognize facial expressions with high accuracy rate. The proposed framework intends to be time and memory efficient, and unlike LBP and WLD it would not only store micro but also macro texture information. Performance of the proposed technique will be measured using both synthetic (i.e. lab-based) and real world face images datasets.

Approximately symmetrical face images for image preprocessing in face recognition and sparse representation based classification

Though most of the faces are axis-symmetrical objects, few real-world face images are axis-symmetrical images. In the past years, there are many studies on face recognition, but only little attention is paid to this issue and few studies to explore and exploit the axis-symmetrical property of faces for face recognition are conducted. In this paper, we take the axis-symmetrical nature of faces into consideration and design a framework to produce approximately axis-symmetrical virtual dictionary for enhancing the accuracy of face recognition. It is noteworthy that the novel algorithm to produce axis-symmetrically virtual face images is mathematically very tractable and quite easy to implement. Extensive experimental results demonstrate the superiority in face recognition of the virtual face images obtained using our method to the original face images. Moreover, experimental results on different databases also show that the proposed method can achieve satisfactory classification accuracy in comparison with state-of-the-art image preprocessing algorithms. The MATLAB code of the proposed method can be available at http://www.yongxu.org/lunwen.html.

M3 CSR: Multi-view, multi-scale and multi-component cascade shape regression

Automatic face alignment is a fundamental step in facial image analysis. However, this problem continues to be challenging due to the large variability of expression, illumination, occlusion, pose, and detection drift in the real-world face images. In this paper, we present a multi-view, multi-scale and multi-component cascade shape regression (M3CSR) model for robust face alignment. Firstly, face view is estimated according to the deformable facial parts for learning view specified CSR, which can decrease the shape variance, alleviate the drift of face detection and accelerate shape convergence. Secondly, multi-scale HoG features are used as the shape-index features to incorporate local structure information implicitly, and a multi-scale optimization strategy is adopted to avoid trapping in local optimum. Finally, a component-based shape refinement process is developed to further improve the performance of face alignment. Extensive experiments on the IBUG dataset and the 300-W challenge dataset demonstrate the superiority of the proposed method over the state-of-the-art methods.

Dual Sparse Constrained Cascade Regression for Robust Face Alignment.

Localizing facial landmarks is a fundamental step in facial image analysis. However, the problem continues to be challenging due to the large variability in expression, illumination, pose, and the existence of occlusions in real-world face images. In this paper, we present a dual sparse constrained cascade regression model for robust face alignment. Instead of using the least-squares method during the training process of regressors, sparse constraint is introduced to select robust features and compress the size of the model. Moreover, sparse shape constraint is incorporated between each cascade regression, and the explicit shape constraints are able to suppress the ambiguity in local features. To improve the model's adaptation to large pose variation, face pose is estimated by five fiducial landmarks located by deep convolutional neuron network, which is used to adaptively design the cascade regression model. To the best of our best knowledge, this is the first attempt to fuse explicit shape constraint (sparse shape constraint) and implicit context information (sparse feature selection) for robust face alignment in the framework of cascade regression. Extensive experiments on nine challenging wild data sets demonstrate the advantages of the proposed method over the state-of-the-art methods.

Face hallucination based on two-dimensional joint learning

In this paper, a face hallucination method based on two-dimensional joint learning is presented. Unlike the existing works on face super-resolution algorithms that first reshape the image or image patch into 1D vector, in our study the spatial construction of the high resolution (HR) and the low resolution (LR) face image are efficiently maintained in the reconstruction procedure. Enlightened by the 1D joint learning approach for image super-resolution, we propose a 2D joint learning algorithm to map the original 2D LR and HR image patch spaces onto a unified feature subspace. Subsequently, the neighbor-embedding (NE) based super-resolution algorithm can be conducted on the unified feature subspace to estimate the reconstruction weights. With these weights, the initial HR facial image can be generated. To refine further the initial HR estimate, the global reconstruction constraint is exploited to improve the quality of reconstruction result. Experiments on the face databases and real-world face images demonstrate the effectiveness of the proposed algorithm.

Low rank driven robust facial landmark regression

Localizing facial landmarks is an essential prerequisite to facial image analysis. However, due to the large variability in expression, illumination, pose and the existence of occlusions in the real-world face images, how to localize facial landmarks more efficiently is still a challenging problem. In this paper, we present a low-rank driven regression model for robust facial landmark localization. Our approach consists of low-rank face frontalization and sparse shape constrained cascade regression steps, which lies on, (1) in terms of the low rank prior of face image, we recover such a low-rank face from its deformed image and the associated deformation despite significant distortion and corruption. Alignment of the recovered frontal face image is more simple and effective. And (2) in terms of the sparse coding of face shape on the shape dictionary learnt from training data, sparse shape constrained cascade regression model is proposed to simultaneously suppress the ambiguity in local features and outlier caused by occlusion, and sparse residual error deviated from low-rank face texture is also utilized to predict the occlusion area. Extensive results on several wild benchmarks such as COFW, LFPW and Helen demonstrate that the proposed method is robust to facial occlusions, pose variations and exaggerated facial expressions.