Heterogeneous Face Images Research Articles

On-the-move heterogeneous face recognition in frequency and spatial domain using sparse representation.

Heterogeneity of a probe image is one of the most complex challenges faced by researchers and implementers of current surveillance systems. This is due to existence of multiple cameras working in different spectral ranges in a single surveillance setup. This paper proposes two different approaches including spatial sparse representations (SSR) and frequency sparse representation (FSR) to recognize on-the-move heterogeneous face images with database of single sample per person (SSPP). SCface database, with five visual and two Infrared (IR) cameras, is taken as a benchmark for experiments, which is further confirmed using CASIA NIR-VIS 2.0 face database with 17580 visual and IR images. Similarity, comparison is performed for different scenarios such as, variation of distances from a camera and variation in sizes of face images and various visual and infrared (IR) modalities. Least square minimization based approach for finding the solution is used to match face images as it makes the recognition process simpler. A side by side comparison of both the proposed approaches with the state-of-the-art, classical, principal component analysis (PCA), kernel fisher analysis (KFA) and coupled kernel embedding (CKE) methods, along with modern low-rank preserving projection via graph regularized reconstruction (LRPP-GRR) method, is also presented. Experimental results suggest that the proposed approaches achieve superior performance.

Robust Cross-domain Pseudo-labeling and Contrastive Learning for Unsupervised Domain Adaptation NIR-VIS Face Recognition.

Near-infrared and visible face recognition (NIR-VIS) is attracting increasing attention because of the need to achieve face recognition in low-light conditions to enable 24-hour secure retrieval. However, annotating identity labels for a large number of heterogeneous face images is time-consuming and expensive, which limits the application of the NIR-VIS face recognition system to larger scale real-world scenarios. In this paper, we attempt to achieve NIR-VIS face recognition in an unsupervised domain adaptation manner. To get rid of the reliance on manual annotations, we propose a novel Robust cross-domain Pseudo-labeling and Contrastive learning (RPC) network which consists of three key components, i.e., NIR cluster-based Pseudo labels Sharing (NPS), Domain-specific cluster Contrastive Learning (DCL) and Inter-domain cluster Contrastive Learning (ICL). Firstly, NPS is presented to generate pseudo labels by exploring robust NIR clusters and sharing reliable label knowledge with VIS domain. Secondly, DCL is designed to learn intra-domain compact yet discriminative representations. Finally, ICL dynamically combines and refines intrinsic identity relationships to guide the instance-level features to learn robust and domain-independent representations. Extensive experiments are conducted to verify an accuracy of over 99% in pseudo label assignment and the advanced performance of RPC network on four mainstream NIR-VIS datasets.

Prepended Domain Transformer: Heterogeneous Face Recognition Without Bells and Whistles

Heterogeneous Face Recognition (HFR) refers to matching face images captured in different domains, such as thermal to visible images (VIS), sketches to visible images, near-infrared to visible, and so on. This is particularly useful in matching visible spectrum images to images captured from other modalities. Though highly useful, HFR is challenging because of the domain gap between the source and target domain. Often, large-scale paired heterogeneous face image datasets are absent, preventing training models specifically for the heterogeneous task. In this work, we propose a surprisingly simple, yet, very effective method for matching face images across different sensing modalities. The core idea of the proposed approach is to add a novel neural network block called Prepended Domain Transformer (PDT) in front of a pre-trained face recognition (FR) model to address the domain gap. Retraining this new block with few paired samples in a contrastive learning setup was enough to achieve state-of-the-art performance in many HFR benchmarks. The PDT blocks can be retrained for several source-target combinations using the proposed general framework. The proposed approach is architecture agnostic, meaning they can be added to any pre-trained FR models. Further, the approach is modular and the new block can be trained with a minimal set of paired samples, making it much easier for practical deployment. The source code and protocols will be made available publicly.

Weighted Feature Histogram of Multi-Scale Local Patch Using Multi-Bit Binary Descriptor for Face Recognition.

Most face recognition methods employ single-bit binary descriptors for face representation. The information from these methods is lost in the process of quantization from real-valued descriptors to binary descriptors, which greatly limits their robustness for face recognition. In this study, we propose a novel weighted feature histogram (WFH) method of multi-scale local patches using multi-bit binary descriptors for face recognition. First, to obtain multi-scale information of the face image, the local patches are extracted using a multi-scale local patch generation (MSLPG) method. Second, with the goal of reducing the quantization information loss of binary descriptors, a novel multi-bit local binary descriptor learning (MBLBDL) method is proposed to extract multi-bit local binary descriptors (MBLBDs). In MBLBDL, a learned mapping matrix and novel multi-bit coding rules are employed to project pixel difference vectors (PDVs) into the MBLBDs in each local patch. Finally, a novel robust weight learning (RWL) method is proposed to learn a set of robust weights for each patch to integrate the MBLBDs into the final face representation. In RWL, a codebook is first constructed by clustering MBLBDs on each local patch to extract a feature histogram. Then, considering that different parts of the face have different degrees of robustness to local changes, a set of weights is learned to concatenate the feature histograms of all local patches into the final representation of a face image. In addition, to further improve the performance for heterogeneous face recognition, a coupled WFH (C-WFH) method is proposed. C-WFH maintains the similarity of the corresponding MBLBDs and feature histograms for a pair of heterogeneous face images by means of a novel coupled feature learning (CFL) method to reduce the modality gap. A series of experiments are conducted on widely used face datasets to analyze the performance of WFH and C-WFH. Extensive experimental results show that WFH and C-WFH outperform state-of-the-art face recognition methods.

Enhancing the Accuracies of Age Estimation With Heterogeneous Databases Using Modified CycleGAN

Age estimation using face images has been widely employed across various fields. Because the characteristics of face images usually vary greatly depending on race, camera type, lighting, and other environmental factors, the recognition ability of untrained heterogeneous face image databases is not accurate by previous methods. Therefore, various attempts have been made where different heterogeneous databases were combined to enable training; however, the training time is extended and diverse environmental variables in databases cannot be sufficiently trained. To address these issues, this study proposes modified cycle-consistent generative adversarial network (CycleGAN) that generates an even distribution of heterogeneous face data, and an age-estimation method that is effective for heterogeneous data based on comparative CNN for age estimation (CCNNAE). In addition, we propose the method of reducing the errors caused by image transformation in modified CycleGAN through adaptive selection between transformed and original face images for the input to CCNNAE, which is based on age similarity between the transformed face image and the original face image. Experiments with two open databases, MORPH and MegaAge databases showed that our method outperformed the state-of-the art methods.

Data Augmentation-Based Joint Learning for Heterogeneous Face Recognition.

Heterogeneous face recognition (HFR) is the process of matching face images captured from different sources. HFR plays an important role in security scenarios. However, HFR remains a challenging problem due to the considerable discrepancies (i.e., shape, style, and color) between cross-modality images. Conventional HFR methods utilize only the information involved in heterogeneous face images, which is not effective because of the substantial differences between heterogeneous face images. To better address this issue, this paper proposes a data augmentation-based joint learning (DA-JL) approach. The proposed method mutually transforms the cross-modality differences by incorporating synthesized images into the learning process. The aggregated data augments the intraclass scale, which provides more discriminative information. However, this method also reduces the interclass diversity (i.e., discriminative information). We develop the DA-JL model to balance this dilemma. Finally, we obtain the similarity score between heterogeneous face image pairs through the log-likelihood ratio. Extensive experiments on a viewed sketch database, forensic sketch database, near-infrared image database, thermal-infrared image database, low-resolution photo database, and image with occlusion database illustrate that the proposed method achieves superior performance in comparison with the state-of-the-art methods.

Sparse graphical representation based discriminant analysis for heterogeneous face recognition

Face images captured in heterogeneous environments, e.g., sketches generated by the artists or composite-generation software, photos taken by common cameras and infrared images captured by corresponding infrared imaging devices, usually subject to large texture (i.e., style) differences. This results in heavily degraded performance of conventional face recognition methods in comparison with the performance on images captured in homogeneous environments. In this paper, we propose a novel sparse graphical representation based discriminant analysis (SGR-DA) approach to address aforementioned face recognition in heterogeneous scenarios. An adaptive sparse graphical representation scheme is designed to represent heterogeneous face images, where a Markov networks model is constructed to generate adaptive sparse vectors. To handle the complex facial structure and further improve the discriminability, a spatial partition-based discriminant analysis framework is presented to refine the adaptive sparse vectors for face matching. We conducted experiments on six commonly used heterogeneous face datasets and experimental results illustrate that our proposed SGR-DA approach achieves superior performance in comparison with state-of-the-art methods.

Wasserstein CNN: Learning Invariant Features for NIR-VIS Face Recognition.

Heterogeneous face recognition (HFR) aims at matching facial images acquired from different sensing modalities with mission-critical applications in forensics, security and commercial sectors. However, HFR presents more challenging issues than traditional face recognition because of the large intra-class variation among heterogeneous face images and the limited availability of training samples of cross-modality face image pairs. This paper proposes the novel Wasserstein convolutional neural network (WCNN) approach for learning invariant features between near-infrared (NIR) and visual (VIS) face images (i.e., NIR-VIS face recognition). The low-level layers of the WCNN are trained with widely available face images in the VIS spectrum, and the high-level layer is divided into three parts: the NIR layer, the VIS layer and the NIR-VIS shared layer. The first two layers aim at learning modality-specific features, and the NIR-VIS shared layer is designed to learn a modality-invariant feature subspace. The Wasserstein distance is introduced into the NIR-VIS shared layer to measure the dissimilarity between heterogeneous feature distributions. W-CNN learning is performed to minimize the Wasserstein distance between the NIR distribution and the VIS distribution for invariant deep feature representations of heterogeneous face images. To avoid the over-fitting problem on small-scale heterogeneous face data, a correlation prior is introduced on the fully-connected WCNN layers to reduce the size of the parameter space. This prior is implemented by a low-rank constraint in an end-to-end network. The joint formulation leads to an alternating minimization for deep feature representation at the training stage and an efficient computation for heterogeneous data at the testing stage. Extensive experiments using three challenging NIR-VIS face recognition databases demonstrate the superiority of the WCNN method over state-of-the-art methods.

Asymmetric Joint Learning for Heterogeneous Face Recognition

Heterogeneous face recognition (HFR) refers to matching a probe face image taken from one modality to face images acquired from another modality. It plays an important role in security scenarios. However, HFR is still a challenging problem due to great discrepancies between cross-modality images. This paper proposes an asymmetric joint learning (AJL) approach to handle this issue. The proposed method transforms the cross-modality differences mutually by incorporating the synthesized images into the learning process which provides more discriminative information. Although the aggregated data would augment the scale of intra-classes, it also reduces the diversity (i.e. discriminative information) for inter-classes. Then, we develop the AJL model to balance this dilemma. Finally, we could obtain the similarity score between two heterogeneous face images through the log-likelihood ratio. Extensive experiments on viewed sketch database, forensic sketch database and near infrared image database illustrate that the proposed AJL-HFR method achieve superior performance in comparison to state-of-the-art methods.

Random sampling for fast face sketch synthesis

Exemplar-based face sketch synthesis plays an important role in both digital entertainment and law enforcement. It generally consists of two parts: neighbor selection and recognition weight representation. In this paper, we proposed a simple but effective method which employs offline random sampling instead of K-NN used in state-of-the-art methods. The proposed random sampling strategy reduces the time consuming for synthesis and has stronger scalability than state-of-the-art methods. In addition, we introduced locality constraint to model the distinct correlations between the test patch and random sampled patches. Extensive experiments on public face sketch databases demonstrate the superiority of the proposed method in comparison to state-of-the-art methods, in terms of both synthesis quality and time consumption. The proposed method could be extended to other heterogeneous face image transformation problems such as face hallucination. We release the source codes of our proposed methods and the evaluation metrics for future study online: http://www.ihitworld.com/RSLCR.html.

Heterogeneous Face Recognition: A Common Encoding Feature Discriminant Approach.

Heterogeneous face recognition is an important, yet challenging problem in face recognition community. It refers to matching a probe face image to a gallery of face images taken from alternate imaging modality. The major challenge of heterogeneous face recognition lies in the great discrepancies between different image modalities. Conventional face feature descriptors, e.g., local binary patterns, histogram of oriented gradients, and scale-invariant feature transform, are mostly designed in a handcrafted way and thus generally fail to extract the common discriminant information from the heterogeneous face images. In this paper, we propose a new feature descriptor called common encoding model for heterogeneous face recognition, which is able to capture common discriminant information, such that the large modality gap can be significantly reduced at the feature extraction stage. Specifically, we turn a face image into an encoded one with the encoding model learned from the training data, where the difference of the encoded heterogeneous face images of the same person can be minimized. Based on the encoded face images, we further develop a discriminant matching method to infer the hidden identity information of the cross-modality face images for enhanced recognition performance. The effectiveness of the proposed approach is demonstrated (on several public-domain face datasets) in two typical heterogeneous face recognition scenarios: matching NIR faces to VIS faces and matching sketches to photographs.

Coupled principal component analysis based face recognition in heterogeneous sensor networks

In this paper, we construct heterogeneous sensor networks (HSN) for face recognition and propose a novel approach named coupled principal component analysis (CPCA) that uses a feature-based representation for heterogeneous face images. We first employ local binary patterns (LBP) to capture the local structure of face images, and then propose CPCA to obtain the global face information. The proposed CPCA could incorporate the information between heterogeneous feature spaces, and therefore it reduces the gap between face images captured from heterogeneous sensors in HSN. Finally, the spare representation is utilized for matching heterogeneous face images. The experimental results demonstrate that the proposed approach achieves better performance than the state-of-the-art approaches.

Head Pose Estimation with Improved Random Regression Forests

Perception of head pose is useful for many face-related tasks such as face recognition, gaze estimation, and emotion analysis. In this paper, we propose a novel random forest based method for estimating head pose angles from single face images. In order to improve the effectiveness of the constructed head pose predictor, we introduce feature weighting and tree screening into the random forest training process. In this way, the features with more discriminative power are more likely to be chosen for constructing trees, and each of the trees in the obtained random forest usually has high pose estimation accuracy, while the diversity or generalization ability of the forest is not deteriorated. The proposed method has been evaluated on four public databases as well as a surveillance dataset collected by ourselves. The results show that the proposed method can achieve state-of-the-art pose estimation accuracy. Moreover, we investigate the impact of pose angle sampling intervals and heterogeneous face images on the effectiveness of trained head pose predictors.

Common feature discriminant analysis for matching infrared face images to optical face images.

In biometrics research and industry, it is critical yet a challenge to match infrared face images to optical face images. The major difficulty lies in the fact that a great discrepancy exists between the infrared face image and corresponding optical face image because they are captured by different devices (optical imaging device and infrared imaging device). This paper presents a new approach called common feature discriminant analysis to reduce this great discrepancy and improve optical-infrared face recognition performance. In this approach, a new learning-based face descriptor is first proposed to extract the common features from heterogeneous face images (infrared face images and optical face images), and an effective matching method is then applied to the resulting features to obtain the final decision. Extensive experiments are conducted on two large and challenging optical-infrared face data sets to show the superiority of our approach over the state-of-the-art.

Learning Discriminant Face Descriptor

Local feature descriptor is an important module for face recognition and those like Gabor and local binary patterns (LBP) have proven effective face descriptors. Traditionally, the form of such local descriptors is predefined in a handcrafted way. In this paper, we propose a method to learn a discriminant face descriptor (DFD) in a data-driven way. The idea is to learn the most discriminant local features that minimize the difference of the features between images of the same person and maximize that between images from different people. In particular, we propose to enhance the discriminative ability of face representation in three aspects. First, the discriminant image filters are learned. Second, the optimal neighborhood sampling strategy is soft determined. Third, the dominant patterns are statistically constructed. Discriminative learning is incorporated to extract effective and robust features. We further apply the proposed method to the heterogeneous (cross-modality) face recognition problem and learn DFD in a coupled way (coupled DFD or C-DFD) to reduce the gap between features of heterogeneous face images to improve the performance of this challenging problem. Extensive experiments on FERET, CAS-PEAL-R1, LFW, and HFB face databases validate the effectiveness of the proposed DFD learning on both homogeneous and heterogeneous face recognition problems. The DFD improves POEM and LQP by about 4.5 percent on LFW database and the C-DFD enhances the heterogeneous face recognition performance of LBP by over 25 percent.

Regularized Discriminative Spectral Regression Method for Heterogeneous Face Matching

Face recognition is confronted with situations in which face images are captured in various modalities, such as the visual modality, the near infrared modality, and the sketch modality. This is known as heterogeneous face recognition. To solve this problem, we propose a new method called discriminative spectral regression (DSR). The DSR maps heterogeneous face images into a common discriminative subspace in which robust classification can be achieved. In the proposed method, the subspace learning problem is transformed into a least squares problem. Different mappings should map heterogeneous images from the same class close to each other, while images from different classes should be separated as far as possible. To realize this, we introduce two novel regularization terms, which reflect the category relationships among data, into the least squares approach. Experiments conducted on two heterogeneous face databases validate the superiority of the proposed method over the previous methods.