Face Verification Task Research Articles

Deep Correlation Feature Learning for Face Verification in the Wild

Convolutional neural networks (CNNs) commonly uses the softmax loss function as the supervision signal. In order to enhance the discriminative power of the deeply learned features, this letter proposes a new supervision signal, called correlation loss, for face verification task. Specifically, the correlation loss encourages the large correlation between the deep feature vectors and their corresponding weight vectors in softmax loss. With the joint supervision of softmax loss and correlation loss, the deep correlation feature learning (DCFL) network can learn the deep features with both the interclass separability and the intraclass compactness, which are highly discriminative for face verification. More importantly, by applying the weight vector of softmax function as the class prototype, the proposed correlation loss function is easy to be optimized during the backpropatation of CNN. Finally, the DCFL method achieves 99.55% and 96.06% face verification accuracy using a 64-layer ResNet on the labeled face in-the-Wild (LFW) and you-tube face (YTF) benchmark, respectively.

Convolutional neural network with nonlinear competitive units

Convolutional Neural Network (CNN) has been an important breakthrough in pattern recognition in recent years. Nevertheless, with the increase in complexity, CNN becomes more difficult to train. To alleviate the problem of training difficulties, we propose a novel nonlinear unit, called Nonlinear Competitive Unit (NCU). By comparing the elements from different network layers and selecting the larger signals element-wisely, it can not only strengthen feature propagation but also accelerate the convergence of CNN. This unit can be regarded as a feature fusion method as well as a kind of activation function. We evaluate our NCU-based models for face verification task and visual classification task on four benchmark datasets. The experimental results demonstrate the superior performance of our models over many state-of-the-art methods, which shows the advantage and potential of the NCU in networks.

Joint Bayesian guided metric learning for end-to-end face verification

Abstract In this work, we address the problem of face verification, namely determining whether a pair of face images belongs to the same or different subjects. Previous works often consider solving the problem of face verification in two steps: feature extraction and face recognition, resulting in a fragmented procedure. We argue that these techniques, although working well, fail to explicitly exploit a full end-to-end framework for face verification, which has received much attention and achieved significant improvements recently. In this paper, we propose a novel Joint Bayesian guided metric learning technique for dealing with the face verification task, which well integrates the above two steps of face verification into an end-to-end convolutional neural network (CNN) architecture. In the training stage, an initial neural network, which has the similar architecture with GoogLeNet CNN model, is firstly pre-trained by optimizing classification-based objective functions on the publicly available CASIA WebFace database. Based on constructed face pairs dataset from CASIA WebFace and LFW datasets, we then fine-tune the whole network parameters under the guide of the learned knowledge, which is obtained from the highly successful Joint Bayesian model. This guided learning procedure, which can also be seen as a metric learning technique, can further update network parameters for discriminating face pairs. In the testing process, the outputs by this unified network are discriminated with a threshold value to produce the ultimate prediction for the face verification task. Comprehensive evaluations over the LFW dataset well demonstrate the encouraging face verification performance of our proposed framework.

Large Age-Gap face verification by feature injection in deep networks

This paper introduces a new method for face verification across large age gaps and also a dataset containing variations of age in the wild, the Large Age-Gap (LAG) dataset, with images ranging from child/young to adult/old. The proposed method exploits a deep convolutional neural network (DCNN) pre-trained for the face recognition task on a large dataset and then fine-tuned for the large age-gap face verification task. Fine-tuning is performed in a Siamese architecture using a contrastive loss function. A feature injection layer is introduced to boost verification accuracy, showing the ability of the DCNN to learn a similarity metric leveraging external features. Experimental results on the LAG dataset show that our method is able to outperform the face verification solutions in the state of the art considered.

Discriminant Analysis on Riemannian Manifold of Gaussian Distributions for Face Recognition With Image Sets.

To address the problem of face recognition with image sets, we aim to capture the underlying data distribution in each set and thus facilitate more robust classification. To this end, we represent image set as the Gaussian mixture model (GMM) comprising a number of Gaussian components with prior probabilities and seek to discriminate Gaussian components from different classes. Since in the light of information geometry, the Gaussians lie on a specific Riemannian manifold, this paper presents a method named discriminant analysis on Riemannian manifold of Gaussian distributions (DARG). We investigate several distance metrics between Gaussians and accordingly two discriminative learning frameworks are presented to meet the geometric and statistical characteristics of the specific manifold. The first framework derives a series of provably positive definite probabilistic kernels to embed the manifold to a high-dimensional Hilbert space, where conventional discriminant analysis methods developed in Euclidean space can be applied, and a weighted Kernel discriminant analysis is devised which learns discriminative representation of the Gaussian components in GMMs with their prior probabilities as sample weights. Alternatively, the other framework extends the classical graph embedding method to the manifold by utilizing the distance metrics between Gaussians to construct the adjacency graph, and hence the original manifold is embedded to a lower-dimensional and discriminative target manifold with the geometric structure preserved and the interclass separability maximized. The proposed method is evaluated by face identification and verification tasks on four most challenging and largest databases, YouTube Celebrities, COX, YouTube Face DB, and Point-and-Shoot Challenge, to demonstrate its superiority over the state-of-the-art.To address the problem of face recognition with image sets, we aim to capture the underlying data distribution in each set and thus facilitate more robust classification. To this end, we represent image set as the Gaussian mixture model (GMM) comprising a number of Gaussian components with prior probabilities and seek to discriminate Gaussian components from different classes. Since in the light of information geometry, the Gaussians lie on a specific Riemannian manifold, this paper presents a method named discriminant analysis on Riemannian manifold of Gaussian distributions (DARG). We investigate several distance metrics between Gaussians and accordingly two discriminative learning frameworks are presented to meet the geometric and statistical characteristics of the specific manifold. The first framework derives a series of provably positive definite probabilistic kernels to embed the manifold to a high-dimensional Hilbert space, where conventional discriminant analysis methods developed in Euclidean space can be applied, and a weighted Kernel discriminant analysis is devised which learns discriminative representation of the Gaussian components in GMMs with their prior probabilities as sample weights. Alternatively, the other framework extends the classical graph embedding method to the manifold by utilizing the distance metrics between Gaussians to construct the adjacency graph, and hence the original manifold is embedded to a lower-dimensional and discriminative target manifold with the geometric structure preserved and the interclass separability maximized. The proposed method is evaluated by face identification and verification tasks on four most challenging and largest databases, YouTube Celebrities, COX, YouTube Face DB, and Point-and-Shoot Challenge, to demonstrate its superiority over the state-of-the-art.

Improving CNN Performance Accuracies With Min-Max Objective.

We propose a novel method for improving performance accuracies of convolutional neural network (CNN) without the need to increase the network complexity. We accomplish the goal by applying the proposed Min-Max objective to a layer below the output layer of a CNN model in the course of training. The Min-Max objective explicitly ensures that the feature maps learned by a CNN model have the minimum within-manifold distance for each object manifold and the maximum between-manifold distances among different object manifolds. The Min-Max objective is general and able to be applied to different CNNs with insignificant increases in computation cost. Moreover, an incremental minibatch training procedure is also proposed in conjunction with the Min-Max objective to enable the handling of large-scale training data. Comprehensive experimental evaluations on several benchmark data sets with both the image classification and face verification tasks reveal that employing the proposed Min-Max objective in the training process can remarkably improve performance accuracies of a CNN model in comparison with the same model trained without using this objective.

Bio‐inspired image enhancement derived from a ‘rank order coding’ model

In this study, the authors propose a new method to enhance image information, based on wavelet decomposition and original partial reconstruction of image. This reconstruction called ‘asynchronous reconstruction’ is not carried out in the same way as the usual sequential one. It is based on rank order coding. In fact, while sequential reconstruction is to sum all or a part of the responses obtained for each scale of ‘coarse to fine’ decomposition, asynchronous reconstruction tries to be closer to human brain which uses a limited number of frequency channels. Actually, after wavelet decomposition, responses are sorted from top down for each pixel of the image. Final asynchronous reconstruction for each pixel is obtained by adding a chosen number of wavelet responses, beginning by the maximum response. So, at a given level of reconstruction, the pixel values do not come from the same frequency channels. The interest of this method has been tested on a face verification task using the IV2 biometric database. Stopping criterion for reconstruction can be a constant number of wavelet responses to use, but an adaptive process has been also investigated. Three criteria are explored: standard deviation, entropy and lost edges ratio.

MetricFusion: Generalized metric swarm learning for similarity measure

Learning distance metrics for measuring the similarity between two data points in unsupervised and supervised pattern recognition has been widely studied in unconstrained face verification tasks. Motivated by the fact that enforcing single distance metric learning for verification via an empirical score threshold is not robust in uncontrolled experimental conditions, we therefore propose to obtain a metric swarm by learning local patches alike sub-metrics simultaneously that naturally formulates a generalized metric swarm learning (GMSL) model with a joint similarity score function solved by an efficient alternative optimization algorithm. Further, each sample pair is represented as a similarity vector via the well-learned metric swarm, such that the face verification task becomes a generalized SVM-alike classification problem. Therefore, the verification can be enforced in the represented metric swarm space that can well improve the robustness of verification under irregular data structure. Experiments are preliminarily conducted using several UCI benchmark datasets for solving general classification problem. Further, the face verification experiments on real-world LFW and PubFig datasets demonstrate that our proposed model outperforms several state-of-the-art metric learning methods.

Multi-Directional Multi-Level Dual-Cross Patterns for Robust Face Recognition.

To perform unconstrained face recognition robust to variations in illumination, pose and expression, this paper presents a new scheme to extract "Multi-Directional Multi-Level Dual-Cross Patterns" (MDML-DCPs) from face images. Specifically, the MDML-DCPs scheme exploits the first derivative of Gaussian operator to reduce the impact of differences in illumination and then computes the DCP feature at both the holistic and component levels. DCP is a novel face image descriptor inspired by the unique textural structure of human faces. It is computationally efficient and only doubles the cost of computing local binary patterns, yet is extremely robust to pose and expression variations. MDML-DCPs comprehensively yet efficiently encodes the invariant characteristics of a face image from multiple levels into patterns that are highly discriminative of inter-personal differences but robust to intra-personal variations. Experimental results on the FERET, CAS-PERL-R1, FRGC 2.0, and LFW databases indicate that DCP outperforms the state-of-the-art local descriptors (e.g., LBP, LTP, LPQ, POEM, tLBP, and LGXP) for both face identification and face verification tasks. More impressively, the best performance is achieved on the challenging LFW and FRGC 2.0 databases by deploying MDML-DCPs in a simple recognition scheme.

Distance Metric Learning Using Privileged Information for Face Verification and Person Re-Identification.

In this paper, we propose a new approach to improve face verification and person re-identification in the RGB images by leveraging a set of RGB-D data, in which we have additional depth images in the training data captured using depth cameras such as Kinect. In particular, we extract visual features and depth features from the RGB images and depth images, respectively. As the depth features are available only in the training data, we treat the depth features as privileged information, and we formulate this task as a distance metric learning with privileged information problem. Unlike the traditional face verification and person re-identification tasks that only use visual features, we further employ the extra depth features in the training data to improve the learning of distance metric in the training process. Based on the information-theoretic metric learning (ITML) method, we propose a new formulation called ITML with privileged information (ITML+) for this task. We also present an efficient algorithm based on the cyclic projection method for solving the proposed ITML+ formulation. Extensive experiments on the challenging faces data sets EUROCOM and CurtinFaces for face verification as well as the BIWI RGBD-ID data set for person re-identification demonstrate the effectiveness of our proposed approach.

High-Resolution Face Verification Using Pore-Scale Facial Features

Face recognition methods, which usually represent face images using holistic or local facial features, rely heavily on alignment. Their performances also suffer a severe degradation under variations in expressions or poses, especially when there is one gallery per subject only. With the easy access to high-resolution (HR) face images nowadays, some HR face databases have recently been developed. However, few studies have tackled the use of HR information for face recognition or verification. In this paper, we propose a pose-invariant face-verification method, which is robust to alignment errors, using the HR information based on pore-scale facial features. A new keypoint descriptor, namely, pore-Principal Component Analysis (PCA)-Scale Invariant Feature Transform (PPCASIFT)-adapted from PCA-SIFT-is devised for the extraction of a compact set of distinctive pore-scale facial features. Having matched the pore-scale features of two-face regions, an effective robust-fitting scheme is proposed for the face-verification task. Experiments show that, with one frontal-view gallery only per subject, our proposed method outperforms a number of standard verification methods, and can achieve excellent accuracy even the faces are under large variations in expression and pose.

Learning Multiscale Active Facial Patches for Expression Analysis.

In this paper, we present a new idea to analyze facial expression by exploring some common and specific information among different expressions. Inspired by the observation that only a few facial parts are active in expression disclosure (e.g., around mouth, eye), we try to discover the common and specific patches which are important to discriminate all the expressions and only a particular expression, respectively. A two-stage multitask sparse learning (MTSL) framework is proposed to efficiently locate those discriminative patches. In the first stage MTSL, expression recognition tasks are combined to located common patches. Each of the tasks aims to find dominant patches for each expression. Secondly, two related tasks, facial expression recognition and face verification tasks, are coupled to learn specific facial patches for individual expression. The two-stage patch learning is performed on patches sampled by multiscale strategy. Extensive experiments validate the existence and significance of common and specific patches. Utilizing these learned patches, we achieve superior performances on expression recognition compared to the state-of-the-arts.

Sparsely encoded local descriptor for face verification

A novel Sparsely Encoded Local Descriptor (SELD) is proposed for face verification. Different from traditional hard or soft quantization methods, we exploit linear regression (LR) model with sparsity and non-negativity constraints to extract more discriminative features (i.e. sparse codes) from local image patches sampled pixel-wisely. Sum-pooling is then imposed to integrate all the sparse codes within each block partitioned from the whole face image. Whitened Principal Component Analysis (WPCA) is finally used to suppress noises and reduce the dimensionality of the pooled features, which thus results in the so-called SELD. To validate the proposed method, comprehensive experiments are conducted on face verification task to compare SELD with the existing related methods in terms of three variable component modules: K-means or K-SVD for dictionary learning, hard/soft assignment or regression model for encoding, as well as sum-pooling or max-pooling for pooling. Experimental results show that our method achieves a competitive accuracy compared with the state-of-the-art methods on the challenging Labeled Faces in the Wild (LFW) database.

Covariance descriptor based on bio-inspired features for person re-identification and face verification

Avoiding the use of complicated pre-processing steps such as accurate face and body part segmentation or image normalization, this paper proposes a novel face/person image representation which can properly handle background and illumination variations. Denoted as gBiCov, this representation relies on the combination of Biologically Inspired Features (BIF) and Covariance descriptors [1]. More precisely, gBiCov is obtained by computing and encoding the difference between BIF features at different scales. The distance between two persons can then be efficiently measured by computing the Euclidean distance of their signatures, avoiding some time consuming operations in Riemannian manifold required by the use of Covariance descriptors. In addition, the recently proposed KISSME framework [2] is adopted to learn a metric adapted to the representation. To show the effectiveness of gBiCov, experiments are conducted on three person re-identification tasks (VIPeR, i-LIDS and ETHZ) and one face verification task (LFW), on which competitive results are obtained. As an example, the matching rate at rank 1 on the VIPeR dataset is of 31.11%, improving the best previously published result by more than 10.

Face verification of age separated images under the influence of internal and external factors

In this paper we study the task of face verification of age-separated images with the presence of various internal and external factors. We propose a hierarchical local binary pattern (HLBP) feature descriptor for robust face representation across age. The effective representation by HLBP across minimal age, illumination, and expression variations combined with its hierarchical computation provides a discriminative representation of the face image. The proposed face descriptor is combined with an AdaBoost classification framework to model the face verification task as a two-class problem. Experimental results on the FG-NET and MORPH aging datasets indicate that the performance of the proposed framework is robust with respect to images of both adults and children. A detailed empirical analysis on the effects of internal (age gap, gender, and ethnicity) and external (pose, expressions, facial hair, and glasses) factors in the face verification performance is also studied. The results indicate that the verification accuracy reduces as the age gap between the image pair increases. A quantitative comparison on the effects of gender on verification performance by both humans and the proposed machine learning approach is provided. The analysis indicate that the cues aid humans in verifying image pairs with large age gaps, while it aids machines for all age gaps. However, the cues mislead humans in the case of images of children and extra-personal pairs with large age gaps. Our analyses indicate that the pose and expression variations affect the performance, despite training with such variations, while facial hair and glasses act as discriminative cues. A study on the effects of ethnicity indicate that non-linear algorithms have insignificant effect in performance with the use of both generalized and individual ethnicity models when compared with linear algorithms.

Emulating biological strategies for uncontrolled face recognition

Face recognition technology is of great significance for applications involving national security and crime prevention. Despite enormous progress in this field, machine-based system is still far from the goal of matching the versatility and reliability of human face recognition. In this paper, we show that a simple system designed by emulating biological strategies of human visual system can largely surpass the state-of-the-art performance on uncontrolled face recognition. In particular, the proposed system integrates dual retinal texture and color features for face representation, an incremental robust discriminant model for high level face coding, and a hierarchical cue-fusion method for similarity qualification. We demonstrate the strength of the system on the large-scale face verification task following the evaluation protocol of the Face Recognition Grand Challenge (FRGC) version 2 Experiment 4. The results are surprisingly well: Its modules significantly outperform their state-of-the-art counterparts, such as Gabor image representation, local binary patterns, and enhanced Fisher linear discriminant model. Furthermore, applying the integrated system to the FRGC version 2 Experiment 4, the verification rate at the false acceptance rate of 0.1 percent reaches to 93.12 percent.

Prediction of eigenvalues and regularization of eigenfeatures for human face verification

We present a prediction and regularization strategy for alleviating the conventional problems of LDA and its variants. A procedure is proposed for predicting eigenvalues using few reliable eigenvalues from the range space. Entire eigenspectrum is divided using two control points, however, the effective low-dimensional discriminative vectors are extracted from the whole eigenspace. The estimated eigenvalues are used for regularization of eigenfeatures in the eigenspace. These prediction and regularization enable to perform discriminant evaluation in the full eigenspace. The proposed method is evaluated and compared with eight popular subspace based methods for face verification task. Experimental results on popular face databases show that our method consistently outperforms others.

Face Verification Using Modeled Eigenspectrum

Face verification is different from face identification task. Some traditional subspace methods that work well in face identification may suffer from severe over-fitting problem when applied for the verification task. Conventional discriminative methods such as linear discriminant analysis (LDA) and its variants are highly sensitive to the training data, which hinders them from achieving high verification accuracy. This work proposes an eigenspectrum model that alleviates the over-fitting problems by replacing the unreliable small and zero eigenvalues with the model values. It also enables the discriminant evaluation in the whole space to extract the low dimensional features effectively. The proposed approach is evaluated and compared with 8 popular subspace based methods for a face verification task. Experimental results on three face databases show that the proposed method consistently outperforms others.

A Viewpoint Invariant, Sparsely Registered, Patch Based, Face Verifier

Sparsely registering a face (i.e., locating 2–3 fiducial points) is considered a much easier task than densely registering one; especially with varying viewpoints. Unfortunately, the converse tends to be true for the task of viewpoint-invariant face verification; the more registration points one has the better the performance. In this paper we present a novel approach to viewpoint invariant face verification which we refer to as the “patch-whole” algorithm. The algorithm is able to obtain good verification performance with sparsely registered faces. Good performance is achieved by not assuming any alignment between gallery and probe view faces, but instead trying to learn the joint likelihood functions for faces of similar and dissimilar identities. Generalization is encouraged by factorizing the joint gallery and probe appearance likelihood, for each class, into an ensemble of “patch-whole” likelihoods. We make an additional contribution in this paper by reviewing existing approaches to viewpoint-invariant face verification and demonstrating how most of them fall into one of two categories; namely viewpoint-generative or viewpoint-discriminative. This categorization is instructive as it enables us to compare our “patch-whole” algorithm to other paradigms in viewpoint-invariant face verification and also gives deeper insights into why the algorithm performs so well.

Measuring the performance of face localization systems

The purpose of face localization is to determine the coordinates of a face in a given image. It is a fundamental research area in computer vision because it serves, as a necessary first step in any face processing system, such as automatic face recognition, face tracking or expression analysis. Most of these techniques assume, in general, that the face region has been perfectly localized. Therefore, their performances depend widely on the accuracy of the face localization process. The purpose of this paper is to mainly show that the error made during the localization process may have different impacts on the final application. We first show the influence of localization errors on the face verification task and then empirically demonstrate the problems of current localization performance measures when applied to this task. In order to properly evaluate the performance of a face localization algorithm, we then propose to embed the final application (here face verification) into the performance measuring process. Using two benchmark databases, BANCA and XM2VTS, we proceed by showing empirically that our proposed method to evaluate localization algorithms better matches the final verification performance.