FRGC Database Research Articles

3D Expression-Invariant Face Verification Based on Transfer Learning and Siamese Network for Small Sample Size

Three-dimensional (3D) face recognition has become a trending research direction in both industry and academia. However, traditional facial recognition methods carry high computational costs and face data storage costs. Deep learning has led to a significant improvement in the recognition rate, but small sample sizes represent a new problem. In this paper, we present an expression-invariant 3D face recognition method based on transfer learning and Siamese networks that can resolve the small sample size issue. First, a landmark detection method utilizing the shape index was employed for facial alignment. Then, a convolutional network (CNN) was constructed with transfer learning and trained with the aligned 3D facial data for face recognition, enabling the CNN to recognize faces regardless of facial expressions. Following that, the weighted trained CNN was shared by a Siamese network to build a 3D facial recognition model that can identify faces even with a small sample size. Our experimental results showed that the proposed method reached a recognition rate of 0.977 on the FRGC database, and the network can be used for facial recognition with a single sample.

Face recognition based on dictionary learning and subspace learning

Dictionary learning plays an important role in sparse representation based face recognition. Many dictionary learning algorithms have been successfully applied to face recognition. However, for corrupted data because of noise or face variations (e.g. occlusion and large pose variation), their performances decline due to the disparity between domains. In this paper, we propose a face recognition algorithm based on dictionary learning and subspace learning (DLSL). In DLSL, a new subspace learning algorithm (SL) is proposed by using sparse constraint, low-rank technology and our label relaxation model to reduce the disparity between domains. Meanwhile, we propose a high-performance dictionary learning algorithm (HPDL) by constructing the embedding term, non-local self-similarity term, and time complexity drop term. In the obtained subspace, we use HPDL to classify these mapped test samples. DLSL is compared with other 28 algorithms on FRGC, LFW, CVL, Yale B and AR face databases. Experimental results show that DLSL achieves better performance than those 28 algorithms, including many state-of-the-art algorithms, such as recurrent regression neural network (RRNN), multimodal deep face recognition (MDFR) and projective low-rank representation (PLR).

Face recognition using feature coding and nonlocal constraint-based sparse representation classifier

We propose a face recognition method using distinctive triangle encoding and nonlocal constraint-based sparse representation (TENCSR). With TENCSR, first the pixel values of all images are used as the low-level features. Next, a clustering method is proposed by considering the density distribution of target data, named shared weights support vector data description (SW-SVDD), which makes the obtained decision boundary closer to the optimal. On the basis of SW-SVDD, a more distinctive triangle encoding (MDTE) method is introduced by considering the cluster center information and the size information of cluster, which makes the encoded features more distinctive. Then the high-level features are obtained by encoding those low-level features using MDTE. Meanwhile, a nonlocal constraint-based sparse representation classifier (NC-SRC) is proposed by the biological discovery that dissimilar inputs have dissimilar codes. Finally, those high-level features are classified by the proposed NC-SRC. Experimental results on Georgia Tech, CVL, IMM, FRGC, and AR databases show that our TENCSR outperforms some state-of-the-art algorithms.

Color space construction by optimizing luminance and chrominance components for face recognition

In recent face recognition techniques utilizing the color information, researchers tried to select one conventional color space or learn a color space from the given training data to achieve better performance. RQCr, DCS and ZRG-NII color spaces have gained reputation as effective color spaces in which the face recognition performs better than in the others. However, at the moment, how to construct effective color spaces for face recognition has not been thoroughly studied. In this paper, we propose a color space LuC1C2 based on a framework of constructing effective color spaces for face recognition tasks. It is composed of one luminance component Lu and two chrominance components C1, C2. The luminance component Lu is selected from 4 different luminance candidates by comparing their R,G,B coefficients and color sensor properties. For the two chrominance components C1, C2, the directions of their transform vectors are determined by the discriminant analysis and the covariance analysis in the chrominance subspace of the RGB color space. The magnitudes of their transform vectors are determined by the discriminant values of Lu, C1, C2. Extensive experiments are conducted on 4 benchmark databases to evaluate our proposed color space LuC1C2. The experimental results obtained by using 2 different color features and 3 different dimension reduction methods show that our proposed color space LuC1C2 achieves consistently better face recognition performance than state-of-the-art color spaces on 3 databases. We also show that the proposed color space achieves higher face verification rate than the state of the arts on FRGC database. Furthermore, the face verification performance is improved significantly by combining CNN features with simple raw-pixel features from the proposed LuC1C2 color space on both LFW and FRGC databases.

Face recognition against illuminations using two directional multi-level threshold-LBP and DCT

In this paper, a new approach named as the Two Directional Multi-level Threshold-LBP Fusion (2D---MTLBP-F) is proposed to solve the problem of face recognition against illuminations. The proposed approach utilizes the Threshold Local Binary Pattern (TLBP) in combination with Discrete Cosine Transform (DCT). The utilization of LBP with different thresholds can produce different levels of information, which in turn can be used to improve performance for face recognition against illuminations. First, all images are normalised using a DCT normalisation technique in order to reduce negative effects of noise, blur or illumination. Secondly, the normalised images are transformed into 61 levels of TLBP with thresholds from ?30 to 30 and then the normalised DCT image is fused into these TLBP layers as it contains a different type of information in frequency domain. Thirdly, in the training stage, the 2D---MTLBP-F model is trained by searching for the best combination among these 62 layers (61 TLBP +1 DCT image) based on an idea from two dimensional multiple color fusion (2D---MCF). Fourthly, in testing stage for face recognition, all testing and gallery images are transformed into the 2D---MTLBP-F model, and face recognition is performed using the sparse sensing classifier (SRC). Finally, extensive experimental results on five different databases show that the proposed approach has achieved the highest recognition rates in different lighting conditions as well as in uncontrolled environment for FRGC database. In comparison with TLBP and the recently proposed approach of Multi-Scale Logarithm Difference Edge-maps (MSLDE), the proposed approach also achieves much better results on all used datasets.

A Discriminative Face Geometric Feature-Based Face Recognition

Numerous feature descriptors have been proposed for holistic facial analysis with satisfactory performances. However, few studies have been conducted on how to model the concept of facial beauty for face-related tasks. In addition, existing learning-based face recognition methods suffer from efficiency problems and rely heavily on a sufficient database. In this paper, we develop an efficient method which leverages the perception of human facial beauty for face recognition. Our work is notably different from previous face recognition works in several aspects: (1) we derive a set of facial features based on the fruits of facial attractiveness and beauty analysis research; (2) compared with traditional optimization-based learning methods, we propose simple yet effective C4.5 decision tree as classification model; (3) proposed method is invariant to both gesture variations and facial occlusions, which achieves good performance compared with a series of state-of-the-art methods. More importantly, we propose visualization experiments to verify the discriminative attributes of extracted features. In this paper, we start by preprocessing images in order to enhance image quality. Then, inspired by the facial beauty research, we design a set of new feature descriptors defined as natural feature. The extracted natural feature is embedded into C4.5 decision tree for supervised learning and classification. Extensive experiments are conducted on the AR database, FRGC database and Extended Yale B database. Comprehensive experiments and in-depth analysis verify the effectiveness and competitiveness of the proposed method.

Local generic representation for patch uLBP-based face recognition with single training sample per subject

In this paper, we propose a novel paradigm of Patch uniform Local Binary Patterns (PuLBP) based Local Generic Representation (LGR) for face recognition. Indeed, we introduce a new block in which an uLBP is used to approximate both reference and variation subsets. Thus, we concentrate on the challenging problem of a single sample per person in a gallery set. Particularly, the main problem is whether only one training subject per class is available. One of the novelties of our technique is to generate virtual samples of each subject. The new sample generic image in a gallery set is adopted to produce the intra-personal variations of different individuals. We illustrate the experimental results of our new algorithm on different benchmark databases, including the AR face database, the Extended Yale B face database, the FRGC database and the FEI database.

Expression robust 3D face landmarking using thresholded surface normals

3D face recognition is an increasing popular modality for biometric authentication, for example in the iPhoneX. Landmarking plays a significant role in region based face recognition algorithms. The accuracy and consistency of the landmarking will directly determine the effectiveness of feature extraction and hence the overall recognition performance. While surface normals have been shown to provide high performing features for face recognition, their use in landmarking has not been widely explored. To this end, a new 3D facial landmarking algorithm based on thresholded surface normal maps is proposed, which is applicable to widely used 3D face databases. The benefits of employing surface normals are demonstrated for both facial roll and yaw rotation calibration and nasal landmarks localization. Results on the Bosphorus, FRGC and BU-3DFE databases show that the detected landmarks possess high within-class consistency and accuracy under different expressions. For several key landmarks the performance achieved surpasses that of state-of-the-art techniques and is also training free and computationally efficient. The use of surface normals therefore provides a useful representation of the 3D surface and the proposed landmarking algorithm provides an effective approach to localising the key nasal landmarks.

Face Recognition via Collaborative Representation: Its Discriminant Nature and Superposed Representation.

Collaborative representation methods, such as sparse subspace clustering (SSC) and sparse representation-based classification (SRC), have achieved great success in face clustering and classification by directly utilizing the training images as the dictionary bases. In this paper, we reveal that the superior performance of collaborative representation relies heavily on the sufficiently large class separability of the controlled face datasets such as Extended Yale B. On the uncontrolled or undersampled dataset, however, collaborative representation suffers from the misleading coefficients of the incorrect classes. To address this limitation, inspired by the success of linear discriminant analysis (LDA), we develop a superposed linear representation classifier (SLRC) to cast the recognition problem by representing the test image in term of a superposition of the class centroids and the shared intra-class differences. In spite of its simplicity and approximation, the SLRC largely improves the generalization ability of collaborative representation, and competes well with more sophisticated dictionary learning techniques, on the experiments of AR and FRGC databases. Enforced with the sparsity constraint, SLRC achieves the state-of-the-art performance on FERET database using single sample per person.

Face recognition against occlusions via colour fusion using 2D-MCF model and SRC

In this paper, we propose a new method to tackle the problem of face recognition with occlusions via effectively using colour information. As most of current existing approaches for this problem focus on occlusion detection first as a pre-processing step and then solve recognition problem via occlusion removal, we will develop an effective image representation technique using fused colour information to increase recognition accuracy without occlusion detection. For such purpose, we would first devise a novel local representation for face images because occlusion often appears locally. Technically, we use the recently proposed two dimensional multi-colour fusion (2D-MCF) model in our previous work for image representation, as such model has been proven to be able to achieve very high performance in terms of recognition accuracy. Then, we use the Partitioned sparse sensing recognition (P-SRC) approach for face recognition since this approach is robust for occlusions and does not need any information about the occluded areas as required in most of the existing works. Finally, extensive experiments conducted on four different databases show the superiority of the proposed method over several existing approaches including P-SRC, the Reconstruction based methods and other state of the art methods. The proposed method can improve the face recognition accuracy by up to 24.5%, 3.8%, 25% and 2.86% for the AR database, the Curtin database, the FRGC database and the Busphorus database, respectively.

Illumination invariant single face image recognition under heterogeneous lighting condition

Illumination problem is still a bottleneck of robust face recognition system, which demands extracting illumination invariant features. In this field, existing works only consider the variations caused by lighting direction or magnitude (denoted as homogeneous lighting), but the effect of spectral wavelength is always ignored and thus existing illumination invariant descriptors have its limitation on processing face images under different spectral wavelengths (denoted as heterogeneous lighting). We propose a novel gradient based descriptor, namely Logarithm Gradient Histogram (LGH), which takes the illumination direction, magnitude and the spectral wavelength together into consideration, so that it can handle both homogeneous and heterogeneous lightings. Our proposal contributes in three-folds: (1) we incorporate LMSN-LoG filter to eliminate the lighting effect of each image and extract two illumination invariant components, namely logarithm gradient orientation (LGO) and logarithm gradient magnitude (LGM); (2) we propose an effective post-processing strategy to make our model tolerant to noise and generate a histogram representation to integrate both LGO and LGM; (3) we present solid theoretical analysis on the illumination invariant properties of our proposed descriptors. Extensive experimental results on CMU-PIE, Extended YaleB, FRGC and HFB databases are reported to verify the effectiveness of our proposed model.

Three-dimensional (3D) facial recognition and prediction

This paper provides solution to the problem in identifying humans from their 3D facial characteristics. For this reason, a standard 3D facial recognition system was built and used in this research work. This work proposes two novel fusion schemes where the first one employs a confidence-aided combination approach, and the second one implements a two-level serial integration method. Recognition simulations performed on the 3DRMA and the FRGC databases show that: (1) generic face template-based rigid registration of faces is better than the non-rigid variant, (2) principal curvature directions and surface normal have better discriminative power, (3) representing faces using local patch descriptors can both reduce the feature dimensionality and improve the identification rate, and (4) confidence-assisted fusion rules and serial two-stage fusion schemes have a potential to improve the accuracy when compared to other decision-level fusion rules.

Learning Low-Rank Class-Specific Dictionary and Sparse Intra-Class Variant Dictionary for Face Recognition.

Face recognition is challenging especially when the images from different persons are similar to each other due to variations in illumination, expression, and occlusion. If we have sufficient training images of each person which can span the facial variations of that person under testing conditions, sparse representation based classification (SRC) achieves very promising results. However, in many applications, face recognition often encounters the small sample size problem arising from the small number of available training images for each person. In this paper, we present a novel face recognition framework by utilizing low-rank and sparse error matrix decomposition, and sparse coding techniques (LRSE+SC). Firstly, the low-rank matrix recovery technique is applied to decompose the face images per class into a low-rank matrix and a sparse error matrix. The low-rank matrix of each individual is a class-specific dictionary and it captures the discriminative feature of this individual. The sparse error matrix represents the intra-class variations, such as illumination, expression changes. Secondly, we combine the low-rank part (representative basis) of each person into a supervised dictionary and integrate all the sparse error matrix of each individual into a within-individual variant dictionary which can be applied to represent the possible variations between the testing and training images. Then these two dictionaries are used to code the query image. The within-individual variant dictionary can be shared by all the subjects and only contribute to explain the lighting conditions, expressions, and occlusions of the query image rather than discrimination. At last, a reconstruction-based scheme is adopted for face recognition. Since the within-individual dictionary is introduced, LRSE+SC can handle the problem of the corrupted training data and the situation that not all subjects have enough samples for training. Experimental results show that our method achieves the state-of-the-art results on AR, FERET, FRGC and LFW databases.

Automatic 2.5-D Facial Landmarking and Emotion Annotation for Social Interaction Assistance.

People with low vision, Alzheimer's disease, and autism spectrum disorder experience difficulties in perceiving or interpreting facial expression of emotion in their social lives. Though automatic facial expression recognition (FER) methods on 2-D videos have been extensively investigated, their performance was constrained by challenges in head pose and lighting conditions. The shape information in 3-D facial data can reduce or even overcome these challenges. However, high expenses of 3-D cameras prevent their widespread use. Fortunately, 2.5-D facial data from emerging portable RGB-D cameras provide a good balance for this dilemma. In this paper, we propose an automatic emotion annotation solution on 2.5-D facial data collected from RGB-D cameras. The solution consists of a facial landmarking method and a FER method. Specifically, we propose building a deformable partial face model and fit the model to a 2.5-D face for localizing facial landmarks automatically. In FER, a novel action unit (AU) space-based FER method has been proposed. Facial features are extracted using landmarks and further represented as coordinates in the AU space, which are classified into facial expressions. Evaluated on three publicly accessible facial databases, namely EURECOM, FRGC, and Bosphorus databases, the proposed facial landmarking and expression recognition methods have achieved satisfactory results. Possible real-world applications using our algorithms have also been discussed.

Local gradient‐based illumination invariant face recognition using local phase quantisation and multi‐resolution local binary pattern fusion

A local-based illumination insensitive face recognition algorithm is proposed which is the combination of image normalisation and illumination invariant descriptors. Illumination insensitive representation of image is obtained based on the ratio of gradient amplitude to the original image intensity and partitioned into smaller sub-blocks. Local phase quantisation and multi-scale local binary pattern, extract the sub-regions characteristics. Distance measurements of local nearest neighbour classifiers are fused at the score level to find the best match and decision-level fusion combines the results of two matching techniques. Entropy, class posterior probability and mutual information are utilised as the weights of fusion components. Simulation results on the YaleB, Extended YaleB, AR, Multi-PIE and FRGC databases show the improved performance of the proposed algorithm under severe illumination with low computational complexity and no reconstruction or training requirement.

KinectFaceDB: A Kinect Database for Face Recognition

The recent success of emerging RGB-D cameras such as the Kinect sensor depicts a broad prospect of 3-D data-based computer applications. However, due to the lack of a standard testing database, it is difficult to evaluate how the face recognition technology can benefit from this up-to-date imaging sensor. In order to establish the connection between the Kinect and face recognition research, in this paper, we present the first publicly available face database (i.e., KinectFaceDB 1 ) based on the Kinect sensor. The database consists of different data modalities (well-aligned and processed 2-D, 2.5-D, 3-D, and video-based face data) and multiple facial variations. We conducted benchmark evaluations on the proposed database using standard face recognition methods, and demonstrated the gain in performance when integrating the depth data with the RGB data via score-level fusion. We also compared the 3-D images of Kinect (from the KinectFaceDB) with the traditional high-quality 3-D scans (from the FRGC database) in the context of face biometrics, which reveals the imperative needs of the proposed database for face recognition research. 1 Online at http://rgb-d.eurecom.fr

Face recognition based on curvelets and local binary pattern features via using local property preservation

In this paper, we propose a new feature extraction approach for face recognition based on Curvelet transform and local binary pattern operator. The motivation of this approach is based on two observations. One is that Curvelet transform is a new anisotropic multi-resolution analysis tool, which can effectively represent image edge discontinuities; the other is that local binary pattern operator is one of the best current texture descriptors for face images. As the curvelet features in different frequency bands represent different information of the original image, we extract such features using different methods for different frequency bands. Technically, the lowest frequency band component is processed using the local binary pattern method, and only the medium frequency band components are normalized. And then, we combine them to create a feature set, and use the local preservation projection to reduce its dimension. Finally, we classify the test samples using the nearest neighbor classifier in the reduced space. Extensive experiments on the Yale database, the extended Yale B database, the PIE pose 09 database, and the FRGC database illustrate the effectiveness of the proposed method.

Masquerade attack on transform-based binary-template protection based on perceptron learning

With the increasing deployment of biometric systems, security of the biometric systems has become an essential issue to which serious attention has to be given. To prevent unauthorized access to a biometric system, protection has to be provided to the enrolled biometric templates so that if the database is compromised, the stored information will not enable any adversary to impersonate the victim in gaining an illegal access. In the past decade, transform-based template protection that stores binary one-way-transformed templates (e.g. Biohash) has appeared being one of the benchmark template protection techniques. While the security of such approach lies in the non-invertibility of the transform (e.g. given a transformed binary template, deriving the corresponding face image is infeasible), we will prove in this paper that, irrespective of whether the algorithm of transform-based approach is revealed, a synthetic face image can be constructed from the binary template and the stolen token (storing projection and discretization parameters) to obtain a highly-probable positive authentication response. Our proposed masquerade attack algorithms are mainly composed of a combination of perceptron learning and customized hill climbing algorithms. Experimental results show that our attack algorithms achieve very promising results where the best setting of our attack achieves 100% and 98.3% rank one recognition rates for the CMU PIE and FRGC databases correspondingly when the binarization algorithm (transformation plus discretization) is known; and 85.29% and 46.57% rank one recognition rates for the CMU PIE and FRGC databases correspondingly when the binarization algorithm is unknown.

EM-GPA: Generalized Procrustes analysis with hidden variables for 3D shape modeling

Aligning shapes is essential in many computer vision problems and generalized Procrustes analysis (GPA) is one of the most popular algorithms to align shapes. However, if some of the shape data are missing, GPA cannot be applied. In this paper, we propose EM-GPA, which extends GPA to handle shapes with hidden (missing) variables by using the expectation-maximization (EM) algorithm. For example, 2D shapes can be considered as 3D shapes with missing depth information due to the projection of 3D shapes into the image plane. For a set of 2D shapes, EM-GPA finds scales, rotations and 3D shapes along with their mean and covariance matrix for 3D shape modeling. A distinctive characteristic of EM-GPA is that it does not enforce any rank constraint often appeared in other work and instead uses GPA constraints to resolve the ambiguity in finding scales, rotations, and 3D shapes. The experimental results show that EM-GPA can recover depth information accurately even when the noise level is high and there are a large number of missing variables. By using the images from the FRGC database, we show that EM-GPA can successfully align 2D shapes by taking the missing information into consideration. We also demonstrate that the 3D mean shape and its covariance matrix are accurately estimated. As an application of EM-GPA, we construct a 2D+3D AAM (active appearance model) using the 3D shapes obtained by EM-GPA, and it gives a similar success rate in model fitting compared to the method using real 3D shapes. EM-GPA is not limited to the case of missing depth information, but it can be easily extended to more general cases.

Adaptive discriminant learning for face recognition

Face recognition from Single Sample per Person (SSPP) is extremely challenging because only one sample is available for each person. While many discriminant analysis methods, such as Fisherfaces and its numerous variants, have achieved great success in face recognition, these methods cannot work in this scenario, because more than one sample per person are needed to calculate the within-class scatter matrix. To address this problem, we propose Adaptive Discriminant Analysis (ADA) in which the within-class scatter matrix of each enrolled subject is inferred using his/her single sample, by leveraging a generic set with multiple samples per person. Our method is motivated from the assumption that subjects who look alike to each other generally share similar within-class variations. In ADA, a limited number of neighbors for each single sample are first determined from the generic set by using kNN regression or Lasso regression. Then, the within-class scatter matrix of this single sample is inferred as the weighted average of the within-class scatter matrices of these neighbors based on the arithmetic mean or Riemannian mean. Finally, the optimal ADA projection directions can be computed analytically by using the inferred within-class scatter matrices and the actual between-class scatter matrix. The proposed method is evaluated on three databases including FERET database, FRGC database and a large real-world passport-like face database. The extensive results demonstrate the effectiveness of our ADA when compared with the existing solutions to the SSPP problem.