BU-3DFE Datasets Research Articles

A landmark-free approach for automatic, dense and robust correspondence of 3D faces

Global dense registration of 3D faces commonly prioritizes correspondences of facial landmarks which are fiducial points for the anatomical structures. However, it is not always easy to pre-annotate the landmarks accurately in raw scans of 3D faces. Contrary to the current state-of-the-art in dense 3D face correspondence, we propose a general framework without pre-annotated landmarks, which promotes its robustness and allows the meshes to deform in a uniform manner. The proposed framework includes two stages: first the correspondences are established using a template face; and then we select some well-reconstructed samples to build a prior model and leverage it into the correspondence process of other samples. In both stages, the dense registration is revisited in two perspectives: semantic and topological correspondence. In the latter stage, we further incorporate shape and normal statistics of 3D faces to regularize the correspondence process for more robust results. This provides a feasible way to handle data with noises and occlusions, as well as large deformation caused by facial expressions. Our basic idea is to gradually refine the correspondence of individual points in a way global-to-local. At the same time, we solve the local-to-global deformation based on the refined correspondences. The two processes are alternated, and aided by some confidence checks for each individual points. In the experiments, the proposed method is evaluated both qualitatively and quantitatively on three datasets including two publicly available ones: FRGC v2.0 and BU-3DFE datasets, demonstrating its effectiveness.

Improving multi-view facial expression recognition through two novel texture-based feature representations

Although several automatic computer systems have been proposed to address facial expression recognition problems, the majority of them still fail to cope with some requirements of many practical application scenarios. In this paper, one of the most influential and common issues raised in practical application scenarios when applying automatic facial expression recognition system, head pose variation, is comprehensively explored and investigated. In order to do this, two novel texture feature representations are proposed for implementing multi-view facial expression recognition systems in practical environments. These representations combine the block-based techniques with Local Ternary Pattern-based features, providing a more informative and efficient feature representation of the facial images. In addition, an in-house multi-view facial expression database has been designed and collected to allow us to conduct a detailed research study of the effect of out-of-plane pose angles on the performance of a multi-view facial expression recognition system. Along with the proposed in-house dataset, the proposed system is tested on two well-known facial expression databases, CK+ and BU-3DFE datasets. The obtained results shows that the proposed system outperforms current state-of-the-art 2D facial expression systems in the presence of pose variations.

Multi-View Facial Expression Recognition with Multi-View Facial Expression Light Weight Network

Facial expression recognition for frontal faces has become a well-established research area in the last two decades. However, non-frontal facial expression recognition hasn’t been paid much attention until recently. In this paper, we propose an MVFE-LightNet (Multi-View Facial Expression Light Weight Network) for multi-view facial expression recognition. To this end, we first applied MTCNN for facial detection and alignment and then did preprocessing like normalization and data augmentation. Finally, we put the images into MVFE-LightNet to extract sub-space features of facial expressions with various poses. A depthwise separable residual convolution module architecture was designed to reduce the parameters of the model and lessen the chance of overfitting. Experiments were implemented on Radboud Faces Database and BU-3DFE dataset. We demonstrated that our method could effectively improve the recognition accuracy, and achieved the accuracy of 95.6% and 88.7% respectively for the Radboud and BU-3DFE.

Weighted-fusion feature of MB-LBPUH and HOG for facial expression recognition

Obtaining a useful and discriminative feature for facial expression recognition (FER) is a hot research topic in computer vision. In this paper, we propose a novel facial expression representation for FER. Firstly, we select the appropriate parameter of multi-scale block local binary pattern uniform histogram (MB-LBPUH) operator to filter the facial images for representing the holistic structural features. Then, normalizing the filtered images into a uniform basis reduces the computational complexity and remains the full information. An MB-LBPUH feature and a HOG feature are concatenated to fuse a new feature representation for characterizing facial expressions. At the same time, weighting the MB-LBPUH feature can remove the data unbalance from a fusion feature. The weighted-fusion feature reflects not only global facial expressions structure patterns but also characterizes local expression texture appearance and shape. Finally, we utilize principal component analysis for dimensionality reduction and employ support vector machine to classification. Experimental results demonstrate that the proposed algorithm exhibits superior performance compared with the existing algorithms on JAFFE, CK+, and BU-3DFE datasets.

A Coarse-to-Fine Approach for 3D Facial Landmarking by Using Deep Feature Fusion

Facial landmarking locates the key facial feature points on facial data, which provides not only information on semantic facial structures, but also prior knowledge for other kinds of facial analysis. However, most of the existing works still focus on the 2D facial image which may suffer from lighting condition variations. In order to address this limitation, this paper presents a coarse-to-fine approach to accurately and automatically locate the facial landmarks by using deep feature fusion on 3D facial geometry data. Specifically, the 3D data is converted to 2D attribute maps firstly. Then, the global estimation network is trained to predict facial landmarks roughly by feeding the fused CNN (Convolutional Neural Network) features extracted from facial attribute maps. After that, input the local fused CNN features extracted from the local patch around each landmark estimated previously, and other local models are trained separately to refine the locations. Tested on the Bosphorus and BU-3DFE datasets, the experimental results demonstrated effectiveness and accuracy of the proposed method for locating facial landmarks. Compared with existed methods, our results have achieved state-of-the-art performance.

Nasal Patches and Curves for Expression-Robust 3D Face Recognition.

The potential of the nasal region for expression robust 3D face recognition is thoroughly investigated by a novel five-step algorithm. First, the nose tip location is coarsely detected and the face is segmented, aligned and the nasal region cropped. Then, a very accurate and consistent nasal landmarking algorithm detects seven keypoints on the nasal region. In the third step, a feature extraction algorithm based on the surface normals of Gabor-wavelet filtered depth maps is utilised and, then, a set of spherical patches and curves are localised over the nasal region to provide the feature descriptors. The last step applies a genetic algorithm-based feature selector to detect the most stable patches and curves over different facial expressions. The algorithm provides the highest reported nasal region-based recognition ranks on the FRGC, Bosphorus and BU-3DFE datasets. The results are comparable with, and in many cases better than, many state-of-the-art 3D face recognition algorithms, which use the whole facial domain. The proposed method does not rely on sophisticated alignment or denoising steps, is very robust when only one sample per subject is used in the gallery, and does not require a training step for the landmarking algorithm.

Expression-invariant face recognition using depth and intensity dual-tree complex wavelet transform features

A new multimodal expression-invariant face recognition method is proposed by extracting features of rigid and semirigid regions of the face which are less affected by facial expressions. Dual-tree complex wavelet transform is applied in one decomposition level to extract the desired feature from range and intensity images by transforming the regions into eight subimages, consisting of six band-pass subimages to represent face details and two low-pass subimages to represent face approximates. The support vector machine has been used to classify both feature fusion and score fusion modes. To test the algorithm, BU-3DFE and FRGC v2.0 datasets have been selected. The BU-3DFE dataset was tested by low intensity versus high intensity and high intensity versus low intensity strategies using all expressions in both training and testing stages in different levels. Findings include the best rank-1 identification rate of 99.8% and verification rate of 100% at a 0.1% false acceptance rate. The FRGC v2.0 was tested by the neutral versus non-neutral strategy, which applies images without expression in training and with expression in the testing stage, thereby achieving the best rank-1 identification rate of 93.5% and verification rate of 97.4% at a 0.1% false acceptance rate.

Statistical shape modelling for expression-invariant face analysis and recognition

Paper introduces a 3-D shape representation scheme for automatic face analysis and identification, and demonstrates its invariance to facial expression. The core of this scheme lies on the combination of statistical shape modelling and non-rigid deformation matching. While the former matches 3-D faces with facial expression, the latter provides a low-dimensional feature vector that controls the deformation of model for matching the shape of new input, thereby enabling robust identification of 3-D faces. The proposed scheme is also able to handle the pose variation without large part of missing data. To assist the establishment of dense point correspondences, a modified free-form-deformation based on B-spline warping is applied with the help of extracted landmarks. The hybrid iterative closest point method is introduced for matching the models and new data. The feasibility and effectiveness of the proposed method was investigated using standard publicly available Gavab and BU-3DFE datasets, which contain faces with expression and pose changes. The performance of the system was compared with that of nine benchmark approaches. The experimental results demonstrate that the proposed scheme provides a competitive solution for face recognition.

Local circular patterns for multi-modal facial gender and ethnicity classification

Gender and ethnicity are both key demographic attributes of human beings and they play a very fundamental and important role in automatic machine based face analysis, therefore, there has been increasing attention for face based gender and ethnicity classification in recent years. In this paper, we present an effective and efficient approach on this issue by combining both boosted local texture and shape features extracted from 3D face models, in contrast to the existing ones that only depend on either 2D texture or 3D shape of faces. In order to comprehensively represent the difference between different genders or ethnicities, we propose a novel local descriptor, namely local circular patterns (LCP). LCP improves the widely utilized local binary patterns (LBP) and its variants by replacing the binary quantization with a clustering based one, resulting in higher discriminative power as well as better robustness to noise. Meanwhile the following Adaboost based feature selection finds the most discriminative gender- and race-related features and assigns them with different weights to highlight their importance in classification, which not only further raises the performance but reduces the time and memory cost as well. Experimental results achieved on the FRGC v2.0 and BU-3DFE datasets clearly demonstrate the advantages of the proposed method.

A unified probabilistic framework for automatic 3D facial expression analysis based on a Bayesian belief inference and statistical feature models

Textured 3D face models capture precise facial surfaces along with the associated textures, making it possible for an accurate description of facial activities. In this paper, we present a unified probabilistic framework based on a novel Bayesian Belief Network (BBN) for 3D facial expression and Action Unit (AU) recognition. The proposed BBN performs Bayesian inference based on Statistical Feature Models (SFM) and Gibbs–Boltzmann distribution and feature a hybrid approach in fusing both geometric and appearance features along with morphological ones. When combined with our previously developed morphable partial face model (SFAM), the proposed BBN has the capacity of conducting fully automatic facial expression analysis. We conducted extensive experiments on the two public databases, namely the BU-3DFE dataset and the Bosphorus dataset. When using manually labeled landmarks, the proposed framework achieved an average recognition rate of 94.2% and 85.6% for the 7 and 16AU on face data from the Bosphorus dataset respectively, and 89.2% for the six universal expressions on the BU-3DFE dataset. Using the landmarks automatically located by SFAM, the proposed BBN still achieved an average recognition rate of 84.9% for the six prototypical facial expressions. These experimental results demonstrate the effectiveness of the proposed approach and its robustness in landmark localization errors.

An efficient 3D face recognition approach based on the fusion of novel local low-level features

We present a novel 3D face recognition approach based on low-level geometric features that are collected from the eyes-forehead and the nose regions. These regions are relatively less influenced by the deformations that are caused by facial expressions. The extracted features revealed to be efficient and robust in the presence of facial expressions. A region-based histogram descriptor computed from these features is used to uniquely represent a 3D face. A Support Vector Machine (SVM) is then trained as a classifier based on the proposed histogram descriptors to recognize any test face. In order to combine the contributions of the two facial regions (eyes-forehead and nose), both feature-level and score-level fusion schemes have been tested and compared. The proposed approach has been tested on FRGC v2.0 and BU-3DFE datasets through a number of experiments and a high recognition performance was achieved. Based on the results of “neutral vs. non-neutral” experiment of FRGC v2.0 and “low-intensity vs. high-intensity” experiment of BU-3DFE, the feature-level fusion scheme achieved verification rates of 97.6% and 98.2% at 0.1% False Acceptance Rate (FAR) and identification rates of 95.6% and 97.7% on the two datasets respectively. The experimental results also have shown that the feature-level fusion scheme outperformed the score-level fusion one.

On the simultaneous recognition of identity and expression from BU-3DFE datasets

We propose a new linear model, based on resampling 3D face meshes to convert them to 3D matrices, to recognize identity and expression simultaneously. This contrasts with bilinear models currently used with 3D meshes. The matrices are amenable to algebraic operations for facial data analysis and synthesis. Facial emotion is represented as a linear combination of its identity and expression using principal components extracted from training data as neutral-to-emotion deformations. The linear model is applicable to other mesh data with pose variations after correction using recently available techniques. The proposed approach avoids the problem of correspondence between pairs of person’s neutral and emotion meshes for estimating facial deformations used as features. Identity and expression recognition accuracies, obtained by representing resampled matrices as linear combinations of composite depth-color (gray) PCs, are better than the results in the literature on both simultaneous identity-expression using bilinear models and expression-only recognition using deformable models, facial action codes, distances between pairs of annotated facial points as features and others. The proposed framework can also be used to generate synthetic matrices displaying a wide array of natural and mixed emotions for any chosen identity. A byproduct is the result that second-order deformations as features do not seem to perform as effectively as first-order deformations for identity and expression recognition.