BU-3DFE Database Research Articles

Low Rank Graph Regularization Embedding for 2D+3D Facial Expression Recognition

In this paper, a novel low rank graph regularization embedding for 3D facial expression recognition (LRGREFER) approach is proposed, in which the core tensor is utilized to characterize the low-rank attribute among the samples combined with the factor matrices with the graph regularization embedding. At first, a model based on a 4D tensor is constructed from the facial expression data. By Tucker decomposing the constructed 4D tensor, a resulting core tensor and factor matrices in different tensor modes are utilized to characterize the low-rankness among samples. Because of the loss of information during modelling the 4D tensor, the missing data from partly observed facial expression data are recovered by embedding the tensor completion. Finally, the proposed model is handled and solved by adopting the alternating direction method of multipliers (ADMM). Meanwhile, the classification prediction of facial expressions are implemented by Multi-class-SVM. Numerical experiments are conducted on BU-3DFE database. The experiment results have been verified that our proposed approach is more competitive.

Low Rank Tucker Decomposition for 2D+3D Facial Expression Recognition

In this paper, a novel low rank Tucker decomposition for 3D facial expression recognition (LRTDFER) approach is proposed, in which the core tensor is utilized to characterize the low-rank attribute among the samples combined with the factor matrices with the graph regularization embedding. At first, a model based on a 4D tensor is constructed from the facial expression data. By Tucker decomposing the constructed 4D tensor, a resulting core tensor and factor matrices in different tensor modes are utilized to characterize the low-rankness among samples. Because of the loss of information during modelling the 4D tensor, the missing data from partly observed facial expression data are recovered by embedding the tensor completion. Finally, the proposed model is handled and solved by adopting the alternating direction method of multipliers (ADMM). Meanwhile, the classification prediction of facial expressions are implemented by Multi-class-SVM. Numerical experiments are conducted on BU-3DFE database. The experiment results have been verified that our proposed approach is more competitive.

A novel lossless compression framework for facial depth images in expression recognition

With the development of AR and VR, depth images are widely used for facial expression analysis and recognition. To reduce the storage size and save bandwidth, an efficient compression framework is desired. In this paper, we propose a novel lossless compression framework for facial depth images in expression recognition. In the proposed framework, two steps are designed to remove the redundancy in the facial depth images, which are data preparing and bitstream encoding operations. In the data preparing operation, the original image is represented by the same and different parts between the left and right sides. In the bitstream encoding operation, these parts are compressed to get the final bitstream. The proposed framework is implemented and examined on the BU-3DFE Database. Experimental result shows that the proposed technique outperforms existing lossless compression frameworks in terms of compression efficiency, and the average data size is reduced to 25.27% by the proposed framework.

Orthogonalization-Guided Feature Fusion Network for Multimodal 2D+3D Facial Expression Recognition

As 2D and 3D data present different views of the same face, the features extracted from them can be both complementary and redundant. In this paper, we present a novel and efficient orthogonalization-guided feature fusion network, namely OGF <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^2$</tex-math></inline-formula> Net, to fuse the features extracted from 2D and 3D faces for facial expression recognition. While 2D texture maps are fed into a 2D feature extraction pipeline (FE2DNet), the attribute maps generated from 3D data are concatenated as input of the 3D feature extraction pipeline (FE3DNet). The two networks are separately trained at the first stage and frozen in the second stage for late feature fusion, which can well address the unavailability of a large number of 3D+2D face pairs. To reduce the redundancies among features extracted from 2D and 3D streams, we design an orthogonal loss-guided feature fusion network to orthogonalize the features before fusing them. Experimental results show that the proposed method significantly outperforms the state-of-the-art algorithms on both the BU-3DFE and Bosphorus databases. While accuracies as high as 89.05% (P1 protocol) and 89.07% (P2 protocol) are achieved on the BU-3DFE database, an accuracy of 89.28% is achieved on the Bosphorus database. The complexity analysis also suggests that our approach achieves a higher processing speed while simultaneously requiring lower memory costs.

A semantic facial expression intensity descriptor based on information granules

This paper investigates a granular description for facial expression intensity in which characterization of motion-based expression features is presented by a collection of information granules. First, the motion-based features are extracted based on the facial landmarks to encode the expression intensity. Secondly, the semantic concepts are then generated by involving various mechanisms of fuzzy clustering based on the motion-based features. The resulting clusters can be viewed as numeric prototypes of the descriptors. The information granules are formed around the prototypes according to the principle of justifiable granularity, which is the fundamental idea of Granular Computing. The proposed granular descriptor for expression intensity is tested on the BU-3DFE database, and the experimental results illustrate that the proposed descriptors based on information granules not only can characterize the semantics of facial changes caused by expressions, but also can facilitate the semantic estimation of facial expression intensity.

3D Face Recognition Analysis Using Random Forest

Face recognition is an emerging field due to the technological advances in camera hardware and for its application in various fields such as the commercial and security sector. Although the existing works in 3D face recognition perform well, a similar experiment setting across classifiers is hard to find, which includes the Random Forest classifier. The aggregations of the classification from each decision tree are the outcome of Random Forest. This paper presents 3D facial recognition using the Random Forest method using the BU-3DFE database, which consists of basic facial expressions. The work using other classifiers such as Neural Network (NN) and Support Vector Machine (SVM) using a similar experiment setting also presented. As for the results, the Random Forest approach has yield 94.71% of recognition rate, which is an encouraging result compared to NN and SVM. In addition, the experiment also yields that fear expression is unique to each human due to a high confidence rate (82%) of subjects with fear expression. Therefore, a lower chance to be mistakenly recognized someone with a fear expression.

Eliminating other-race effect for multi-ethnic facial expression recognition

It has been noticed that the performance of multi-ethnic facial expression recognition is affected by other-race effect significantly. Though this phenomenon has been noticed by psychologists and computer vision researchers for decades, the mechanism of other-race effect is still unknown and few work has been done to compensate or remove this effect. This work proposes an ICA-based method to eliminate the other-race effect in automatic 3D facial expression recognition. Firstly, the depth features are extracted from 3D local facial patches, and independent component analysis is applied to project the features into a subspace in which the projected features are mutually independent. The ethnic-related features and expression-related features are supposed to be separated in ICA subspace. Hence, ethnic-sensitive features are then determined by an entropy-based feature selection method and discarded to depress their influence on facial expression recognition. The proposed method is evaluated on benchmark BU-3DFE database, and the experimental results reveal that the influence caused by other-race effect can be suppressed effectively with the proposed method.

Unsupervised facial expression recognition using domain adaptation based dictionary learning approach

Abstract Over the past years, dictionary learning (DL) based methods have achieved excellent performance in facial expression recognition (FER), where training and testing data are usually presumed to have the same distributions. But in the practical scenarios, this assumption is often broken, especially when training and testing data come from different databases, a.k.a. the cross-database FER problem. In this paper, we focus on the unsupervised cross-domain FER problem where all the samples in target domain are completely unannotated. To address this problem, we propose an unsupervised domain adaptive dictionary learning (UDADL) model to bridge source domain and target domain by learning a shared dictionary. The encoding of the two domains on this dictionary are constrained to be mutually embedded on each other. To bypass the solution complexity, we borrow an analysis dictionary to seek for approximate solutions as the latent variable to favor sub-solvers to be analyzed. To evaluate the performance of the proposed UDADL model, we conduct extensive experiments on the widely used Multi-PIE and BU-3DFE databases. The experimental results demonstrated that the proposed UDADL method outperforms recent domain adaptation FER methods and achieved the state-of-the-art performance.

An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition

We present a fully automatic multimodal 2D + 3D feature-based facial expression recognition approach and demonstrate its performance on the BU–3DFE database. Our approach combines multi-order gradient-based local texture and shape descriptors in order to achieve efficiency and robustness. First, a large set of fiducial facial landmarks of 2D face images along with their 3D face scans are localized using a novel algorithm namely incremental Parallel Cascade of Linear Regression (iPar–CLR). Then, a novel Histogram of Second Order Gradients (HSOG) based local image descriptor in conjunction with the widely used first-order gradient based SIFT descriptor are used to describe the local texture around each 2D landmark. Similarly, the local geometry around each 3D landmark is described by two novel local shape descriptors constructed using the first-order and the second-order surface differential geometry quantities, i.e., Histogram of mesh Gradients (meshHOG) and Histogram of mesh Shape index (curvature quantization, meshHOS). Finally, the Support Vector Machine (SVM) based recognition results of all 2D and 3D descriptors are fused at both feature-level and score-level to further improve the accuracy. Comprehensive experimental results demonstrate that there exist impressive complementary characteristics between the 2D and 3D descriptors. We use the BU–3DFE benchmark to compare our approach to the state-of-the-art ones. Our multimodal feature-based approach outperforms the others by achieving an average recognition accuracy of 86.32%. Moreover, a good generalization ability is shown on the Bosphorus database.

Robust Head Pose Estimation Using a 3D Morphable Model

Head pose estimation from single 2D images has been considered as an important and challenging research task in computer vision. This paper presents a novel head pose estimation method which utilizes the shape model of the Basel face model and five fiducial points in faces. It adjusts shape deformation according to Laplace distribution to afford the shape variation across different persons. A new matching method based on PSO (particle swarm optimization) algorithm is applied both to reduce the time cost of shape reconstruction and to achieve higher accuracy than traditional optimization methods. In order to objectively evaluate accuracy, we proposed a new way to compute the pose estimation errors. Experiments on the BFM-synthetic database, the BU-3DFE database, the CUbiC FacePix database, the CMU PIE face database, and the CAS-PEAL-R1 database show that the proposed method is robust, accurate, and computationally efficient.

Metodología de remallado basado en la curvatura y orientado a modelos 3D del rostro humano

Remeshing techniques seek to modify a mesh in order to adapt it to an specific application. This work proposes a remeshing methodology specialized in the human face, whose purpose is to reduce the number of faces and vertices that requires the mesh, keeping the characteristics of the human anatomy. Curvature information that highlights the intrinsic anisotropy of natural geometry or geometry of human origin is used to accomplish this. Results were polygonal anisotropic meshes, composed mainly of quadrilaterals, with less than 50% of the points and faces of the initial mesh, that maintain the anatomical features for models of the face in a neutral expression, or expressions of happiness, disgust, fear, angry, surprise, and sadness. The methodology was validated with models from the BU-3DFE database, and the quality of the obtained results were evaluated against the remeshing achieved when a technique of simplification based on quadric error metrics is used

Towards a robust affect recognition: Automatic facial expression recognition in 3D faces

Facial expressions are a powerful tool that communicates a person’s emotional state and subsequently his/her intentions. Compared to 2D face images, 3D face images offer more granular cues that are not available in the 2D images. However, one major setback of 3D faces is that they impose a higher dimensionality than 2D faces. In this paper, we attempt to address this problem by proposing a fully automatic 3D facial expression recognition model that tackles the high dimensionality problem in a twofold solution. First, we transform the 3D faces into the 2D plane using conformal mapping. Second, we propose a Differential Evolution (DE) based optimization algorithm to select the optimal facial feature set and the classifier parameters simultaneously. The optimal features are selected from a pool of Speed Up Robust Features (SURF) descriptors of all the prospective facial points. The proposed model yielded an average recognition accuracy of 79% using the Bosphorus database and 79.36% using the BU-3DFE database. In addition, we exploit the facial muscular movements to enhance the probability estimation (PE) of Support Vector Machine (SVM). Joint application of feature selection with the proposed enhanced PE (EPE) yielded an average recognition accuracy of 84% using the Bosphorus database and 85.81% using the BU-3DFE database, which is statistically significantly better (at p<0.01 and p<0.001, respectively) if compared to the individual exploit of the optimal features only.

Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns

3D facial expression recognition has been greatly promoted for overcoming the inherent drawbacks of 2D facial expression recognition and has achieved superior recognition accuracy to the 2D. In this paper, a novel holistic, full-automatic approach for 3D facial expression recognition is proposed. First, 3D face models are represented in 2D-image-like structure which makes it possible to take advantage of the wealth of 2D methods to analyze 3D models. Then an enhanced facial representation, namely polytypic multi-block local binary patterns (P-MLBP), is proposed. The P-MLBP involves both the feature-based irregular divisions to depict the facial expressions accurately and the fusion of depth and texture information of 3D models to enhance the facial feature. Based on the BU-3DFE database, three kinds of classifiers are employed to conduct 3D facial expression recognition for evaluation. Their experimental results outperform the state of the art and show the effectiveness of P-MLBP for 3D facial expression recognition. Therefore, the proposed strategy is validated for 3D facial expression recognition; and its simplicity opens a promising direction for fully automatic 3D facial expression recognition.

Utilizing 3D flow of points for facial expression recognition

This paper presents an approach to recognize Facial Expressions of different intensities using 3D flow of facial points. 3D flow is the geometrical displacement (in 3D) of a facial point from its position in a neutral face to that in the expressive face. Experiments are performed on 3D face models from the BU-3DFE database. Four different intensities of expressions are used for analyzing the relevance of intensity of the expression for the task of FER. It was observed that high intensity expressions are easier to recognize and there is a need to develop algorithms for recognizing low intensity facial expressions. The proposed features outperform difference of facial distances and 2D optical flow. Performances of two classifiers, SVM and LDA are compared wherein SVM performs better. Feature selection did not prove useful.

Isometric deformation invariant 3D shape recognition

Intra-shape deformations complicate 3D shape recognition and therefore need proper modeling. Thereto, an isometric deformation model is used in this paper. The method proposed does not need explicit point correspondences for the comparison of 3D shapes. The geodesic distance matrix is used as an isometry-invariant shape representation. Two approaches are described to arrive at a sampling order invariant shape descriptor: the histogram of geodesic distance matrix values and the set of largest singular values of the geodesic distance matrix. Shape comparison is performed by comparison of the shape descriptors using the χ2-distance as dissimilarity measure. For object recognition, the results obtained demonstrate the singular value approach to outperform the histogram-based approach, as well as the state-of-the-art multidimensional scaling technique, the ICP baseline algorithm and other isometric deformation modeling methods found in literature. Using the TOSCA database, a rank-1 recognition rate of 100% is obtained for the identification scenario, while the verification experiments are characterized by a 1.58% equal error rate. External validation demonstrates that the singular value approach outperforms all other participants for the non-rigid object retrieval contests in SHREC 2010 as well as SHREC 2011. For 3D face recognition, the rank-1 recognition rate is 61.9% and the equal error rate is 11.8% on the BU-3DFE database. This decreased performance is attributed to the fact that the isometric deformation assumption only holds to a limited extent for facial expressions. This is also demonstrated in this paper.

Feature level analysis for 3D facial expression recognition

3D facial expression recognition has great potential in human computer interaction and intelligent robot systems. In this paper, we propose a two-step approach which combines both the feature selection and the feature fusion techniques to choose more comprehensive and discriminative features for 3D facial expression recognition. In the feature selection stage, we utilize a novel normalized cut-based filter (NCBF) algorithm to select the high relevant and low redundant geometrically localized features (GLF) and surface curvature features (SCF), respectively. Then in the feature fusion stage, PCA is performed on the selected GLF and SCF in order to avoid the curse-of-dimensionality challenge. Finally, the processed GLF and SCF are fused together to capture the most discriminative information in 3D expressional faces. Experiments are carried out on the BU-3DFE database, and the proposed approach outperforms the conventional methods by providing more competitive results.

Shape analysis of local facial patches for 3D facial expression recognition

In this paper we address the problem of 3D facial expression recognition. We propose a local geometric shape analysis of facial surfaces coupled with machine learning techniques for expression classification. A computation of the length of the geodesic path between corresponding patches, using a Riemannian framework, in a shape space provides a quantitative information about their similarities. These measures are then used as inputs to several classification methods. The experimental results demonstrate the effectiveness of the proposed approach. Using multiboosting and support vector machines (SVM) classifiers, we achieved 98.81% and 97.75% recognition average rates, respectively, for recognition of the six prototypical facial expressions on BU-3DFE database. A comparative study using the same experimental setting shows that the suggested approach outperforms previous work.

A deformation model to reduce the effect of expressions in 3D face recognition

In 3D face recognition, most work utilizes the rigid parts of face surfaces for matching to exclude the distortion caused by expressions. However, across a broad range of expressions, the rigid parts may not always be uniform and cover large parts of faces. On the other hand, the non-rigid regions of face surfaces also contain useful information for recognition. In this paper, we include the non-rigid regions besides the rigid parts for 3D face recognition. A deformation model is proposed to deform the non-rigid regions to the shapes that are more similar between intra-personal samples but less similar between inter-personal samples. Together with the rigid regions, the deformed parts make samples more discriminable so that the effect of expressions is reduced. The first part of our model uses the target gradient fields from enrolled samples to depress the distortion of the non-rigid regions. The gradient field works in the differential domain. According to the Poisson equation, a smooth deformed shape can be computed by a linear system. The second part of the model is the definition of a surface property that determines the deformation ability of different face regions. Unlike the target gradient fields that improve the similarity of intra-personal samples, the original topology and surface property can keep inter-personal samples sufficiently dissimilar. Our deformation model can be used to improve existing 3D face recognition methods. Experiments are carried out on FRGC and BU-3DFE databases. There are about 8–10% improvements obtained after applying this deformation model to the baseline ICP method. Compared with other deformation models, the experimental results show that our model has advantages on both recognition performance and computational efficiency.

Manifold‐Based 3D Face Caricature Generation with Individualized Facial Feature Extraction

AbstractCaricature is an interesting art to express exaggerated views of different persons and things through drawing. The face caricature is popular and widely used for different applications. To do this, we have to properly extract unique/specialized features of a person's face. A person's facial feature not only depends on his/her natural appearance, but also the associated expression style. Therefore, we would like to extract the neutural facial features and personal expression style for different applicaions. In this paper, we represent the 3D neutral face models in BU–3DFE database by sparse signal decomposition in the training phase. With this decomposition, the sparse training data can be used for robust linear subspace modeling of public faces. For an input 3D face model, we fit the model and decompose the 3D model geometry into a neutral face and the expression deformation separately. The neutral geomertry can be further decomposed into public face and individualized facial feature. We exaggerate the facial features and the expressions by estimating the probability on the corresponding manifold. The public face, the exaggerated facial features and the exaggerated expression are combined to synthesize a 3D caricature for a 3D face model. The proposed algorithm is automatic and can effectively extract the individualized facial features from an input 3D face model to create 3D face caricature.

Automatic Face Segmentation and Facial Landmark Detection in Range Images

We present a methodology for face segmentation and facial landmark detection in range images. Our goal was to develop an automatic process to be embedded in a face recognition system using only depth information as input. To this end, our segmentation approach combines edge detection, region clustering, and shape analysis to extract the face region, and our landmark detection approach combines surface curvature information and depth relief curves to find the nose and eye landmarks. The experiments were performed using the two available versions of the Face Recognition Grand Challenge database and the BU-3DFE database, in order to validate our proposed methodology and its advantages for 3-D face recognition purposes. We present an analysis regarding the accuracy of our segmentation and landmark detection approaches. Our results were better compared to state-of-the-art works published in the literature. We also performed an evaluation regarding the influence of the segmentation process in our 3-D face recognition system and analyzed the improvements obtained when applying landmark-based techniques to deal with facial expressions.