Bosphorus Database Research Articles

A 3D Occlusion Facial Recognition Network Based on a Multi-Feature Combination Threshold

In this work, we propose a 3D occlusion facial recognition network based on a multi-feature combination threshold (MFCT-3DOFRNet). First, we design and extract the depth information of the 3D face point cloud, the elevation, and the azimuth angle of the normal vector as new 3D facially distinctive features, so as to improve the differentiation between 3D faces. Next, we propose a multi-feature combinatorial threshold that will be embedded at the input of the backbone network to implement the removal of occlusion features in each channel image. To enhance the feature extraction capability of the neural network for missing faces, we also introduce a missing face data generation method that enhances the training samples of the network. Finally, we use a Focal-ArcFace loss function to increase the inter-class decision boundaries and improve network performance during the training process. The experimental results show that the method has excellent recognition performance for unoccluded faces and also effectively improves the performance of 3D occlusion face recognition. The average Top-1 recognition rate of the proposed MFCT-3DOFRNet for the Bosphorus database is 99.52%, including 98.94% for occluded faces and 100% for unoccluded faces. For the UMB-DB dataset, the average Top-1 recognition rate is 95.08%, including 93.41% for occluded faces and 100% for unoccluded faces. These 3D face recognition experiments show that the proposed method essentially meets the requirements of high accuracy and good robustness.

3D Facial Landmark Localization for cephalometric analysis.

Cephalometric analysis is an important and routine task in the medical field to assess craniofacial development and to diagnose cranial deformities and midline facial abnormalities. The advance of 3D digital techniques potentiated the development of 3D cephalometry, which includes the localization of cephalometric landmarks in the 3D models. However, manual labeling is still applied, being a tedious and time-consuming task, highly prone to intra/inter-observer variability. In this paper, a framework to automatically locate cephalometric landmarks in 3D facial models is presented. The landmark detector is divided into two stages: (i) creation of 2D maps representative of the 3D model; and (ii) landmarks' detection through a regression convolutional neural network (CNN). In the first step, the 3D facial model is transformed to 2D maps retrieved from 3D shape descriptors. In the second stage, a CNN is used to estimate a probability map for each landmark using the 2D representations as input. The detection method was evaluated in three different datasets of 3D facial models, namely the Texas 3DFR, the BU3DFE, and the Bosphorus databases. An average distance error of 2.3, 3.0, and 3.2 mm were obtained for the landmarks evaluated on each dataset. The obtained results demonstrated the accuracy of the method in different 3D facial datasets with a performance competitive to the state-of-the-art methods, allowing to prove its versability to different 3D models. Clinical Relevance- Overall, the performance of the landmark detector demonstrated its potential to be used for 3D cephalometric analysis.

Reinforcement learning coupled with finite element modeling for facial motion learning

Background and ObjectiveFacial palsy patients or patients with facial transplantation have abnormal facial motion due to altered facial muscle functions and nerve damage. Computer-aided system and physics-based models have been developed to provide objective and quantitative information. However, the predictive capacity of these solutions is still limited to explore the facial motion patterns with emerging properties. The present study aims to couple the reinforcement learning and the finite element modeling for facial motion learning and prediction. MethodsA novel modeling workflow for learning facial motion was developed. A physically-based model of the face within the Artisynth modeling platform was used. Information exchange protocol was proposed to link reinforcement learning and rigid multi-bodies dynamics outcomes. Two reinforcement learning algorithms (deep deterministic policy gradient (DDPG) and Twin-delayed DDPG (TD3)) were used and implemented to drive the simulations of symmetry-oriented and smile movements. Numerical outcomes were compared to experimental observations (Bosphorus database) for evaluation and validation purposes. ResultsAs result, after more than 100 episodes of exploring the environment, the agent starts to learn from previous trials and can find the optimal policy after more than 300 episodes of training. Regarding the symmetry-oriented motion, the muscle excitations predicted by the trained agent help to increase the value of reward from R = -2.06 to R = -0.23, which counts for ∼89% improvement of the symmetry value of the face. For smile-oriented motion, two points at the edge of the mouth move up 0.35 cm, which is within the range of movements estimated from the Bosphorus database (0.4 ± 0.32 cm). ConclusionsThe present study explored the muscle excitation patterns by coupling reinforcement learning with a detailed finite element model of the face. We developed, for the first time, a novel coupling scheme to integrate the finite element simulation into the reinforcement learning process for facial motion learning. As perspectives, this present workflow will be applied for facial palsy and facial transplantation patients to guide and optimize the functional rehabilitation program.

3D face expression recognition with ensemble deep learning exploring congruent features among expressions

AbstractAutomatic and accurate 3D face expression categorization has been a challenging task in many computer vision applications. This article presents a novel approach named 3D face expression recognition with ensemble deep learning (3D‐FER‐EDL). The framework comprises three levels. Each of the first two levels gives preference to a subset of labels, considering the congruent features among expressions. Each 3D expressive face is represented as a CDS features vector, which combines corner, distance, and slope features. CDS feature vector is given as input to the first level learners. Expression probability vectors from the first level learners are jointly given as the input to the second level learners. The probability vectors from level one and two are given as the input to the third level, which is the meta classifier. The experiments are conducted on the Bosphorus database and the CASIA 3D face database to evaluate the effectiveness of the proposed approach.

Enhanced facial expression recognition using 3D point sets and geometric deep learning.

Facial expression recognition plays an essential role in human conversation and human-computer interaction. Previous research studies have recognized facial expressions mainly based on 2D image processing requiring sensitive feature engineering and conventional machine learning approaches. The purpose of the present study was to recognize facial expressions by applying a new class of deep learning called geometric deep learning directly on 3D point cloud data. Two databases (Bosphorus and SIAT-3DFE) were used. The Bosphorus database includes sixty-five subjects with seven basic expressions (i.e., anger, disgust, fearness, happiness, sadness, surprise, and neutral). The SIAT-3DFE database has 150 subjects and 4 basic facial expressions (neutral, happiness, sadness, and surprise). First, preprocessing procedures such as face center cropping, data augmentation, and point cloud denoising were applied on 3D face scans. Then, a geometric deep learning model called PointNet++ was applied. A hyperparameter tuning process was performed to find the optimal model parameters. Finally, the developed model was evaluated using the recognition rate and confusion matrix. The facial expression recognition accuracy on the Bosphorus database was 69.01% for 7 expressions and could reach 85.85% when recognizing five specific expressions (anger, disgust, happiness, surprise, and neutral). The recognition rate was 78.70% with the SIAT-3DFE database. The present study suggested that 3D point cloud could be directly processed for facial expression recognition by using geometric deep learning approach. In perspectives, the developed model will be applied for facial palsy patients to guide and optimize the functional rehabilitation program.

Orthogonalization-Guided Feature Fusion Network for Multimodal 2D+3D Facial Expression Recognition

As 2D and 3D data present different views of the same face, the features extracted from them can be both complementary and redundant. In this paper, we present a novel and efficient orthogonalization-guided feature fusion network, namely OGF <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$^2$</tex-math></inline-formula> Net, to fuse the features extracted from 2D and 3D faces for facial expression recognition. While 2D texture maps are fed into a 2D feature extraction pipeline (FE2DNet), the attribute maps generated from 3D data are concatenated as input of the 3D feature extraction pipeline (FE3DNet). The two networks are separately trained at the first stage and frozen in the second stage for late feature fusion, which can well address the unavailability of a large number of 3D+2D face pairs. To reduce the redundancies among features extracted from 2D and 3D streams, we design an orthogonal loss-guided feature fusion network to orthogonalize the features before fusing them. Experimental results show that the proposed method significantly outperforms the state-of-the-art algorithms on both the BU-3DFE and Bosphorus databases. While accuracies as high as 89.05% (P1 protocol) and 89.07% (P2 protocol) are achieved on the BU-3DFE database, an accuracy of 89.28% is achieved on the Bosphorus database. The complexity analysis also suggests that our approach achieves a higher processing speed while simultaneously requiring lower memory costs.

Powerful processing to three-dimensional facial recognition using triple information

&lt;p&gt;&lt;span&gt;Face Detection plays a crucial role in identifying individuals and criminals in Security, surveillance, and footwork control systems. Face Recognition in the human is superb, and pictures can be easily identified even after years of separation. These abilities also apply to changes in a facial expression such as age, glasses, beard, or little change in the face. This method is based on 150 three-dimensional images using the Bosphorus database of a high range laser scanner in a Bogaziçi University in Turkey. This paper presents powerful processing for face recognition based on a combination of the salient information and features of the face, such as eyes and nose, for the detection of three-dimensional figures identified through analysis of surface curvature. The Trinity of the nose and two eyes were selected for applying principal component analysis algorithm and support vector machine to revealing and classification the difference between face and non-face. The results with different facial expressions and extracted from different angles have indicated the efficiency of our powerful processing.&lt;/span&gt;&lt;/p&gt;

3D face reconstruction from mugshots: Application to arbitrary view face recognition

Mugshots while routinely acquired by law enforcement agencies are under utilized by automated face recognition systems. In this paper, we propose a regression based approach to reconstruct textured full 3D face models from multi-view mugshot images. Using landmarks from the input frontal and profile mugshots of a subject, our method reconstructs his⧹her 3D face shape via either linear or nonlinear regressors. The texture of the mugshot images is mapped to the reconstructed 3D face shape via an efficient seamless texture recovery scheme. Compared with existing 3D face reconstruction methods, the proposed method more effectively utilizes the three-view mugshot face images collected during booking. The reconstructed 3D faces are used to generate realistic multi-view face images to enlarge the gallery and facilitate arbitrary view face recognition. Evaluation experiments have been done on BFM and Bosphorus databases in terms of reconstruction accuracy, and on Multi-PIE and Color FERET databases in terms of recognition accuracy. The results show that the proposed method can reduce the 3D face reconstruction error of the best competitive method from 2.31 mm to 1.88 mm, and improve the recognition accuracy of state-of-the-art deep learning based face matchers by as much as ~4% on Multi-PIE and ~2% on Color FERET despite the high baseline set by them.

A multi-scale three-dimensional face recognition approach with sparse representation-based classifier and fusion of local covariance descriptors

In this paper, an efficient multi-scale hybrid approach is proposed to tackle two main problems in three-dimensional (3D) face recognition, namely the singularity of scale features representation and underexplored locality in dictionary learning. The multi-scale features space representation is developed based on the new 3D faces generated by the Gaussian filter. The locality-sensitive Riemannian sparse representation-based classifier is also constructed to accurately recognize faces with various expressions, poses and occlusions. Two sets of face recognition experiment, one that includes expression variations, and the another that includes pose and occlusion variations, are conducted to compare the performance of the proposed approach against other benchmark 3D face recognition algorithms. The recognition accuracies of the proposed algorithm to both Neutral vs. Neutral achieved on Face Recognition Grand Challenge (FRGC) v2.0 database and Bosphorus database are 100%.

Human Identification Using Selected Features From Finger Geometric Profiles

A finger biometric system at an unconstrained environment is presented in this paper. A technique for hand image normalization is implemented at the preprocessing stage that decomposes the main hand contour into finger-level shape representation. This normalization technique follows subtraction of transformed binary image from binary hand contour image to generate the left-side of finger profiles (LSFPs). Then, XOR is applied to LSFP image and hand contour image to produce the right side of finger profiles. During feature extraction, initially, 30 geometric features are computed from every normalized finger. The rank-based forward–backward greedy algorithm is followed to select relevant features and to enhance classification accuracy. Two different subsets of features containing 9 and 12 discriminative features per finger are selected for two separate experimentations those use the ${k}$ -nearest neighbor and the random forest (RF) for classification on the Bosphorus hand database. The experiments with the selected features of four fingers except the thumb have obtained improved performances compared to features extracted from five fingers and also other existing methods evaluated on the Bosphorus database. The best identification accuracies of 96.56% and 95.92% using the RF classifier have been achieved for the right- and left-hand images of 638 subjects, respectively. An equal error rate of 0.078 is obtained for both types of the hand images.

Questionnaires or Inner Feelings: Who Measures the Engagement Better?

This work proposes an innovative method for evaluating users’ engagement, combining the User Engagement Scale (UES) questionnaire and a facial expression recognition (FER) system, active research topics of increasing interest in the human–computer interaction domain (HCI). The subject of the study is a 3D simulator that reproduces a virtual FabLab in which users can approach and learn 3D modeling software and 3D printing. During the interaction with the virtual environment, a structured-light camera acquires the face of the participant in real-time, to catch its spontaneous reactions and compare them with the answers to the UES closed-ended questions. FER methods allow overcoming some intrinsic limits in the adoption of questioning methods, such as the non-sincerity of the interviewees and the lack of correspondence with facial expressions and body language. A convolutional neural network (CNN) has been trained on the Bosphorus database (DB) to perform expression recognition and the classification of the video frames in three classes of engagement (deactivation, average activation, and activation) according to the model of emotion developed by Russell. The results show that the two methodologies can be integrated to evaluate user engagement, to combine weighted answers and spontaneous reactions and to increase knowledge for the design of the new product or service.

Bimodal person recognition using dorsal-vein and finger-vein images

Abstract Nowadays, human recognition with biometric characteristic have been investigated in many researchers. However, traditional biometric methods are not always robust against counterfeit and spoof attacks. The challenge of this work is to develop a bimodal biometric system, based on more reliable and robust characteristics against counterfeiting. Indeed, biometric system based on the venous network of the hand gives higher recognition rate compared to other systems. Furthermore, the fusion of two prints minimums the error rate and counterfeits. In our biometric system, the most challenging phase is the feature extraction step. For this, we propose a new feature extraction approach based on the concatenation of the pyramid of Difference of Gaussian (DoG) and Local Line Binary Patterns (LLBP) histograms (H-DoG_LLBP). To evaluate the proposed system, we opted for Support Vector Machine (SVM) and Artificial Neural Network (ANN) classifiers. The experimental study is based on the dorsal hand vein BOSPHORUS database and finger vein MMCBNU_6000 database. Our system presents an area under curve (AUC) equal to 0.99, and 0.001 mean square error (MSE) with ANN, and 0.0042 equal error rate (EER) with SVM.

Perspective Morphometric Criteria for Facial Beauty and Proportion Assessment

Common sense usually considers the assessment of female human attractiveness to be subjective. Nevertheless, in the past decades, several studies and experiments showed that an objective component in beauty assessment exists and can be strictly related, even if it does not match, with proportions of features. Proportions can be studied through analysis of the face, which relies on landmarks, i.e., specific points on the facial surface, which are shared by everyone, and measurements between them. In this work, several measures have been gathered from studies in the literature considering datasets of beautiful women to build a set of measures that can be defined as suggestive of female attractiveness. The resulting set consists of 29 measures applied to a public dataset, the Bosphorus database, whose faces have been both analyzed by the developed methodology based on the expanded set of measures and judged by human observers. Results show that the set of chosen measures is significant in terms of attractiveness evaluation, confirming the key role of proportions in beauty assessment; furthermore, the sorting of identified measures has been performed to identify the most significant canons involved in the evaluation.

Sparse ICP With Resampling and Denoising for 3D Face Verification

Three-dimensional face recognition has shown its potential to obtain higher recognition accuracy than 2D methods. Among numerous face recognition methods, registration of two faces is comparatively intuitive. We propose a rigid registration method using surface resampling and denoising, which lowers the impact on registration residuals caused by sampling difference and noise, significantly improving the accuracy. While sparsity-inducing norms reduce sensitivity to outliers and missing data, with preprocessing and region segmentation methods, our registration method is applied to face verification. Without data-driven learning or training, only residuals of rigid registration are utilized, and verification rates at 0.1% FAR are as follows: 100% for $n$ versus $n$ , 96.9% for $n$ versus all, and 98.6% for ROC III experiment on FRGC v2.0 database, and 100% for $n$ versus $n$ and 95.7% for $n$ versus all on Bosphorus database. Experiments show that the proposed algorithm outperforms the state-of-the-art algorithms and is preferable in a verification scenario.

3D geometry-based face recognition in presence of eye and mouth occlusions

This study proposes a novel occlusions detection and restoration strategy. The aim is to success with 3D face recognition even when faces are partially occluded by external objects. The method, which relies on geometrical facial properties, is designed for managing two types of facial occlusions (eye and mouth occlusions due to hands). First occlusions are detected and (if present) classified, by considering their effects on the 3D points cloud. Then, the occluded regions are progressively removed, and finally, the non-occluded symmetrical regions are used to restore the missing information. After the restoration process, face recognition is performed relying on the restored facial information and on the localized landmarks. The landmarking methodology relies on derivatives and on 12 differential geometry descriptors. The discriminating features adopted for facial comparison include shape index histograms, Euclidean and geodetical distances between landmarks, facial curves, and nose volume. Obtained recognition rates, evaluated on the whole Bosphorus database and on our private dataset, ranging from 92.55 to 97.20% depending on the completeness of data.

3D Facial Expression Recognition Using Multi-channel Deep Learning Framework

Facial expression offers an important way of detecting the affective state of a human being. It plays a major role in various fields such as the estimation of students’ attention level in online education, intelligent transportation systems and interactive games. This paper proposes a facial expression recognition system in which two channels of featured images are used to represent a 3D facial scan. Features are extracted from the local binary pattern and local directional pattern using a fine-tuned pre-trained AlexNet and a shallow convolutional neural network. The feature sets are then fused together using canonical correlation analysis. The fused feature set is fed into a multi-support vector machine (mSVM) classifier to classify the expressions into seven basic categories: anger, disgust, fear, happiness, neutral, sadness and surprise. Experiments were carried out on the Bosphorus database using tenfold cross-validation with mutually exclusive training and testing samples. The results show an average accuracy of 87.69% using an mSVM classifier with a polynomial kernel and demonstrate that the system performs better by characterizing the peculiarities in facial expressions than alternative state-of-the-art approaches.

Reg3DFacePtCd: Registration of 3D Point Clouds Using a Common Set of Landmarks for Alignment of Human Face Images

The present work proposes a new method Reg3DFacePtCd for registration of point clouds. The key contribution of the present method is that an unknown face in 3D point cloud form is given to the system and is registered to the already existing known 3D face point clouds using a fast 3D face registration method. The novelty of the present technique is that at first the alignment and registration parameters are found out by initially registering eight key points of the unknown source model to that of the known model. Next, the rest of the point clouds of the unknown model are registered to that of the known model using the same parameters found as above. The main method used for alignment is iterative closest point (ICP) using point-based technique followed by registration in the least squares sense. Mainly there are two significant contributions. Firstly, we have developed a new mathematical model facial landmark point based model across poses to obtain the target or the known model to which all the unknown models will be registered. Secondly, a novel way to accelerate point cloud matching by reducing the number of points has been proposed. Using a small number of points necessarily would speed up the registration process but may inculcate errors. So, to determine the registration quality of the fundamental eight key points on which the entire registration process is based, a new robust metric namely ICV (ICP certainty vector) consisting of several key components have been used. Finally, we have addressed several important face registration issues like pre-processing, convergence and quality of registration of the entire facial point cloud based on the eight key points. Extensive experimentation on Frav3D, GavabDB, and the Bosphorus databases on a high-performance computing environment show the novelty and robustness of the method.

FERLrTc: 2D+3D facial expression recognition via low-rank tensor completion

In this paper, a 4D tensor model is firstly constructed to explore efficient structural information and correlations from multi-modal data (both 2D and 3D face data). As the dimensionality of the generated 4D tensor is high, a tensor dimensionality reduction technique is in need. Since many real-world high-order data often reside in a low dimensional subspace, Tucker decomposition as a powerful technique is utilized to capture multilinear low-rank structure and to extract useful information from the generated 4D tensor data. Our goal is to use Tucker decomposition to obtain a set of core tensors with smaller sizes and factor matrices which are projected into the 4D tensor data for classification prediction. To characterize the involved similarities of the 4D tensor, the low-rank and sparse representation is built in terms of the low-rank structure of factor matrices and the sparsity of the core tensor in the Tucker decomposition of the generated 4D tensor. A tensor completion (TC) framework is embedded to recover the missing information in the 4D tensor modeling process. Thus, a novel tensor dimensionality reduction approach for 2D+3D facial expression recognition via low-rank tensor completion (FERLrTC) is proposed to solve the factor matrices in a majorization–minimization manner by using a rank reduction strategy. Numerical experiments are conducted with a full implementation on the BU-3DFE and Bosphorus databases and synthetic data to illustrate the effectiveness of the proposed approach.

HYBRID IMPROVED BACTERIAL SWARM OPTIMIZATION ALGORITHM FOR HAND-BASED MULTIMODAL BIOMETRIC AUTHENTICATION SYSTEM

This paper proposes a Hybrid Improved Bacterial Swarm (HIBS) optimization algorithm for the minimization of Equal Error Rate (EER) as a performance measure in a hand-based multimodal biometric authentication system. The hybridization of the algorithm was conducted by incorporating Bacterial Foraging Optimization (BFO) and Particle Swarm Optimization (PSO) algorithm to mitigate weaknesses in slow and premature convergence. In the proposed HIBS algorithm, the slow convergence of BFO algorithm was mitigated by using the random walk procedure of Firefly algorithm as an adaptive varying step size instead of using fixed step size. Concurrently, the local optima trap (i.e. premature convergence) of PSO algorithm was averted by using mutation operator. The HIBS algorithm was tested using benchmark functions and compared against classical BFO, PSO and other hybrid algorithms like Genetic Algorithm-Bacterial Foraging Optimization (GA-BFO), Genetic Algorithm-Particle Swarm Optimization (GA-PSO) and other BFO-PSO algorithms to prove its exploration and exploitation ability. It was observed from the experimental results that the EER values, after the influence of the proposed HIBS algorithm, dropped to 0.0070% and 0.0049% from 1.56% and 0.86% for the right and left hand images of the Bosphorus database, respectively. The results indicated the ability of the proposed HIBS in optimization problem where it optimized relevant weights in an authentication system.

Triaxial Slicing for 3-D Face Recognition From Adapted Rotational Invariants Spatial Moments and Minimal Keypoints Dependence

This letter presents a multiple slicing model for three-dimensional (3-D) images of human face, using the frontal, sagittal, and transverse orthogonal planes. The definition of the segments depends on just one key point, the nose tip, which makes it simple and independent of the detection of several key points. For facial recognition, attributes based on adapted 2-D spatial moments of Hu and 3-D spatial invariant rotation moments are extracted from each segment. Tests with the proposed model using the Bosphorus Database for neutral vs nonneutral ROC I experiment, applying linear discriminant analysis as classifier and more than one sample for training, achieved $98.7\%$ of verification rate at $0.1\%$ of false acceptance rate. By using the support vector machine as classifier the rank1 experiment recognition rates of $99\%$ and $95.4\%$ have been achieved for a neutral vs neutral and for a neutral vs non-neutral, respectively. These results approach the state-of-the-art using Bosphorus Database and even surpasses it when anger and disgust expressions are evaluated. In addition, we also evaluate the generalization of our method using the FRGC v2.0 database and achieve competitive results, making the technique promising, especially for its simplicity.