3D Facial Images Research Articles

Identification and analysis of photometric points on 2D facial images: a machine learning approach in orthodontics

The lack of an effective and automated facial landmark identification tool has prompted us to design and develop a smart machine learning approach. The study aims to address two objectives. The primary objective is to assess the effectiveness and accuracy of algorithmic methodology in identifying and analysing facial landmarks on two dimensional (2D) facial images and the secondary objective is to understand the clinical application of automation in facial landmark identification. The study has utilised 418 facial landmark points and 220 landmark measures from 22 2D facial images of volunteers. The study has used a deep learning algorithm ‘You Only Look Once (YOLO)’ to determine the accuracy of the developed system and its clinical applications. The system identified 418 landmarks in total with facial recognition being 100%. Of the total 220 landmark measures, the system provided 48 (21.81%) measures in the error range of 0 to 1 mm, 75 (34.09%) measures in the error range of 2 to 3 mm, 92 (41.81%) measures in the error range of 4 to 5 mm followed by 5 (2.2%) measures in the range of 6 mm. The smart and innovative approach provides valuable training and a helpful tool for the students performing the clinical facial analysis. The automated system with its effective and efficient algorithm delivers fast and reliable landmark identification and analysis.

Face Recognition Algorithm Based on 3D Point Cloud Acquired by Mixed Image Sensor

In order to solve the problem of low recognition accuracy in recognition of 3D face images collected by traditional sensors, a face recognition algorithm for 3D point cloud collected by mixed image sensors is proposed. The algorithm first uses the 3D wheelbase to expand the face image edge. According to the 3D wheelbase, the noise of extended image is detected, and median filtering is used to eliminate the detected noise. Secondly, the priority of the boundary pixels to recognize the face image in the denoising image recognition process is determined, and the key parts such as the illuminance line are analyzed, so that the recognition of the 3D point cloud face image is completed. Experiments show that the proposed algorithm improves the recognition accuracy of 3D face images, which recognition time is lower than that of the traditional algorithm by about 4 times, and the recognition efficiency is high.

Histogram-Based CRC for 3D-Aided Pose-Invariant Face Recognition.

Traditional Collaborative Representation-based Classification algorithms for face recognition (CRC) usually suffer from data uncertainty, especially if it includes various poses and illuminations. To address this issue, in this paper, we design a new CRC method using histogram statistical measurement (H-CRC) combined with a 3D morphable model (3DMM) for pose-invariant face classification. First, we fit a 3DMM to raw images in the dictionary to reconstruct the 3D shapes and textures. The fitting results are used to render numerous virtual samples of 2D images that are frontalized from arbitrary poses. In contrast to other distance-based evaluation algorithms for collaborative (or sparse) representation-based methods, the histogram information of all the generated 2D face images is subsequently exploited. Second, we use a histogram-based metric learning to evaluate the most similar neighbours of the test sample, which aims to obtain ideal result for pose-invariant face recognition using the designed histogram-based 3DMM model and online pruning strategy, forming a unified 3D-aided CRC framework. The proposed method achieves desirable classification results that are conducted on a set of well-known face databases, including ORL, Georgia Tech, FERET, FRGC, PIE and LFW.

A generic framework for editing and synthesizing multimodal data with relative emotion strength

AbstractEmotion is considered to be a core element in performances. In computer animation, both body motions and facial expressions are two popular mediums for a character to express the emotion. However, there has been limited research in studying how to effectively synthesize these two types of character movements using different levels of emotion strength with intuitive control, which is difficult to be modeled effectively. In this work, we explore a common model that can be used to represent the emotion for the applications of body motions and facial expressions synthesis. Unlike previous work that encode emotions into discrete motion style descriptors, we propose a continuous control indicator called emotion strength by controlling which a data‐driven approach is presented to synthesize motions with fine control over emotions. Rather than interpolating motion features to synthesize new motion as in existing work, our method explicitly learns a model mapping low‐level motion features to the emotion strength. Because the motion synthesis model is learned in the training stage, the computation time required for synthesizing motions at run time is very low. We further demonstrate the generality of our proposed framework by editing 2D face images using relative emotion strength. As a result, our method can be applied to interactive applications such as computer games, image editing tools, and virtual reality applications, as well as offline applications such as animation and movie production.

Automated craniofacial landmarks detection on 3D image using geometry characteristics information

BackgroundIndirect anthropometry (IA) is one of the craniofacial anthropometry methods to perform the measurements on the digital facial images. In order to get the linear measurements, a few definable points on the structures of individual facial images have to be plotted as landmark points. Currently, most anthropometric studies use landmark points that are manually plotted on a 3D facial image by the examiner. This method is time-consuming and leads to human biases, which will vary from intra-examiners to inter-examiners when involving large data sets. Biased judgment also leads to a wider gap in measurement error. Thus, this work aims to automate the process of landmarks detection to help in enhancing the accuracy of measurement. In this work, automated craniofacial landmarks (ACL) on a 3D facial image system was developed using geometry characteristics information to identify the nasion (n), pronasale (prn), subnasale (sn), alare (al), labiale superius (ls), stomion (sto), labiale inferius (li), and chelion (ch). These landmarks were detected on the 3D facial image in .obj file format. The IA was also performed by manually plotting the craniofacial landmarks using Mirror software. In both methods, once all landmarks were detected, the eight linear measurements were then extracted. Paired t-test was performed to check the validity of ACL (i) between the subjects and (ii) between the two methods, by comparing the linear measurements extracted from both ACL and AI. The tests were performed on 60 subjects (30 males and 30 females).ResultsThe results on the validity of the ACL against IA between the subjects show accurate detection of n, sn, prn, sto, ls and li landmarks. The paired t-test showed that the seven linear measurements were statistically significant when p < 0.05. As for the results on the validity of the ACL against IA between the methods, ACL is more accurate when p ≈ 0.03.ConclusionsIn conclusion, ACL has been validated with the eight landmarks and is suitable for automated facial recognition. ACL has proved its validity and demonstrated the practicability to be used as an alternative for IA, as it is time-saving and free from human biases.

Artificial Immune System with Negative Selection Applied to Facial Biometry Based on Binary Pattern Characteristics

This work aims to explore resources and alternatives for 3D facial biometry using Binary Patterns. A 3D facial geometry image is converted into two 2D representations, appointed as descriptors: A Depth Map and an Azimuthal Projection Distance Image. The first is known as traditional facial geometry, and the second is normal facial geometry that is able to capture the information of different geometries. The characteristics of Local Binary Patterns, Local Phase Quantisers and Gabor Binary Patterns were used with the Depth Map and Azimuthal Projection Distance Image to produce six new facial descriptors: 3D Local Binary Patterns, Local Azimuthal Binary Patterns, Local Depth Phase Quantisers, Local Azimuthal Phase Patterns, and Local Depth Gabor Binary Pattern Magnitudes and Phases. Then, this work uses the immune concept to propose a new approach to realize facial biometry, where the eight new facial descriptors were applied to an Artificial Intelligence algorithm named Artificial Immune Systems of Negative Selection. The analysis of the results shows the efficiency, robustness, precision and reliability of this approach, encouraging further research in this area.

Integration of digital maxillary dental casts with 3D facial images in orthodontic patients.

To evaluate three-dimensional (3D) accuracy and reliability of nonradiographic dentofacial images integrated with a two-step method. 3D facial images, cone-beam computed tomography (CBCT) images and digital maxillary dental casts were obtained from 20 pre-orthodontic subjects. Digital dental casts were integrated into 3D facial images using a two-step method based on the anterior tooth area. 3D coordinate values of five dental landmarks were identified in both dentofacial images and CBCT images. The accuracy of the integration method was assessed with paired t-tests between dentofacial images and CBCT-based reference standards. Intraclass correlation coefficients (ICCs) were assessed for the reliability of dentofacial images and CBCT-based images. Analysis of variance and Kruskal-Wallis tests evaluated the accuracy of the method in different dimensions. There was no statistical difference between dentofacial images and CBCT reference standards in both translational and rotational dimensions (P > .05). Translational mean absolute errors for full dentitions were within 0.42 mm and ICCs were over 0.998 in x, y, and z directions. Rotational mean absolute errors for full dentitions were within 0.92° and ICCs over 0.734 in pitch, yaw, and roll orientations. Integration errors were significantly greater in the first molar, z-translation, and pitch rotation (P < .05). Integrating 3D dentofacial images with the two-step method is precise and acceptable for clinical diagnostics and scientific purposes. Errors were greater in the molar region, z-translation, and pitch rotation.

Face Alignment via 3D-Assisted Features

We present a practical 3D-assited face alignment framework based on cascaded regression in this paper. The 3D information embedded in 2D face image is utilized to calculate two novel components to improve the performance of 2D methods in unconstrained face alignment. The two novel components for 2D image features are the projected local patch and the visibility of each landmark. First, we propose to extract the landmark related features in the projected local patches on 2D image from the corresponding 3D face model. Local patches of a fixed landmark in 3D face models for different 2D images cover the same region of face anatomically. The extracted features are more accurate for further locations regression of landmarks. Second, we propose to estimate the visibilities of 2D landmarks based on 3D face model, which are proven to be vital to address large pose face alignment problem. In this paper, we adopt Local Binary Features (LBF) to extract landmark related features in the proposed framework, and name the new method as 3D-Assisted LBF (3DALBF). An extensive evaluation on two face databases shows that 3DALBF can achieve better alignment results than the original 2D method and maintain the speed advantage of 2D method over 3D method.

Shortcut-Upsampling Block for 3D Face Reconstruction and Dense Alignment via Position Map

Joint 3D face reconstruction and dense alignment based on a single image has always been a challenge in computer vision. Recently, some work achieves these two goals simultaneously by adopting the position map to represent a 3D face. In this paper, we design a novel upsampling block, denoted as shortcut-upsampling block, to construct the position map. We first utilize the residual blocks to extract and downsample the feature maps from the 2D face image, and then utilize the proposed shortcut-upsampling blocks to convert the feature maps to the corresponding position map. Shortcut-upsampling block allows our model to be only 121M and achieve excellent performance on 3D face reconstruction and dense alignment. Compared with other deep learning methods based on 3D Morphable Model(3DMM), our model is the fastest when reconstructing a 3D face. In addition, through introducing a dynamic weight loss function, our model can converge the loss to a lower value and obtain a better performance on 3D face reconstruction.

Three Dimension Face Recognition Based on Gabor Transformation and Support Vector Machine

ABSTRACT Face recognition based on 2D face images is still challenged by the large change of illumination, pose and expression after more than 10 years’ research and its recognition rate is still far away from satisfaction under the change of the above three factors. Gabor wavelet and support vector machine (SVM) are studied. Gabor wavelet has good biological neurons function, and good adaptive change to illumination. Furthermore, SVM can provide good classification effect. Gabor wavelet and SVM are combined with PCAto improve recognition rate of human face. Besides, we put forward a kind of Gabor wavelet feature extraction method based on three-dimensional human face contour line. Five different types of human face recognition are used to verify the effectiveness of this algorithm. The experiment results show that the proposed scheme has higher recognition rate than several traditional schemes.

Machine Learning Approaches for Prediction of Facial Rejuvenation Using Real and Synthetic Data

This paper proposes a novel machine learning approaches to predict the outcome of facial rejuvenation prior to a cosmetic procedure. This is achieved by estimating the required amount of dermal filler volume that needs to be applied on the face by learning the underlying structural mapping from the pretreatment and posttreatment 3D face images. We develop and train our proposed deep neural network, called Rejuv3DNet, designed specifically to predict the dermal filler volume. We also propose the kernel regression (KR)-based model to validate and improve our volume estimation results using regression. Our other contributions include the development of the first 3D face cosmetic dataset, which consists of real-world pretreatment and posttreatment 3D face images and a novel technique for the generation of synthetic cosmetic treatment 3D face images. Our experimental results show that the proposed Rejuv3DNet and the KR model achieve 62.5% and 66.67%, respectively, on real-world data, while these techniques achieve a prediction accuracy of 75.2% and 89.5%, and 77.2% and 90.1% on our two different synthetic datasets. Our proposed techniques have been found to be computationally efficient, achieving near real-time prediction performance. The reported accuracies are our preliminary results for proof of concept, which can be improved with more data. The proposed approach has the potential for further investigation in the cosmetic surgery domain.

On Learning 3D Face Morphable Model from In-the-wild Images

As a classic statistical model of 3D facial shape and albedo, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as, the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of in-the-wild face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, lighting, shape and albedo parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and albedo parameters to the 3D shape and albedo, respectively. With the projection parameter, lighting, 3D shape, and albedo, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment, 3D reconstruction, and face editing. Source code and additional results can be found at our project page: http://cvlab.cse.msu.edu/project-nonlinear-3dmm.html.

Joint Face Alignment and 3D Face Reconstruction with Application to Face Recognition.

Face alignment and 3D face reconstruction are traditionally accomplished as separated tasks. By exploring the strong correlation between 2D landmarks and 3D shapes, in contrast, we propose a joint face alignment and 3D face reconstruction method to simultaneously solve these two problems for 2D face images of arbitrary poses and expressions. This method, based on a summation model of 3D faces and cascaded regression in 2D and 3D shape spaces, iteratively and alternately applies two cascaded regressors, one for updating 2D landmarks and the other for 3D shape. The 3D shape and the landmarks are correlated via a 3D-to-2D mapping matrix, which is updated in each iteration to refine the location and visibility of 2D landmarks. Unlike existing methods, the proposed method can fully automatically generate both pose-and-expression-normalized (PEN) and expressive 3D faces and localize both visible and invisible 2D landmarks. Based on the PEN 3D faces, we devise a method to enhance face recognition accuracy across poses and expressions. Both linear and nonlinear implementations of the proposed method are presented and evaluated in this paper. Extensive experiments show that the proposed method can achieve the state-of-the-art accuracy in both face alignment and 3D face reconstruction, and benefit face recognition owing to its reconstructed PEN 3D face.

Reverse Engineering in Orthodontics.

Three-dimensional (3D) imaging has advanced greatly and is used extensively in orthodontics. It is worth outlining and reviewing the developments of reverse engineering (RE) as its applications are growing more widespread and diverse. Data from an existing object are used to create a digital model. A traditional RE process is usually performed in these stages: (1) obtaining data, (2) restructuring the surfaces, and (3) creating a useful model. They are classified as (1) laser projection based and (2) fringe projection based. This digital technology has been used in creating 3D model scanning, 3D digital model superimposition, diagnostic setup, volumetric assessment of tooth wear, soft tissue facial analysis, incorporation of digital model to 3D facial image, lip position and smile reproducibility, analysis of tooth position after orthodontic treatment, and anthropometric measurements. This system has proven itself to have a varied probability of applications and researches in the field of orthodontics. Similar to every single system, even RE has its own benefits and shortcomings. The complexity of the process and high cost are the major disadvantages reported so far. Rapid advancement of this technology possibly will rapidly inverse the negative results that emerged previously. As a future work, innovative use of RE technology is necessary to make this system triumph in the field of orthodontics.

Three-Dimensional Soft-Tissue Evaluation in Patients with Cleft Lip and Palate.

BackgroundIn cleft lip and palate (CLP) patients, the shape of the facial soft tissues shows variety in 3 dimensions (3D). Two-dimensional (2D) photographs and radiographs are insufficient in the examination of these anomalies. The aim of this retrospective study was to examine the soft tissue and craniofacial characteristics of individuals with nonsyndromic unilateral cleft lip and palate (UCLP), bilateral cleft lip and palate (BCLP), skeletal Class III malocclusions, or skeletal Class I malocclusions using 3D facial imaging.Material/MethodsThe entire study group consisted of a total of 158 patients, aged 8–32 years: 29 of the patients had UCLP, 22 BCLP, 54 had skeletal Class III malocclusions, and 53 had skeletal Class I malocclusions. 3D stereophotogrammetric soft-tissue recordings of all patients were analyzed. ANOVA and the Kruskal-Wallis test were performed to compare the groups.ResultsStatistically significant differences were observed among the groups in terms of linear, angular, proportional. and volumetric measurements. While nasal differences were not observed in the Class III group, nose and upper-lip deformities were common in the CLP groups. Upper-lip projection was reduced in all 3 groups. In the Class III patients, the lower lip and chin were more prominent than in the other groups. The facial convexity angle was increased in the CLP and Class III groups. The upper-lip volume was decreased in the BCLP, the UCLP, and the Class III groups.ConclusionsPatients with skeletal Class III or CLP anomalies showed significantly different soft-tissue characteristics than the Class I control group. 3D stereophotogrammetric facial imaging is an easy and noninvasive method that can be used in examination and recording of these facial deformities. It is possible to make volumetric measurements using this method.

Spatially Dense 3D Facial Heritability and Modules of Co-heritability in a Father-Offspring Design.

Introduction: The human face is a complex trait displaying a strong genetic component as illustrated by various studies on facial heritability. Most of these start from sparse descriptions of facial shape using a limited set of landmarks. Subsequently, facial features are preselected as univariate measurements or principal components and the heritability is estimated for each of these features separately. However, none of these studies investigated multivariate facial features, nor the co-heritability between different facial features. Here we report a spatially dense multivariate analysis of facial heritability and co-heritability starting from data from fathers and their children available within ALSPAC. Additionally, we provide an elaborate overview of related craniofacial heritability studies.Methods: In total, 3D facial images of 762 father-offspring pairs were retained after quality control. An anthropometric mask was applied to these images to establish spatially dense quasi-landmark configurations. Partial least squares regression was performed and the (co-)heritability for all quasi-landmarks (∼7160) was computed as twice the regression coefficient. Subsequently, these were used as input to a hierarchical facial segmentation, resulting in the definition of facial modules that are internally integrated through the biological mechanisms of inheritance. Finally, multivariate heritability estimates were obtained for each of the resulting modules.Results: Nearly all modular estimates reached statistical significance under 1,000,000 permutations and after multiple testing correction (p ≤ 1.3889 × 10-3), displaying low to high heritability scores. Particular facial areas showing the greatest heritability were similar for both sons and daughters. However, higher estimates were obtained in the former. These areas included the global face, upper facial part (encompassing the nasion, zygomas and forehead) and nose, with values reaching 82% in boys and 72% in girls. The lower parts of the face only showed low to moderate levels of heritability.Conclusion: In this work, we refrain from reducing facial variation to a series of individual measurements and analyze the heritability and co-heritability from spatially dense landmark configurations at multiple levels of organization. Finally, a multivariate estimation of heritability for global-to-local facial segments is reported. Knowledge of the genetic determination of facial shape is useful in the identification of genetic variants that underlie normal-range facial variation.

Dictionary Integration Using 3D Morphable Face Models for Pose-Invariant Collaborative-Representation-Based Classification

The paper presents a dictionary integration algorithm using 3D morphable face models (3DMM) for pose-invariant collaborative-representation-based face classification. To this end, we first fit a 3DMM to the 2D face images of a dictionary to reconstruct the 3D shape and texture of each image. The 3D faces are used to render a number of virtual 2D face images with arbitrary pose variations to augment the training data, by merging the original and rendered virtual samples to create an extended dictionary. Second, to reduce the information redundancy of the extended dictionary and improve the sparsity of reconstruction coefficient vectors using collaborative-representation-based classification (CRC), we exploit an on-line class elimination scheme to optimise the extended dictionary by identifying the training samples of the most representative classes for a given query. The final goal is to perform pose-invariant face classification using the proposed dictionary integration method and the on-line pruning strategy under the CRC framework. Experimental results obtained for a set of well-known face data sets demonstrate the merits of the proposed method, especially its robustness to pose variations.

Six NSCL/P Loci Show Associations With Normal-Range Craniofacial Variation.

Objectives: Orofacial clefting is one of the most prevalent craniofacial malformations. Previous research has demonstrated that unaffected relatives of patients with non-syndromic cleft lip with/without cleft palate (NSCL/P) show distinctive facial features, which can be an expression of underlying NSCL/P susceptibility genes. These results support the hypothesis that genes involved in the occurrence of a cleft also play a role in normal craniofacial development. In this study, we investigated the influence of genetic variants associated with NSCL/P on normal-range variation in facial shape.Methods: A literature review of genome wide association studies (GWAS) investigating the genetic etiology of NSCL/P was performed, resulting in a list of 75 single nucleotide polymorphisms (SNPs) located in 38 genetic loci. Genotype data were available for 65 of these selected SNPs in three datasets with a combined sample size of 7,418 participants of European ancestry, whose 3D facial images were also available. The effect of each SNP was tested using a multivariate canonical correlation analysis (CCA) against 63 hierarchically-constructed facial segments in each of the three datasets and meta-analyzed. This allowed for the investigation of associations between SNPs known to be involved in NSCL/P and normal-range facial shape variations in a global-to-local perspective, without preselecting specific facial shape features or characteristics.Results: Six NSCL/P SNPs showed significant associations with variation in normal-range facial morphology. rs6740960 showed significant effects in the chin area (p = 3.71 × 10−28). This SNP lies in a non-coding area. Another SNP, rs227731 near the NOG gene, showed a significant effect in the philtrum area (p = 1.96 × 10−16). Three SNPs showed significant effects on the shape of the nose. rs742071 (p = 8.71 × 10−14), rs34246903 (p = 6.87 × 10−12), and rs10512248 (p = 8.4 × 10−9). Respectively, these SNPs are annotated to PAX7, MSX1, and PTCH1. Finally, rs7590268, an intron variant of THADA, showed an effect in the shape of the supraorbital ridge (p = 3.84 × 10−7).Conclusions: This study provides additional evidence NSCL/P-associated genetic variants influence normal-range craniofacial morphology, with significant effects observed for the chin, the nose, the supraorbital ridges and the philtrum area.

SNPs Associated With Testosterone Levels Influence Human Facial Morphology.

Many factors influence human facial morphology, including genetics, age, nutrition, biomechanical forces, and endocrine factors. Moreover, facial features clearly differ between males and females, and these differences are driven primarily by the influence of sex hormones during growth and development. Specific genetic variants are known to influence circulating sex hormone levels in humans, which we hypothesize, in turn, affect facial features. In this study, we investigated the effects of testosterone-related genetic variants on facial morphology. We tested 32 genetic variants across 22 candidate genes related to levels of testosterone, sex hormone-binding globulin (SHGB) and dehydroepiandrosterone sulfate (DHEAS) in three cohorts of healthy individuals for which 3D facial surface images were available (Pittsburgh 3DFN, Penn State and ALSPAC cohorts; total n = 7418). Facial shape was described using a recently developed extension of the dense-surface correspondence approach, in which the 3D facial surface was partitioned into a set of 63 hierarchically organized modules. Each variant was tested against each of the facial surface modules in a multivariate genetic association-testing framework and meta-analyzed. Additionally, the association between these candidate SNPs and five facial ratios was investigated in the Pittsburgh 3DFN cohort. Two significant associations involving intronic variants of SHBG were found: both rs12150660 (p = 1.07E-07) and rs1799941 (p = 6.15E-06) showed an effect on mandible shape. Rs8023580 (an intronic variant of NR2F2-AS1) showed an association with the total and upper facial width to height ratios (p = 9.61E-04 and p = 7.35E-04, respectively). These results indicate that testosterone-related genetic variants affect normal-range facial morphology, and in particular, facial features known to exhibit strong sexual dimorphism in humans.

Single Image 3D Face Reconstruction Based on Statistical Model

Single image 3D face reconstruction technology has a broad application in the life, also has a high technical difficulties. This paper proposes a single image 3D face reconstruction technology based on statistical model. The method established 3D face statistical model based on 3D face database in advance, and trained 2D facial feature points parameter model based on the 2D face database using the algorithm of SDM. when reconstruct 3D face model from single image, first we use 2D facial feature points parameter model to extract face feature points; Then, according to 3D face model, we use adaptive learning factor gradient descent method to iteratively optimize the energy function, to get statistical model parametric vector, namely got the 3D face model corresponding to the 2D face images. The results show that use the proposed method to rebuild the faces have high similarity.