3D Facial Landmark Research Articles

3D Face Reconstruction with Geometry Details from a Single Image.

3D face reconstruction from a single image is a classical and challenging problem, with wide applications in many areas. Inspired by recent works in face animation from RGBD or monocular video inputs, we develop a novel method for reconstructing 3D faces from unconstrained 2D images, using a coarse-to-fine optimization strategy. First, a smooth coarse 3D face is generated from an example-based bilinear face model, by aligning the projection of 3D face landmarks with 2D landmarks detected from the input image. Afterwards, using local corrective deformation fields, the coarse 3D face is refined using photometric consistency constraints, resulting in a medium face shape. Finally, a shape-from-shading method is applied on the medium face to recover fine geometric details. Our method outperforms stateof- the-art approaches in terms of accuracy and detail recovery, which is demonstrated in extensive experiments using real world models and publicly available datasets.

Accurate landmarking from 3D facial scans by CNN and cascade regression

Facial landmarking locates the key facial feature points on facial data, which provides not only information on semantic facial structures, but also prior knowledge for other types of facial analysis. However, most of the existing works still focus on the 2D facial image which is quite sensitive to the lighting condition changes. In order to address this limitation, this paper proposed a coarse-to-fine method only based on the 3D facial scan data extracted from professional equipment to automatically and accurately estimate the landmark localization. Specifically, we firstly trained a convolutional neural network (CNN) to initialize the face landmarks instead of the mean shape. Then the proposed cascade regression networks learn the mapping function between 3D facial geometry feature and landmarks location. Tested on Bosphorus database, the experimental results demonstrated effectiveness and accuracy of the proposed method for [Formula: see text] landmarks. Compared with other methods, the results in several points demonstrate state-of-the-art performance.

Expression robust 3D face landmarking using thresholded surface normals

3D face recognition is an increasing popular modality for biometric authentication, for example in the iPhoneX. Landmarking plays a significant role in region based face recognition algorithms. The accuracy and consistency of the landmarking will directly determine the effectiveness of feature extraction and hence the overall recognition performance. While surface normals have been shown to provide high performing features for face recognition, their use in landmarking has not been widely explored. To this end, a new 3D facial landmarking algorithm based on thresholded surface normal maps is proposed, which is applicable to widely used 3D face databases. The benefits of employing surface normals are demonstrated for both facial roll and yaw rotation calibration and nasal landmarks localization. Results on the Bosphorus, FRGC and BU-3DFE databases show that the detected landmarks possess high within-class consistency and accuracy under different expressions. For several key landmarks the performance achieved surpasses that of state-of-the-art techniques and is also training free and computationally efficient. The use of surface normals therefore provides a useful representation of the 3D surface and the proposed landmarking algorithm provides an effective approach to localising the key nasal landmarks.

Modelling of Orthogonal Craniofacial Profiles

We present a fully-automatic image processing pipeline to build a set of 2D morphable models of three craniofacial profiles from orthogonal viewpoints, side view, front view and top view, using a set of 3D head surface images. Subjects in this dataset wear a close-fitting latex cap to reveal the overall skull shape. Texture-based 3D pose normalization and facial landmarking are applied to extract the profiles from 3D raw scans. Fully-automatic profile annotation, subdivision and registration methods are used to establish dense correspondence among sagittal profiles. The collection of sagittal profiles in dense correspondence are scaled and aligned using Generalised Procrustes Analysis (GPA), before applying principal component analysis to generate a morphable model. Additionally, we propose a new alternative alignment called the Ellipse Centre Nasion (ECN) method. Our model is used in a case study of craniosynostosis intervention outcome evaluation, and the evaluation reveals that the proposed model achieves state-of-the-art results. We make publicly available both the morphable models and the profile dataset used to construct it.

Viewpoint-Consistent 3D Face Alignment.

Most approaches to face alignment treat the face as a 2D object, which fails to represent depth variation and is vulnerable to loss of shape consistency when the face rotates along a 3D axis. Because faces commonly rotate three dimensionally, 2D approaches are vulnerable to significant error. 3D morphable models, employed as a second step in 2D+3D approaches are robust to face rotation but are computationally too expensive for many applications, yet their ability to maintain viewpoint consistency is unknown. We present an alternative approach that estimates 3D face landmarks in a single face image. The method uses a regression forest-based algorithm that adds a third dimension to the common cascade pipeline. 3D face landmarks are estimated directly, which avoids fitting a 3D morphable model. The proposed method achieves viewpoint consistency in a computationally efficient manner that is robust to 3D face rotation. To train and test our approach, we introduce the Multi-PIE Viewpoint Consistent database. In empirical tests, the proposed method achieved simple yet effective head pose estimation and viewpoint consistency on multiple measures relative to alternative approaches.

Role for 2D image generated 3D face models in the rehabilitation of facial palsy.

The outcome for patients diagnosed with facial palsy has been shown to be linked to rehabilitation. Dense 3D morphable models have been shown within the computer vision to create accurate representations of human faces even from single 2D images. This has the potential to provide feedback to both the patient and medical expert dealing with the rehabilitation plan. It is proposed that a framework for the creation and measuring of patient facial movement consisting of a hybrid 2D facial landmark fitting technique which shows better accuracy in testing than current methods and 3D model fitting.

Robust facial landmark detection and tracking across poses and expressions for in-the-wild monocular video

We present a novel approach for automatically detecting and tracking facial landmarks across poses and expressions from in-the-wild monocular video data, e.g., YouTube videos and smartphone recordings. Our method does not require any calibration or manual adjustment for new individual input videos or actors. Firstly, we propose a method of robust 2D facial landmark detection across poses, by combining shape-face canonical-correlation analysis with a global supervised descent method. Since 2D regression-based methods are sensitive to unstable initialization, and the temporal and spatial coherence of videos is ignored, we utilize a coarse-todense 3D facial expression reconstruction method to refine the 2D landmarks. On one side, we employ an in-the-wild method to extract the coarse reconstruction result and its corresponding texture using the detected sparse facial landmarks, followed by robust pose, expression, and identity estimation. On the other side, to obtain dense reconstruction results, we give a face tracking flow method that corrects coarse reconstruction results and tracks weakly textured areas; this is used to iteratively update the coarse face model. Finally, a dense reconstruction result is estimated after it converges. Extensive experiments on a variety of video sequences recorded by ourselves or downloaded from YouTube show the results of facial landmark detection and tracking under various lighting conditions, for various head poses and facial expressions. The overall performance and a comparison with state-of-art methods demonstrate the robustness and effectiveness of our method.

Miscellaneous expertise of 3D facial landmarks in recent literature

As the interest in human face grows, facial landmarks become more and more important for a large variety of fields and applications. Multipurpose medical is evidently leading in this sense, but others such as skull study for crime scenes, sex estimation, and attractiveness quantification, morphological and cephalometric analyses are present. A cluster analysis of the examined papers is performed depending on scope, landmarking method, and facial database features. The purpose is to face these topics by providing the reader with a comprehensive view of what 3D facial landmarks are and what "they have been up to" in 2014 and 2015. The aim is to offer to users the very up-to-date scenario, the best outcomes, i.e., the latest frontier of landmarks' talents and skills. The third dimension allowed to select the most prominent contributions, especially in terms of scientific advance innovativeness.

Miscellaneous expertise of 3D facial landmarks in recent literature

As the interest in human face grows, facial landmarks become more and more important for a large variety of fields and applications. Multipurpose medical is evidently leading in this sense, but others such as skull study for crime scenes, sex estimation, and attractiveness quantification, morphological and cephalometric analyses are present. A cluster analysis of the examined papers is performed depending on scope, landmarking method, and facial database features. The purpose is face these topics by providing the reader with a comprehensive view of what 3D facial landmarks are and what they have been up to in 2014 and 2015. The aim is offer users the very up-to-date scenario, the best outcomes, i.e., the latest frontier of landmarks' talents and skills. The third dimension allowed select the most prominent contributions, especially in terms of scientific advance innovativeness.

Persistent homology to analyse 3D faces and assess body weight gain

In this paper, we analyse patterns in face shape variation due to weight gain. We propose the use of persistent homology descriptors to get geometric and topological information about the configuration of anthropometric 3D face landmarks. In this way, evaluating face changes boils down to comparing the descriptors computed on 3D face scans taken at different times. By applying dimensionality reduction techniques to the dissimilarity matrix of descriptors, we get a space in which each face is a point and face shape variations are encoded as trajectories in that space. Our results show that persistent homology is able to identify features which are well related to overweight and may help assessing individual weight trends. The research was carried out in the context of the European project SEMEOTICONS, which developed a multisensory platform which detects and monitors over time facial signs of cardio-metabolic risk.

Pose-invariant face recognition with homography-based normalization

Pose-invariant face recognition (PIFR) refers to the ability that recognizes face images with arbitrary pose variations. Among existing PIFR algorithms, pose normalization has been proved to be an effective approach which preserves texture fidelity, but usually depends on precise 3D face models or at high computational cost. In this paper, we propose an highly efficient PIFR algorithm that effectively handles the main challenges caused by pose variation. First, a dense grid of 3D facial landmarks are projected to each 2D face image, which enables feature extraction in an pose adaptive manner. Second, for the local patch around each landmark, an optimal warp is estimated based on homography to correct texture deformation caused by pose variations. The reconstructed frontal-view patches are then utilized for face recognition with traditional face descriptors. The homography-based normalization is highly efficient and the synthesized frontal face images are of high quality. Finally, we propose an effective approach for occlusion detection, which enables face recognition with visible patches only. Therefore, the proposed algorithm effectively handles the main challenges in PIFR. Experimental results on four popular face databases demonstrate that the propose approach performs well on both constrained and unconstrained environments.

The 3D Facial Norms Database: Part 1. A Web-Based Craniofacial Anthropometric and Image Repository for the Clinical and Research Community.

With the current widespread use of three-dimensional (3D) facial surface imaging in clinical and research environments, there is a growing demand for high-quality craniofacial norms based on 3D imaging technology. The principal goal of the 3D Facial Norms (3DFN) project was to create an interactive, Web-based repository of 3D facial images and measurements. Unlike other repositories, users can gain access to both summary-level statistics and individual-level data, including 3D facial landmark coordinates, 3D-derived anthropometric measurements, 3D facial surface images, and genotypes from every individual in the dataset. The 3DFN database currently consists of 2454 male and female participants ranging in age from 3 to 40 years. The subjects were recruited at four US sites and screened for a history of craniofacial conditions. The goal of this article is to introduce readers to the 3DFN repository by providing a general overview of the project, explaining the rationale behind the creation of the database, and describing the methods used to collect the data. Sex- and age-specific summary statistics (means and standard deviations) and growth curves for every anthropometric measurement in the 3DFN dataset are provided as a supplement available online. These summary statistics and growth curves can aid clinicians in the assessment of craniofacial dysmorphology.

3D facial landmark localization using texture regression via conformal mapping

3D facial alignment typically requires the accurate estimate of facial landmarks. Most existing landmark detection methods use geometry characterization or resort to regression algorithms performed on point clouds or range images. A method combining both 3D geometry information and 2D texture has rarely been investigated. In this paper, we propose a novel 3D facial landmark localization algorithm, based on conformal geometric mapping, that can convert a 3D model to 2D using both geometry and texture information. Then a two-layers-regression method is used to improve the stability of landmark localization on the 2D geometry images. This method is impervious to pose changes and robust with respect to changes in expression. We evaluated the proposed approach on publicly available datasets and demonstrate how the use of 2D regression methods boosts the robustness and accuracy of 3D facial landmark localization.

An Automatic 3D Facial Landmarking Algorithm Using 2D Gabor Wavelets.

In this paper, we present a novel approach to automatic 3D facial landmarking using 2D Gabor wavelets. Our algorithm considers the face to be a surface and uses map projections to derive 2D features from raw data. Extracted features include texture, relief map, and transformations thereof. We extend an established 2D landmarking method for simultaneous evaluation of these data. The method is validated by performing landmarking experiments on two data sets using 21 landmarks and compared with an active shape model implementation. On average, landmarking error for our method was 1.9 mm, whereas the active shape model resulted in an average landmarking error of 2.3 mm. A second study investigating facial shape heritability in related individuals concludes that automatic landmarking is on par with manual landmarking for some landmarks. Our algorithm can be trained in 30 min to automatically landmark 3D facial data sets of any size, and allows for fast and robust landmarking of 3D faces.

2D facial landmark model design by combining key points and inserted points

Facial landmarks can be used as a compact and effective representation of facial geometry. Although different facial landmark models (LMs) have been used for various face image analysis tasks, there are few works investigating the design of the LMs, which takes care of how many landmarks to use to represent facial geometry and where to locate them. Existing LMs are either not sufficient to approximate face shapes precisely or including many redundant landmarks. This paper aims to develop an optimized LM which can delicately represent facial geometry while keeping as few landmarks as possible. We divide the landmarks into two categories, namely key points (KPs) and inserted points (IPs). KPs include anatomical and mathematical landmarks which are defined with consideration of an anthropometry face model; IPs are pseudo-landmarks that help describe the face shape accurately. An iterative searching scheme is proposed to find a series of optimal positions for IPs until some acceptable error is obtained. The obtained LM includes 54KPs and 44IPs distributed on different facial regions. We compare our LM and other LMs with similar number of points in terms of shape approximation error and landmark detection error. Experimental results show the superiority of our LM in both criteria. The proposed LM is useful for applications that require delicate shape representations and promotes standardization of facial LMs.

3D facial landmarks: Inter-operator variability of manual annotation.

BackgroundManual annotation of landmarks is a known source of variance, which exist in all fields of medical imaging, influencing the accuracy and interpretation of the results. However, the variability of human facial landmarks is only sparsely addressed in the current literature as opposed to e.g. the research fields of orthodontics and cephalometrics. We present a full facial 3D annotation procedure and a sparse set of manually annotated landmarks, in effort to reduce operator time and minimize the variance.MethodFacial scans from 36 voluntary unrelated blood donors from the Danish Blood Donor Study was randomly chosen. Six operators twice manually annotated 73 anatomical and pseudo-landmarks, using a three-step scheme producing a dense point correspondence map. We analyzed both the intra- and inter-operator variability, using mixed-model ANOVA. We then compared four sparse sets of landmarks in order to construct a dense correspondence map of the 3D scans with a minimum point variance.ResultsThe anatomical landmarks of the eye were associated with the lowest variance, particularly the center of the pupils. Whereas points of the jaw and eyebrows have the highest variation. We see marginal variability in regards to intra-operator and portraits. Using a sparse set of landmarks (n=14), that capture the whole face, the dense point mean variance was reduced from 1.92 to 0.54 mm.ConclusionThe inter-operator variability was primarily associated with particular landmarks, where more leniently landmarks had the highest variability. The variables embedded in the portray and the reliability of a trained operator did only have marginal influence on the variability. Further, using 14 of the annotated landmarks we were able to reduced the variability and create a dense correspondences mesh to capture all facial features.

Robust 3D face landmark localization based on local coordinate coding.

In the 3D facial animation and synthesis community, input faces are usually required to be labeled by a set of landmarks for parameterization. Because of the variations in pose, expression and resolution, automatic 3D face landmark localization remains a challenge. In this paper, a novel landmark localization approach is presented. The approach is based on local coordinate coding (LCC) and consists of two stages. In the first stage, we perform nose detection, relying on the fact that the nose shape is usually invariant under the variations in the pose, expression, and resolution. Then, we use the iterative closest points algorithm to find a 3D affine transformation that aligns the input face to a reference face. In the second stage, we perform resampling to build correspondences between the input 3D face and the training faces. Then, an LCC-based localization algorithm is proposed to obtain the positions of the landmarks in the input face. Experimental results show that the proposed method is comparable to state of the art methods in terms of its robustness, flexibility, and accuracy.

Displaced dynamic expression regression for real-time facial tracking and animation

We present a fully automatic approach to real-time facial tracking and animation with a single video camera. Our approach does not need any calibration for each individual user. It learns a generic regressor from public image datasets, which can be applied to any user and arbitrary video cameras to infer accurate 2D facial landmarks as well as the 3D facial shape from 2D video frames. The inferred 2D landmarks are then used to adapt the camera matrix and the user identity to better match the facial expressions of the current user. The regression and adaptation are performed in an alternating manner. With more and more facial expressions observed in the video, the whole process converges quickly with accurate facial tracking and animation. In experiments, our approach demonstrates a level of robustness and accuracy on par with state-of-the-art techniques that require a time-consuming calibration step for each individual user, while running at 28 fps on average. We consider our approach to be an attractive solution for wide deployment in consumer-level applications.

3D landmarking in multiexpression face analysis: a preliminary study on eyebrows and mouth.

The application of three-dimensional (3D) facial analysis and landmarking algorithms in the field of maxillofacial surgery and other medical applications, such as diagnosis of diseases by facial anomalies and dysmorphism, has gained a lot of attention. In a previous work, we used a geometric approach to automatically extract some 3D facial key points, called landmarks, working in the differential geometry domain, through the coefficients of fundamental forms, principal curvatures, mean and Gaussian curvatures, derivatives, shape and curvedness indexes, and tangent map. In this article we describe the extension of our previous landmarking algorithm, which is now able to extract eyebrows and mouth landmarks using both old and new meshes. The algorithm has been tested on our face database and on the public Bosphorus 3D database. We chose to work on the mouth and eyebrows as a separate study because of the role that these parts play in facial expressions. In fact, since the mouth is the part of the face that moves the most and affects mainly facial expressions, extracting mouth landmarks from various facial poses means that the newly developed algorithm is pose-independent. This journal requires that authors assign a level of evidence to each submission to which Evidence-Based Medicine rankings are applicable. This excludes Review Articles, Book Reviews, and manuscripts that concern Basic Science, Animal Studies, Cadaver Studies, and Experimental Studies. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors http://www.springer.com/00266 .

Angular‐partitioned spin image descriptor for robust 3D facial landmark detection

Spin images have been widely used for representing local three-dimensional (3D) shapes in 3D object recognition and 3D facial landmark detection. An improved spin image descriptor is proposed which enhances shape discrimination performance significantly over the spin image. By generating several sub-spin images for angular-partitioned subspaces, the description of unique angular features within local surfaces can be offered. The experimental results show that the proposed angular-partitioned spin image enhances the localisation accuracy of facial landmarks by 47% and detection reliability by 33% as compared to the spin image.