3D Face Model Research Articles

顔イラストからの特徴点抽出と特徴点に基づく 3 次元顔モデル変形

顔イラストからの特徴点抽出と特徴点に基づく 3 次元顔モデル変形

Method Comparison of 3D Facial Reconstruction Coresponding to 2D Image

Recent years, facial reconstruction becomes a much-studied topic. In particular, the past decade has witnessed a renewed interest, generating a large number of research centers and proposing techniques to address them but most have their limitations and drawbacks. The main constrain of attaining an accuracy of the facial information for reconstruction of its 3D counterpart was increasing accuracy for 3D face geometry. Though this topic has gained a lot of concern and popularity, a fully accurate facial reconstruction mechanism has not yet been identified due to the complexity and ambiguity involved. This survey focuses on 3D face reconstruction by presenting comparison methods beetween Shape-from-Shading, the 3D Morphable Model and Structure from Movement based on 2D face analysis with higher accuracy. 3D face model density affects the provided information. The dense of the 3D facial model is; the more information it could provide. 3D facial reconstruction method currently requires a complicated process and high system costs.

3D Face Similarity Measure by Fréchet Distances of Geodesics

3D face similarity is a critical issue in computer vision, computer graphics and face recognition and so on. Since Frechet distance is an effective metric for measuring curve similarity, a novel 3D face similarity measure method based on Frechet distances of geodesics is proposed in this paper. In our method, the surface similarity between two 3D faces is measured by the similarity between two sets of 3D curves on them. Due to the intrinsic property of geodesics, we select geodesics as the comparison curves. Firstly, the geodesics on each 3D facial model emanating from the nose tip point are extracted in the same initial direction with equal angular increment. Secondly, the Frechet distances between the two sets of geodesics on the two compared facial models are computed. At last, the similarity between the two facial models is computed based on the Frechet distances of the geodesics obtained in the second step. We verify our method both theoretically and practically. In theory, we prove that the similarity of our method satisfies three properties: reflexivity, symmetry, and triangle inequality. And in practice, experiments are conducted on the open 3D face database GavaDB, Texas 3D Face Recognition database, and our 3D face database. After the comparison with iso-geodesic and Hausdorff distance method, the results illustrate that our method has good discrimination ability and can not only identify the facial models of the same person, but also distinguish the facial models of any two different persons.

3D-3D facial superimposition between monozygotic twins: A novel morphological approach to the assessment of differences due to environmental factors.

Distinction of one twin with respect to the other, based on external appearance, is challenging; nevertheless, facial morphology may provide individualizing features that may help distinguish twin siblings. This study aims at exposing an innovative method for the facial assessment in monozygotic twins for personal identification, based on the registration and comparison of 3D models of faces. Ten couples of monozygotic twins aged between 25 and 69 years were acquired twice by a stereophotogrammetric system (VECTRA-3D® M3: Canfield Scientific, Inc., Fairfield, NJ); the 3D reconstruction of each person was then registered and superimposed onto the model belonging to the same person (self-matches), the corresponding sibling (twin-matches) and to unrelated participants from the other couples (miss-matches); RMS (root mean square) point-to-point distances were automatically calculated for all the 220 superimpositions. One-way ANOVA was used to evaluate the differences among miss-matches, twin-matches and self-matches (p < .05). RMS values for self-matches, twin-matches and miss-matches were respectively 1.0 mm (SD: 0.3 mm), 1.9 mm (0.5 mm) and 3.4 mm (0.70 mm). Statistically significant differences were found among the three groups (p < .01). Comparing RMS values in the three groups, mean facial variability in twin siblings was 55.9% of that assessed between unrelated persons and about twice higher than that observed between models belonging to the same individual. The present study proposed an innovative method for the facial assessment of twin siblings, based on 3D surface analysis, which may provide additional information concerning the relation between genes and environment.

HMM-based generation of laughter facial expression

This paper proposes a model for visual laughter generation by the means of speaker-dependent training of Hidden Markov Models (HMMs). It is composed of the following parts: 1) facial and 2) and head motions are modeled with separate HMMs while 3) eye-blink are added as a post-processing step on the generated eyelid trajectories.The models are trained on a database of facial expressions recorded on one male subject watching humorous videos. A commercially available marker-based motion capture system was used to record the visual data. A preliminary study has shown that modeling head motion with the same transcriptions as for facial deformation is not the best choice due to the rigidness of the resulting head motion.Finally, the generated facial laughter trajectories are used to animate a 3D face model and the corresponding animation is rendered in a video. An online perception MOS test is conducted to assess the improvement compared to the previous method and to compare with the perception of ground truth trajectories. Results show that the new approach significantly outperforms the previous one.

Annotated face model-based alignment: a robust landmark-free pose estimation approach for 3D model registration

Registering a 3D facial model onto a 2D image is important for constructing pixel-wise correspondences between different facial images. The registration is based on a 3 $$\times $$ 4 dimensional projection matrix, which is obtained from pose estimation. Conventional pose estimation approaches employ facial landmarks to determine the coefficients inside the projection matrix and are sensitive to missing or incorrect landmarks. In this paper, a landmark-free pose estimation method is presented. The method can be used to estimate the matrix when facial landmarks are not available. Experimental results show that the proposed method outperforms several landmark-free pose estimation methods and achieves competitive accuracy in terms of estimating pose parameters. The method is also demonstrated to be effective as part of a 3D-aided face recognition pipeline (UR2D), whose rank-1 identification rate is competitive to the methods that use landmarks to estimate head pose.

Face Alignment in Full Pose Range: A 3D Total Solution.

Face alignment, which fits a face model to an image and extracts the semantic meanings of facial pixels, has been an important topic in the computer vision community. However, most algorithms are designed for faces in small to medium poses (yaw angle is smaller than 45 degree), which lack the ability to align faces in large poses up to 90 degree. The challenges are three-fold. First, the commonly used landmark face model assumes that all the landmarks are visible and is therefore not suitable for large poses. Second, the face appearance varies more drastically across large poses, from the frontal view to the profile view. Third, labelling landmarks in large poses is extremely challenging since the invisible landmarks have to be guessed. In this paper, we propose to tackle these three challenges in an new alignment framework termed 3D Dense Face Alignment (3DDFA), in which a dense 3D Morphable Model (3DMM) is fitted to the image via Cascaded Convolutional Neural Networks. We also utilize 3D information to synthesize face images in profile views to provide abundant samples for training. Experiments on the challenging AFLW database show that the proposed approach achieves significant improvements over the state-of-the-art methods.

Learning a model of facial shape and expression from 4D scans

The field of 3D face modeling has a large gap between high-end and low-end methods. At the high end, the best facial animation is indistinguishable from real humans, but this comes at the cost of extensive manual labor. At the low end, face capture from consumer depth sensors relies on 3D face models that are not expressive enough to capture the variability in natural facial shape and expression. We seek a middle ground by learning a facial model from thousands of accurately aligned 3D scans. Our FLAME model (Faces Learned with an Articulated Model and Expressions) is designed to work with existing graphics software and be easy to fit to data. FLAME uses a linear shape space trained from 3800 scans of human heads. FLAME combines this linear shape space with an articulated jaw, neck, and eyeballs, pose-dependent corrective blendshapes, and additional global expression blendshapes. The pose and expression dependent articulations are learned from 4D face sequences in the D3DFACS dataset along with additional 4D sequences. We accurately register a template mesh to the scan sequences and make the D3DFACS registrations available for research purposes. In total the model is trained from over 33, 000 scans. FLAME is low-dimensional but more expressive than the FaceWarehouse model and the Basel Face Model. We compare FLAME to these models by fitting them to static 3D scans and 4D sequences using the same optimization method. FLAME is significantly more accurate and is available for research purposes (http://flame.is.tue.mpg.de).

Dynamic lip animation from a limited number of control points: Towards an effective audiovisual spoken communication

In audiovisual speech communication, the lower part of the face (mainly lips and jaw) actively participates during speech production. Modeling well lip motion and deformation in audiovisual speech synthesis is important to achieve realism and effective communication. This is essential for challenged population as hard-of-hearing people or new language learners. In this scope, we propose a technique that allows for animation of a human face with realistic lips using a limited number of control points. We have used an articulograph that provides high temporal and spatial precision, allowing tracking the positions of small electromagnetic sensors, even when occluded, which is often the case when tracking the lip movement. In our work, the control point data are first acquired, then fitted to a 3D face model of a human speaker, i.e., each control point is associated with a region of the face by minimizing the distance between the control points and the surface of the face model. Finally, we apply an interpolation scheme of the displacement field between the control points. This displacement field describes the deformation of a surface. In the case of the face, this method is well adapted to animating the region of the face that is highly correlated with speech, specifically the lips and the lower part of the face, even with a very limited number of control points.

Blending Face Details: Synthesizing a Face Using Multiscale Face Models.

Creating realistic 3D face models is a challenging problem in computer graphics because humans are so sensitive to facial abnormalities. The authors propose a method to synthesize a 3D face model using weighted blending of multiscale details from different face models. Using multiscale continuous displacement maps (CDMs), they achieve full correspondences across multiple scales in the parameter space. Their results demonstrate detail transfer across faces with highly different proportions, such as between humans and nonhuman creatures. An artist evaluation also indicated the proposed approach is intuitive and easy to use.

Gaussian mixture 3D morphable face model

3D Morphable Face Models (3DMM) have been used in pattern recognition for some time now. They have been applied as a basis for 3D face recognition, as well as in an assistive role for 2D face recognition to perform geometric and photometric normalisation of the input image, or in 2D face recognition system training. The statistical distribution underlying 3DMM is Gaussian. However, the single-Gaussian model seems at odds with reality when we consider different cohorts of data, e.g. Black and Chinese faces. Their means are clearly different. This paper introduces the Gaussian Mixture 3DMM (GM-3DMM) which models the global population as a mixture of Gaussian subpopulations, each with its own mean. The proposed GM-3DMM extends the traditional 3DMM naturally, by adopting a shared covariance structure to mitigate small sample estimation problems associated with data in high dimensional spaces. We construct a GM-3DMM, the training of which involves a multiple cohort dataset, SURREY-JNU, comprising 942 3D face scans of people with mixed backgrounds. Experiments in fitting the GM-3DMM to 2D face images to facilitate their geometric and photometric normalisation for pose and illumination invariant face recognition demonstrate the merits of the proposed mixture of Gaussians 3D face model.

3D face recognition: Multi-scale strategy based on geometric and local descriptors

Most human expression variations cause a non-rigid deformation of face scans, which is a challenge today. In this article, we present a novel framework for 3D face recognition that uses a geometry and local shape descriptor in a matching process to overcome the distortions caused by expressions in faces. This algorithm consists of four major components. First, the 3D face model is presented at different scales. Second, isometric-invariant features on each scale are extracted. Third, the geometric information is obtained on the 3D surface in terms of radial and level facial curves. Fourth, the feature vectors on each scale are concatenated with their corresponding geometric information. We conducted a number of experiments using two well-known and challenging datasets, namely, the GavabDB and Bosphorus datasets, and superior recognition performance has been achieved. The new system displays an overall rank-1 identification rate of 98.9% for all faces with neutral and non-neutral expressions on the GavabDB database.

Accuracy of generic mesh conformation: The future of facial morphological analysis

Three-dimensional (3D) analysis of the face is required for the assessment of changes following surgery, to monitor the progress of pathological conditions and for the evaluation of facial growth. Sophisticated methods have been applied for the evaluation of facial morphology, the most common being dense surface correspondence. The method depends on the application of a mathematical facial mask known as the generic facial mesh for the evaluation of the characteristics of facial morphology. This study evaluated the accuracy of the conformation of generic mesh to the underlying facial morphology. The study was conducted on 10 non-patient volunteers. Thirty-four 2-mm-diameter self-adhesive, non-reflective markers were placed on each face. These were readily identifiable on the captured 3D facial image, which was captured by Di3D stereophotogrammetry. The markers helped in minimising digitisation errors during the conformation process. For each case, the face was captured six times: at rest and at the maximum movements of four facial expressions. The 3D facial image of each facial expression was analysed. Euclidean distances between the 19 corresponding landmarks on the conformed mesh and on the original 3D facial model provided a measure of the accuracy of the conformation process. For all facial expressions and all corresponding landmarks, these distances were between 0.7 and 1.7 mm. The absolute mean distances ranged from 0.73 to 1.74 mm. The mean absolute error of the conformation process was 1.13 ± 0.26 mm. The conformation of the generic facial mesh is accurate enough for clinical trial proved to be accurate enough for the analysis of the captured 3D facial images.

Facial landmark localization by enhanced convolutional neural network

Facial landmark localization is important to many facial recognition and analysis tasks, such as face attributes analysis, head pose estimation, 3D face modeling, and facial expression analysis. In this paper, we propose a new approach to localizing landmarks in facial image by deep convolutional neural network (DCNN). We make two enhancements on the CNN to adapt it to the feature localization task as follows. First, we replace the commonly used max pooling by depth-wise convolution to obtain better localization performance. Second, we define a response map for each facial points as a 2D probability map indicating the presence likelihood, and train our model with a KL divergence loss. To obtain robust localization results, our approach first takes the expectations of the response maps of enhanced CNN and then applies auto-encoder model to the global shape vector, which is effective to rectify the outlier points by the prior global landmark configurations. The proposed ECNN method achieves 5.32% mean error on the experiments on the 300-W dataset, which is comparable to the state-of-the-art performance on this standard benchmark, showing the effectiveness of our methods.

Lighting-aware face frontalization for unconstrained face recognition

Provide both lighting-recovered and lighting-normalized frontalized images.Basic frontalization with a generic 3D face model by the alignment of only five landmarks.Lighting recovered and normalize...

Role for 2D image generated 3D face models in the rehabilitation of facial palsy.

The outcome for patients diagnosed with facial palsy has been shown to be linked to rehabilitation. Dense 3D morphable models have been shown within the computer vision to create accurate representations of human faces even from single 2D images. This has the potential to provide feedback to both the patient and medical expert dealing with the rehabilitation plan. It is proposed that a framework for the creation and measuring of patient facial movement consisting of a hybrid 2D facial landmark fitting technique which shows better accuracy in testing than current methods and 3D model fitting.

DeepSketch2Face

Face modeling has been paid much attention in the field of visual computing. There exist many scenarios, including cartoon characters, avatars for social media, 3D face caricatures as well as face-related art and design, where low-cost interactive face modeling is a popular approach especially among amateur users. In this paper, we propose a deep learning based sketching system for 3D face and caricature modeling. This system has a labor-efficient sketching interface, that allows the user to draw freehand imprecise yet expressive 2D lines representing the contours of facial features. A novel CNN based deep regression network is designed for inferring 3D face models from 2D sketches. Our network fuses both CNN and shape based features of the input sketch, and has two independent branches of fully connected layers generating independent subsets of coefficients for a bilinear face representation. Our system also supports gesture based interactions for users to further manipulate initial face models. Both user studies and numerical results indicate that our sketching system can help users create face models quickly and effectively. A significantly expanded face database with diverse identities, expressions and levels of exaggeration is constructed to promote further research and evaluation of face modeling techniques.

Using 3D face priors for depth recovery

For repairing inaccurate depth measurements from commodity RGB-D sensors, existing depth recovery methods primarily rely on low-level and rigid prior information. However, as the depth quality deteriorates, the recovered depth maps become increasingly unreliable, especially for non-rigid objects. Thus, additional high-level and non-rigid information is needed to improve the recovery quality. Taking as a starting point the human face that is the primary prior available in many high-level tasks, in this paper, we incorporate face priors into the depth recovery process. In particular, we propose a joint optimization framework that consists of two main steps: transforming the face model for better alignment and applying face priors for improved depth recovery. Face priors from both sparse and dense 3D face models are studied. By comparing with the baseline method on benchmark datasets, we demonstrate that the proposed method can achieve up to 23.8% improvement in depth recovery with more accurate face registrations, bringing inspirations to both non-rigid object modeling and analysis.

Facial feature point detection: A comprehensive survey

This paper presents a comprehensive survey of facial feature point detection with the assistance of abundant manually labeled images. Facial feature point detection favors many applications such as face recognition, animation, tracking, hallucination, expression analysis and 3D face modeling. Existing methods can be categorized into the following four groups: constrained local model (CLM)-based, active appearance model (AAM)-based, regression-based, and other methods. CLM-based methods consist of a shape model and a number of local experts, each of which is utilized to detect a facial feature point. AAM-based methods fit a shape model to an image by minimizing texture synthesis errors. Regression-based methods directly learn a mapping function from facial image appearance to facial feature points. Besides the above three major categories of methods, there are also minor categories of methods which we classify into other methods: graphical model-based methods, joint face alignment methods, independent facial feature point detectors, and deep learning-based methods. Though significant progress has been made, facial feature point detection is limited in its success by wild and real-world conditions: variations across poses, expressions, illuminations, and occlusions. A comparative illustration and analysis of representative methods provide us a holistic understanding and deep insight into facial feature point detection, which also motivates us to explore promising future directions.

3D facial shape reconstruction using macro- and micro-level features from high resolution facial images

Three-dimensional (3D) facial modeling and stereo matching-based methods are widely used for 3D facial reconstruction from 2D single-view and multiple-view images. However, these methods cannot realistically reconstruct 3D faces because they use insufficient numbers of macro-level Facial Feature Points (FFPs). This paper proposes an accurate and person-specific 3D facial reconstruction method that uses ample numbers of macro- and micro-level FFPs to enable coverage of all facial regions of high resolution facial images. Comparisons of 3D facial images reconstructed using the proposed method for ground-truth 3D facial images from the Bosphorus 3D database show that the method is superior to a conventional Active Appearance Model-Structure from Motion (AAM+SfM)-based method in terms of average 3D root mean square error between the reconstructed and ground-truth 3D faces. Further, the proposed method achieved outstanding accuracy in local facial regions such as the cheek—areas where extraction of FFPs is difficult for existing methods.