3D Face Shape Research Articles

3D face shape prediction from a frontal image using cylindrical coordinates and partial least squares

This paper addresses the problem of linearly approximating 3D shape from intensities in the context of facial analysis. In other words, given a frontal pose grayscale input face, the direct estimation of its 3D structure is sought through a regression matrix. Approaches falling into this category generally assume that both 2D and 3D features are defined under Cartesian schemes, which is not optimal for the task of novel view synthesis. The current article aims to overcome this issue by exploiting the 3D structure of faces through cylindrical coordinates, aided by the partial least squares regression. In the context of facial shape analysis, partial least squares builds a set of basis faces, for both grayscale and 3D shape spaces, seeking for maximizing shared covariance between projections of the data along the basis faces. Experimental tests show how the cylindrical representations are suitable for the purposes of linear regression, resulting in a benefit for the generation of novel facial views, showing a potential use in model based face identification.

Video-to-video face authentication system robust to pose variations

High-quality still-to-still (image-to-image) face authentication has shown success under controlled conditions in many safety applications. However, video-to-video face authentication is still challenging due to appearance variations caused by pose changes. In this paper, we propose a video-to-video face authentication system that is robust to pose variations by making use of synthesized frontal face appearance that contains both texture and shape information. To obtain the appearance, we first reconstruct 3D face shape from face feature points detected from the video using active shape model (ASM). Conventional ASM algorithms cannot handle large pose variations and fast head movement exhibited in video sequences. To address these problems, we present a novel prediction-assisted approach that is capable of providing an accurate shape initiation as well as automatically switching on multi-view models for ASM. Then we can generate frontal shape mesh from the reconstructed 3D face shape. Based on the mesh, we synthesize frontal face appearance with the ASM-detected faces in video. For authentication, in order to match the synthesized appearances of enrollment and probe, we propose a 2-directional 2-dimensional client specific fisher’s linear discriminant algorithm. The proposed algorithm is a variant of fisher’s linear discriminant (FLD) and directly computes eigenvectors of image scatter matrices in row and column direction without matrix-to-vector conversion. In experiments, our authentication system is compared with the other state-of-art approaches on public face database and our face database. The results show that our system demonstrates a higher authentication accuracy and pose-robust performance.

Single view-based 3D face reconstruction robust to self-occlusion

Abstract State-of-the-art 3D morphable model (3DMM) is used widely for 3D face reconstruction based on a single image. However, this method has a high computational cost, and hence, a simplified 3D morphable model (S3DMM) was proposed as an alternative. Unlike the original 3DMM, S3DMM uses only a sparse 3D facial shape, and therefore, it incurs a lower computational cost. However, this method is vulnerable to self-occlusion due to head rotation. Therefore, we propose a solution to the self-occlusion problem in S3DMM-based 3D face reconstruction. This research is novel compared with previous works, in the following three respects. First, self-occlusion of the input face is detected automatically by estimating the head pose using a cylindrical head model. Second, a 3D model fitting scheme is designed based on selected visible facial feature points, which facilitates 3D face reconstruction without any effect from self-occlusion. Third, the reconstruction performance is enhanced by using the estimated pose as the initial pose parameter during the 3D model fitting process. The experimental results showed that the self-occlusion detection had high accuracy and our proposed method delivered a noticeable improvement in the 3D face reconstruction performance compared with previous methods.

REPRESENTING 3D FACE FROM POINT CLOUD TO FACE-ALIGNED SPHERICAL DEPTH MAP

We propose a novel representation of 3D face shape which is a key step for feature extraction and face recognition. The input of the proposed methods is unstructured point cloud, which determines the wide applicability of the proposed representation. Our contributions mainly include two parts: Spherical Depth Map (SDM) and face alignment based on SDM. SDM, which can be adopted to many applications, is a special kind of range image utilizing the prior anatomical knowledge of human face. Useful characteristics of SDM facilitate face alignment with higher efficiency and accuracy. Experiments conducted on three popular 3D face databases verify the high efficacy and superiority of the proposed method. The accuracy of face alignment is up to 100% with our strategy. The face verification rates based on the standard protocols are all higher than the baseline performance of FRGC2.0.

Dynamic Cultural Representations of Facial Expressions of Emotion are not Universal

Six ‘universal’ facial expressions – ‘Happy,’ ‘Surprise,’ ‘Fear,’ ‘Disgust,’ ‘Anger,’ and ‘Sadness’ – are defined by specific, static patterns of facial muscle activation (Facial Action Coding System codes, FACS). However, systematic differences in facial expression recognition between Western Caucasians (WC) and East Asians (EA) question the notion of universality, raising a new question: How do different cultures represent facial expressions? Here, we derived culture-specific models of facial expressions using state-of-the-art 4D imaging (dynamics of 3D face shape and texture) combined with reverse correlation techniques. Specifically, we modelled 41 core Action Units (AUs, groups of facial muscles) from certified FACS coders and parameterized each using 6 temporal parameters (peak amplitude; peak latency; onset latency; offset latency; acceleration; deceleration). The 41 AUs and their parameters formed the basis of a pseudo-random generative model of expressive signals. On each trial, we pseudo-randomly selected parametric values for each AU, producing an expressive facial animation (see Figure S1 in Supplementary Material). Ten WC and 10 EA na & iuml;ve observers each categorized 9,600 such animations according to the 6 emotion categories listed above and rated the perceived intensity of the emotion (see Figure S1 in Supplementary Material). We then reverse correlated the dynamic properties of the AUs with the emotion categories they elicited, producing “dynamic classification models” (i.e., expected 4D face information) per emotion and observer. Analyses of the models reveal clear cultural contrasts in (a) the presence/absence of specific AUs predicting the reported EA miscategorizations and (b) radically different temporal dynamics of emotional expression whereby EA observers expect “smoother” emotional displays with lower acceleration and amplitude (see link in Supplementary Material for example videos). For the first time, we reveal cultural diversity in the dynamic signals representing each basic emotion, demonstrating that the complexities of emotion cannot adequately be reduced to a single set of static ‘universal’ signals.

Expression flow for 3D-aware face component transfer

We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending using existing compositing tools results in semantically unnatural composites, since expression is a global effect and the local component in one expression is often incompatible with the shape and other components of the face in another expression. To solve this problem we present Expression Flow, a 2D flow field which can warp the target face globally in a natural way, so that the warped face is compatible with the new facial component to be copied over. To do this, starting with the two input face photos, we jointly construct a pair of 3D face shapes with the same identity but different expressions. The expression flow is computed by projecting the difference between the two 3D shapes back to 2D. It describes how to warp the target face photo to match the expression of the reference photo. User studies suggest that our system is able to generate face composites with much higher fidelity than existing methods.

Smooth adaptive fitting of 3D face model for the estimation of rigid and nonrigid facial motion in video sequences

We propose a 3D wireframe face model alignment for the task of simultaneously tracking of rigid head motion and nonrigid facial expressions in video sequences. The system integrates two levels: (i) at the low level, automatic and accurate location of facial features are obtained via a cascaded optimization algorithm of a 2D shape model, (ii) at the high level, we recover, via minimizing an energy function, the optimal motion parameters of the 3D model, namely the 3D rigid motion parameters and seven nonrigid animation (Action Unit) parameters. In this latter inference, a 3D face shape model (Candide) is automatically fitted to the image sequence via a least squares minimization of the energy, defined as the residual between the projected 3D wireframe model and the 2D shape model, meanwhile imposing temporal and spatial motion-smoothness constraints over the 3D model points. Our proposed system tackles many disadvantages of the optimization and training associated with active appearance models. Extensive fitting and tracking experiments demonstrate the feasibility, accuracy and effectiveness of the developed methods. Qualitative and quantitative performance of the proposed system on several facial sequences, indicate its potential usefulness for multimedia applications, as well as facial expression analysis.

Three-dimensional gender differences in facial form of children in the North East of England

The aim of the prospective cross-sectional morphometric study was to explore three dimensional (3D) facial shape and form (shape plus size) variation within and between 8- and 12-year-old Caucasian children; 39 males age-matched with 41 females. The 3D images were captured using a stereophotogrammeteric system, and facial form was recorded by digitizing 39 anthropometric landmarks for each scan. The x, y, z coordinates of each landmark were extracted and used to calculate linear and angular measurements. 3D landmark asymmetry was quantified using Generalized Procrustes Analysis (GPA) and an average face was constructed for each gender. The average faces were superimposed and differences were visualized and quantified. Shape variations were explored using GPA and PrincipalComponent Analysis. Analysis of covariance and Pearson correlation coefficients were used to explore gender differences and to determine any correlation between facial measurements and height or weight. Multivariate analysis was used to ascertain differences in facial measurements or 3D landmark asymmetry. There were no differences in height or weight between genders. There was a significant positive correlation between facial measurements and height and weight and statistically significant differences in linear facial width measurements between genders. These differences were related to the larger size of males rather than differences in shape. There were no age- or gender-linked significant differences in 3D landmark asymmetry. Shape analysis confirmed similarities between both males and females for facial shape and form in 8- to 12-year-old children. Any differences found were related to differences in facial size rather than shape.

얼굴 깊이 추정을 이용한 3차원 얼굴 생성 및 추적 방법

얼굴의 3차원 정보는 얼굴 인식이나 얼굴 합성, Human Computer Interaction (HCI) 등 다양한 분야에서 유용하게 이용될 수 있다. 그러나 일반적으로 3차원 정보는 3D 스캐너와 같은 고가의 장비를 이용하여 획득되기 때문에 얼굴의 3차원 정보를 얻기 위해서는 많은 비용이 요구된다. 본 논문에서는 일반적으로 손쉽게 얻을 수 있는 2차원의 얼굴 영상 시퀀스로부터 효과적으로 3차월 얼굴 형태를 추적하고 재구성하기 위한 3차원 Active Appearance Model (3D-AAM) 방법을 제안한다. 얼굴의 3차원 변화 정보를 추정하기 위해 학습 영상은 정면 얼굴 포즈로 다양한 얼굴 표정 변화를 포함한 영상과 표정 변화를 갖지 않으면서 서로 크게 다른 얼굴 포즈를 갖는 영상으로 구성한다. 입력 영상의 3차원 얼굴 변화를 추정하기 위해 먼저 서로 다른 포즈를 갖는 학습 영상으로부터 얼굴의 각 특징점(Land-mark)의 기하학적 변화를 이용하여 깊이 정보를 추정하고 추정된 특징점의 깊이 정보를 입력 영상의 2차원 얼굴 변화에 추가하여 최종적으로 입력 얼굴의 3차원 변화를 추정한다. 본 논문에서 제안된 방법은 얼굴의 다양한 표정 변화와 함께 3차원의 얼굴 포즈 변화를 포함한 실험 영상을 이용하여 기존의 AAM에 비해 효과적이면서 빠르게 입력 얼굴을 추적(Fitting)할 수 있으며 입력 영상의 정확한 3차원 얼굴 형태를 생성할 수 있음을 보였다. A 3D face shape derived from 2D images may be useful in many applications, such as face recognition, face synthesis and human computer interaction. To do this, we develop a fast 3D Active Appearance Model (3D-AAM) method using depth estimation. The training images include specific 3D face poses which are extremely different from one another. The landmark's depth information of landmarks is estimated from the training image sequence by using the approximated Jacobian matrix. It is added at the test phase to deal with the 3D pose variations of the input face. Our experimental results show that the proposed method can efficiently fit the face shape, including the variations of facial expressions and 3D pose variations, better than the typical AAM, and can estimate accurate 3D face shape from images.

A SfM-based 3D face reconstruction method robust to self-occlusion by using a shape conversion matrix

This paper presents a 3D face reconstruction method using multiple 2D face images. Structure from motion (SfM) methods, which have been widely used to reconstruct 3D faces, are vulnerable to point correspondence errors caused by self-occlusion. In order to solve this problem, we propose a shape conversion matrix (SCM) which estimates the ground-truth 2D facial feature points (FFPs) from the observed 2D FFPs corrupted by self-occlusion errors. To make the SCM, the training observed 2D FFPs and ground-truth 2D FFPs are collected by using 3D face scans. An observed shape model and a ground-truth shape model are then built to represent the observed 2D FFPs and the ground-truth 2D FFPs, respectively. Finally, the observed shape model parameter is converted to the ground truth shape model parameter via the SCM. By using the SCM, the true locations of the self-occluded FFPs are estimated exactly with simple matrix multiplications. As a result, SfM-based 3D face reconstruction methods combined with the proposed SCM become more robust against point correspondence errors caused by self-occlusion, and the computational cost is significantly reduced. In experiments, the reconstructed 3D facial shape is quantitatively compared with the 3D facial shape obtained from a 3D scanner, and the results show that SfM-based 3D face reconstruction methods with the proposed SCM show a higher accuracy and a faster processing time than SfM-based 3D face reconstruction methods without the SCM.

Objective 3D face recognition: Evolution, approaches and challenges

Face recognition is a natural human ability and a widely accepted identification and authentication method. In modern legal settings, a lot of credence is placed on identifications made by eyewitnesses. Consequently these are based on human perception which is often flawed and can lead to situations where identity is disputed. Therefore, there is a clear need to secure identifications in an objective way based on anthropometric measures. Anthropometry has existed for many years and has evolved with each advent of new technology and computing power. As a result of this, face recognition methodology has shifted from a purely 2D image-based approach to the use of 3D facial shape. However, one of the main challenges still remaining is the non-rigid structure of the face, which can change permanently over varying time-scales and briefly with facial expressions. The majority of face recognition methods have been developed by scientists with a very technical background such as biometry, pattern recognition and computer vision. This article strives to bridge the gap between these communities and the forensic science end-users. A concise review of face recognition using 3D shape is given. Methods using 3D shape applied to data embodying facial expressions are tabulated for reference. From this list a categorization of different strategies to deal with expressions is presented. The underlying concepts and practical issues relating to the application of each strategy are given, without going into technical details. The discussion clearly articulates the justification to establish archival, reference databases to compare and evaluate different strategies.

Representation of 3D face shape and 2D surface reflectance in the ventral temporal cortex

Representation of 3D face shape and 2D surface reflectance in the ventral temporal cortex

Holistic processing of diagnostic 3D face shape as compared to 2D surface reflectance: evidence from face inversion and acquired prosopagnosia

Holistic processing of diagnostic 3D face shape as compared to 2D surface reflectance: evidence from face inversion and acquired prosopagnosia

An experimental illustration of 3D facial shape analysis under facial expressions

The main goal of this paper is to illustrate a geometric analysis of 3D facial shapes in the presence of varying facial expressions. This approach consists of the following two main steps: (1) Each facial surface is automatically denoised and preprocessed to result in an indexed collection of facial curves. During this step, one detects the tip of the nose and defines a surface distance function with that tip as the reference point. The level curves of this distance function are the desired facial curves. (2) Comparisons between faces are based on optimal deformations from one to another. This, in turn, is based on optimal deformations of the corresponding facial curves across surfaces under an elastic metric. The experimental results, generated using a subset of the Face Recognition Grand Challenge v2 data set, demonstrate the success of the proposed framework in recognizing people under different facial expressions. The recognition rates obtained here exceed those for a baseline ICP algorithm on the same data set.

３次元映像情報メディア技術 フリーハンド撮影された多視点スナップショットからの顔形状復元

We propose a method that can reconstruct both a 3D facial shape and camera poses from freehand multi-viewpoint snapshots. This method is based on Active Shape Model (ASM) using a facial shape database. Most ASM methods require an image in which the camera pose is known, but our method does not require this information. First, we choose an initial shape by selecting the model from the database which is most suitable to input images. Then, we improve the model by morphing it to fit the input images. Next, we estimate the camera poses using the morphed model. Finally we repeat the process, improving both the facial shape and the camera poses until the error between the input images and the computed result is minimized. Through experimentation, we show that our method reconstructs the facial shape within 3.5 mm of the ground truth.

Relating intensities with three-dimensional facial shape using partial least squares

The authors apply partial least squares regression to predict three-dimensional (3D) face shape from a single image. PLS describes the relationship between independent (intensity images) and dependent (3D shape) variables by seeking directions in the space of independent variables that are associated with large variations in the space of dependent variables. We use this idea to construct statistical models of intensity and 3D shape that capture strongly linked variations in both spaces. This decomposition leads to the construction of two different models that capture common variations in 3D shape and intensity. Using the intensity model, a set of parameters is obtained from out-of-training intensity examples. These intensity parameters can then be used directly in the 3D shape model to approximate facial shape. Experiments show that prediction is achieved with reasonable accuracy, improving results obtained through canonical correlation analysis.

Lip-Synching Using Speaker-Specific Articulation, Shape and Appearance Models

We describe here the control, shape and appearance models that are built using an original photogrammetric method to capture characteristics of speaker-specific facial articulation, anatomy, and texture. Two original contributions are put forward here: the trainable trajectory formation model that predicts articulatory trajectories of a talking face from phonetic input and the texture model that computes a texture for each 3D facial shape according to articulation. Using motion capture data from different speakers and module-specific evaluation procedures, we show here that this cloning system restores detailed idiosyncrasies and the global coherence of visible articulation. Results of a subjective evaluation of the global system with competing trajectory formation models are further presented and commented.

Reconstructing 3D Shape, Albedo and Illumination from a Single Face Image

AbstractThe morphable model has been employed to efficiently describe 3D face shape and the associated albedo with a reduced set of basis vectors. The spherical harmonics (SH) model provides a compact basis to well approximate the image appearance of a Lambertian object under different illumination conditions. Recently, the SH and morphable models have been integrated for 3D face shape reconstruction. However, the reconstructed 3D shape is either inconsistent with the SH bases or obtained just from landmarks only. In this work, we propose a geometrically consistent algorithm to reconstruct the 3D face shape and the associated albedo from a single face image iteratively by combining the morphable model and the SH model. The reconstructed 3D face geometry can uniquely determine the SH bases, therefore the optimal 3D face model can be obtained by minimizing the error between the input face image and a linear combination of the associated SH bases. In this way, we are able to preserve the consistency between the 3D geometry and the SH model, thus refining the 3D shape reconstruction recursively. Furthermore, we present a novel approach to recover the illumination condition from the estimated weighting vector for the SH bases in a constrained optimization formulation independent of the 3D geometry. Experimental results show the effectiveness and accuracy of the proposed face reconstruction and illumination estimation algorithm under different face poses and multiple‐light‐source illumination conditions.

3D 변형가능 형상 모델 기반 3D 얼굴 모델링

3D 얼굴 모델링은 33공간에서 얼굴을 자유롭게 회전 시켜 다양한 얼굴 자세를 표현하고 조명 효과도 적절하게 모델링 할 수 있으므로, 얼굴 자세, 조명, 표정 등의 표현에 있어서 2D 얼굴 모델링에 비해 보다 정교하며 사실감이 뛰어나 얼굴 인식, 게임, 아바타 등에서 많은 요구가 존재한다. 본 논문에서는 3D 변형 가능 형상 모델에 기반을 둔 3D 얼굴 모델링 방법을 제안한다. 제안된 3D 얼굴 모델링 방법은 먼저 3D 스캐너를 통하여 획득한 3D 얼굴 스캔 데이터를 이용하여 3D 얼굴 변형 가능 형상 모델을 구축한다. 다음, 3D 얼굴 모델링을 하고자 하는 얼굴의 2D 이미지 시퀀스로부터, 해당 얼굴의 특징점들을 검출하고 이들을 매칭하여, 매칭된 특징점들로부터 인수분해 기반 SfM 기법을 이용하여 해당 특징점의 3D 버텍스 좌표 값을 구한다. 이후, 구한 3D 버텍스들을 3D 변형 가능 형상 모델에 정합하여 해당 얼굴의 3D 형상 모델을 얻는다. 또한, 2D 얼굴 이미지 시퀀스들로부터 뷰 독립적인 2D 원통 좌표 텍스쳐 맵을 구하고 이를 이용하여 3D 형상 모델을 렌더링 함으로써, 최종적으로 3B 얼굴 모델을 완성한다. 제안된 3D 얼굴 모델링 방법에 의한 3D 얼굴 모델 생성 과정을 통해서, 본 논문에서 제안한 3D 얼굴 모델링 방법이 기존의 얼굴 모델링 방법들에 비해 상대적으로 빠르고 비교적 정교하게 수행됨을 볼 수 있었다. Since 3D face can be rotated freely in 3D space and illumination effects can be modeled properly, 3D face modeling Is more precise and realistic in face pose, illumination, and expression than 2D face modeling. Thus, 3D modeling is necessitated much in face recognition, game, avatar, and etc. In this paper, we propose a 3D face modeling method based on 3D morphable shape modeling. The proposed 3D modeling method first constructs a 3D morphable shape model out of 3D face scan data obtained using a 3D scanner Next, the proposed method extracts and matches feature points of the face from 2D image sequence containing a face to be modeled, and then estimates 3D vertex coordinates of the feature points using a factorization based SfM technique. Then, the proposed method obtains a 3D shape model of the face to be modeled by fitting the 3D vertices to the constructed 3D morphable shape model. Also, the proposed method makes a cylindrical texture map using 2D face image sequence. Finally, the proposed method builds a 3D face model by rendering the 3D face shape model with the cylindrical texture map. Through building processes of 3D face model by the proposed method, it is shown that the proposed method is relatively easy, fast and precise than the previous 3D face model methods.

The use of 3D face shape modelling in dysmorphology

Facial appearance can be a significant clue in the initial identification of genetic conditions, but their low incidence limits exposure during training and inhibits the development of skills in recognising...