3D Face Research Articles

Face Modeling Based on Deep Learning and Traditional Methods: A Survey

Abstract. 3D face modeling, as an important research field in computer graphics and vision, has undergone remarkable development in recent years. The technology is widely used in various scenarios, like film and television production, virtual reality, augmented reality, facial recognition, security monitoring, and medical diagnosis. This paper aims to systematically explore the main technologies of 3D face modeling, especially the application and development of deep learning methods in it, review traditional face modeling methods, introduce in detail the latest technical advances based on deep learning, and analyze the advantages and disadvantages of these methods in practical applications.

The Validity of a Smartphone-Based Method for Acquiring 3D Images of the Face.

Objectives. To evaluate the accuracy and reproducibility of measurements obtained using the Bellus3D Face Application on a mobile smartphone by comparing them to direct measurements on pre-marked and blank face scans. Materials and Methods. Twenty-five healthy young adults (six males and nineteen females; age range 20-30 years) were included in this prospective cross-sectional study, with the only exclusion criterion being the presence of significant facial hair interfering with the placement and visualization of landmarks. Image acquisitions were performed using an iPhone XR with the Bellus3D FaceApp face scanning application, an iOS application for smartphones. Ten single midfacial and five paired bilateral landmarks were defined and marked. Two face scans were performed on each patient, both on blank and marked faces, and distances were measured directly with calipers and digitally. Results. The random error values were 1.0 mm and 0.4 mm for the manual point placement and measurements and virtual point placement on blank faces, respectively. The two methods used (the direct method and acquisition on faces with landmarks) demonstrate relatively similar reliability (ICC > 0.8); however, a paired t-test showed that the differences between several measurements were statistically significant (p < 0.05). Regardless of the method used, there was a systematic error for various values that included the nose and mouth (p < 0.05). The measurements demonstrating the most significant differences between the methods were those that included the tip of the nose, with the mean differences being -4.4-3.3 mm. The measurements of the distances that estimate face "depth" showed the greatest consistency irrespective of the tested method (p > 0.05 and ICC > 0.8). Conclusions. The use of the Bellus3D FaceApp is precise and reproducible for certain areas of the face, but digital reconstruction errors prohibit, for the time being, the use of this technology in everyday clinical practice. The noted discrepancies were consistent and more prevalent for specific areas such as the tip of the nose. Further investigations are required to determine other sources of error and for other smartphone-based applications released for 3D face image acquisitions.

ExpAvatar: High-Fidelity Avatar Generation of Unseen Expressions with 3D Face Priors

The reconstruction of dynamic head avatars has gained increasing significance, giving rise to various downstream applications such as visual dubbing and digital human creation. Despite recent advancements, generating novel, unseen expressions for a given identity remains challenging in concurrently achieving 1) accurate expression and consistent appearance and 2) high-quality and realistic faces. This paper introduces ExpAvatar, a novel approach crafted to address these challenges. ExpAvatar elaborately leverages the appearance consistency capabilities inherent in 3DMMs-based models along with the robust generalization ability of DDPMs-based models to alleviate appearance drift issues and enhance the generation of unseen expressions. Specifically, ExpAvatar introduces a Face Priors-conditioned Diffusion model (FPDiff) to inject 3D face priors into generation models through fine-tuning. Furthermore, a Face Priors-conditioned Catalyst (FPCatalyst) is employed to enhance the inference efficiency and generation quality. Moreover, we propose a unique confidence-based regularizer function to mitigate the effect of imperfect face-tracking estimates, thereby improving the quality of dynamic neural head avatars. Experimental results demonstrate that ExpAvatar surpasses current state-of-the-art solutions in generating unseen expressions, marking an advancement in the realm of dynamic head avatar synthesis. Code: https://github.com/yuangan/ExpAvatar .

MPF-GAN: an enhanced architecture for 3D face reconstruction

MPF-GAN: an enhanced architecture for 3D face reconstruction

Robotic frameless brain biopsy system enhanced by facial mesh registration

AbstractIn this study, a new approach is presented that eliminates stereotactic frameworks and the use of markers, offering an alternative to traditional brain biopsy systems. The classical biopsy operation involves the registration of magnetic resonance (MR) and computed tomography (CT) information taken from the patient at different times. Typically, the surgeon's planning information, which takes an average of 4 h on MR, is transferred to CT, and the surgical operation commences. However, this approach necessitates two separate acquisitions (MR and CT), adversely affecting patient comfort and increasing the workload. In the proposed system, it is recommended to register MR‐Depth camera data instead of MR‐CT registration. To achieve this, a 3D face pattern is obtained from the data received from the depth camera attached to the robot arm and overlapped with the mesh obtained by segmentation of the MR. It was observed that registration with sub‐millimeter precision was achieved using the CMFreg surface registration technique.

Multi-channel depth segmentation network based on 3D graph convolution algorithm and its application in point cloud segmentation

At the current stage, 3D graph convolutional algorithms face the following issues: (1) The problem of determining the neighborhood in graph convolution. (2) The issue of fusing features at different depths in deep graph convolutional algorithms. (3) The challenge of feature fusion and recognition in multi-path deep graph convolutional algorithms. Based on these, the paper "Multi-Path Deep Segmentation Network Based on 3D Graph Convolutional Algorithm and Its Application in Point Cloud Segmentation" is proposed. Inspired by visual computing theory, the neighborhood for 3D graph convolution is determined based on the receptive field theory, and a 3D graph convolutional algorithm based on visual selectivity is proposed; A single-link deep feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, it achieve the fusion of features at different depths learned by the graph convolutional algorithm by a pyramid learning method; A multi-link feature extraction algorithm for 3D graph convolution based on visual selectivity is proposed, and achieve multi-link feature fusion and segmentation to utilize the RPN calculation method. compared with algorithms such as PointNet, PointNet++, KPConv deform, 3D GCN, and My Algorithm, The segmentation performance and geometric invariance of the proposed algorithm are validated on the ShapeNetPart dataset and the custom Mortise_and_Tenon_DB dataset, and Experiments demonstrate that the proposed algorithm is correct and feasible, offers superior 3D point cloud segmentation performance, and exhibits strong geometric invariance.

Advanced 3D Face Reconstruction from Single 2D Images Using Enhanced Adversarial Neural Networks and Graph Neural Networks.

This paper presents a novel framework for 3D face reconstruction from single 2D images and addresses critical limitations in existing methods. Our approach integrates modified adversarial neural networks with graph neural networks to achieve state-of-the-art performance. Key innovations include (1) a generator architecture based on Graph Convolutional Networks (GCNs) with a novel loss function and identity blocks, mitigating mode collapse and instability; (2) the integration of facial landmarks and a non-parametric efficient-net decoder for enhanced feature capture; and (3) a lightweight GCN-based discriminator for improved accuracy and stability. Evaluated on the 300W-LP and AFLW2000-3D datasets, our method outperforms existing approaches, reducing Chamfer Distance by 62.7% and Earth Mover's Distance by 57.1% on 300W-LP. Moreover, our framework demonstrates superior robustness to variations in head positioning, occlusion, noise, and lighting conditions while achieving significantly faster processing times.

Pain assessment from facial expression images utilizing Statistical Frei-Chen Mask (SFCM)-based features and DenseNet

Estimating pain levels is crucial for patients with serious illnesses, those recovering from brain surgery, and those receiving intensive care etc. An automatic pain intensity estimator is proposed in this study that gathers information about pain and intensity from the user’s expressions. The faces in the database are first cropped using a ‘Chehra’ face detector, which performs well even in wildly uncontrolled environments with a wide range of lighting and position fluctuations. The suggested technique extracts the beneficial and distinct patterns from facial expressions using novel Statistical Frei-Chen Mask (SFCM)-based features and DenseNet-based features. As it offers quick as well as accurate pain identification and pain intensity estimation, the Radial Basis Function Based Extreme Learning Machine (RBF-ELM) is employed for pain recognition and pain intensity level estimation using the characteristics. All the data is kept, updated and protected in the cloud because availability and high-performance decision-making are so important for informing physicians and auxiliary IoT nodes (such as wearable sensors). In addition, cloud computing reduces the time complexity of the training phase of Machine Learning algorithms in situations where it is possible to build a complete cloud/edge architecture by allocating additional computational resources and memory in use. The facial expression images from the UNBC-McMaster Shoulder Pain Expression Archive and 2D face dataset are used to test the proposed method. The measurement of pain intensity uses four stages. When compared to the results from the literature, the proposed work attains enhanced performance.

Weakly Supervised 3D Face Reconstruction with Joint Spatial and Frequency Domain Information

3D face reconstruction is an important research direction in computer vision, and its goal is to recover a 3D face model from a single face picture. In the absence of real 3D face data, how to reconstruct a 3D face with a high degree of realism has become a hot research topic in recent years. Existing reconstruction algorithms usually rely on 3D labels generated from a large number of 2D face images as training data, however, inaccurate data will seriously affect the reconstruction quality. For this reason, the paper proposes a joint spatial-frequency domain decoupled weak supervision to achieve 3D face reconstruction, the main idea of which is to construct a multi-level loss function by using the weakly supervised information extracted from the spatial domain, and separating the frequency-domain information between the input and the rendered image in the frequency domain, and minimizing the difference between the two by difference computation. The method combines deep learning with 3D deformable models to reconstruct 3D models with high quality texture and shape from only a single face image. Quantitative experiments on the AFLW2000-3D and MICC Florence datasets show that the normalized average error in the small pose interval is as low as 2.42%, and the face reconstruction accuracy in the outdoor scene is 0.98 0.22 mm. Qualitative experiments on the MoFa-test, MICA datasets show that when faced with reconstruction with different poses, lighting, and expressions , our method outperforms other state-of-the-art reconstruction methods.

FaReT 2.1: Anatomically precise manipulation of race in 3D face models and a pipeline to import real face scans

FaReT 2.1: Anatomically precise manipulation of race in 3D face models and a pipeline to import real face scans

Face shape transfer via semantic warping

Face reshaping aims to adjust the shape of a face in a portrait image to make the face aesthetically beautiful, which has many potential applications. Existing methods 1) operate on the pre-defined facial landmarks, leading to artifacts and distortions due to the limited number of landmarks, 2) synthesize new faces based on segmentation masks or sketches, causing generated faces to look dissatisfied due to the losses of skin details and difficulties in dealing with hair and background blurring, and 3) project the positions of the deformed feature points from the 3D face model to the 2D image, making the results unrealistic because of the misalignment between feature points. In this paper, we propose a novel method named face shape transfer (FST) via semantic warping, which can transfer both the overall face and individual components (e.g., eyes, nose, and mouth) of a reference image to the source image. To achieve controllability at the component level, we introduce five encoding networks, which are designed to learn feature embedding specific to different face components. To effectively exploit the features obtained from semantic parsing maps at different scales, we employ a straightforward method of directly connecting all layers within the global dense network. This direct connection facilitates maximum information flow between layers, efficiently utilizing diverse scale semantic parsing information. To avoid deformation artifacts, we introduce a spatial transformer network, allowing the network to handle different types of semantic warping effectively. To facilitate extensive evaluation, we construct a large-scale high-resolution face dataset, which contains 14,000 images with a resolution of 1024 × 1024. Superior performance of our method is demonstrated by qualitative and quantitative experiments on the benchmark dataset.

Self-supervised reconstruction of re-renderable facial textures from single image

Reconstructing high-fidelity 3D facial texture from a single image is a quite challenging task due to the lack of complete face information and the domain gap between the 3D face and 2D image. Further, obtaining re-renderable 3D faces has become a strongly desired property in many applications, where the term ’re-renderable’ demands the facial texture to be spatially complete and disentangled with environmental illumination. In this paper, we propose a new self-supervised deep learning framework for reconstructing high-quality and re-renderable facial albedos from single-view images in the wild. Our main idea is to first utilize a prior generation module based on the 3DMM proxy model to produce an unwrapped texture and a globally parameterized prior albedo. Then we apply a detail refinement module to synthesize the final texture with both high-frequency details and completeness. To further make facial textures disentangled with illumination, we propose a novel detailed illumination representation that is reconstructed with the detailed albedo together. We also design several novel regularization losses on both the albedo and illumination maps to facilitate the disentanglement of these two factors. Finally, by leveraging a differentiable renderer, each face attribute can be jointly trained in a self-supervised manner without requiring ground-truth facial reflectance. Extensive comparisons and ablation studies on challenging datasets demonstrate that our framework outperforms state-of-the-art approaches.

Speed-Aware Audio-Driven Speech Animation using Adaptive Windows

We present a novel method that can generate realistic speech animations of a 3D face from audio using multiple adaptive windows. In contrast to previous studies that use a fixed size audio window, our method accepts an adaptive audio window as input, reflecting the audio speaking rate to use consistent phonemic information. Our system consists of three parts. First, the speaking rate is estimated from the input audio using a neural network trained in a self-supervised manner. Second, the appropriate window size that encloses the audio features is predicted adaptively based on the estimated speaking rate. Another key element lies in the use of multiple audio windows of different sizes as input to the animation generator: a small window to concentrate on detailed information and a large window to consider broad phonemic information near the center frame. Finally, the speech animation is generated from the multiple adaptive audio windows. Our method can generate realistic speech animations from in-the-wild audios at any speaking rate, i.e., fast raps, slow songs, as well as normal speech. We demonstrate via extensive quantitative and qualitative evaluations including a user study that our method outperforms state-of-the-art approaches.

Learning facial structural dependency in 3D aligned space for face alignment

Facial structure's statistical characteristics offer pivotal prior information in facial landmark prediction, forming inter-dependencies among different landmarks. Such inter-dependencies ensure that predictions adhere to the shape distribution typical of natural faces. In challenging scenarios like occlusions or extreme facial poses, this structure becomes indispensable, which can help to predict elusive landmarks based on more discernible ones. While current deep learning methods do capture these landmark dependencies, it's often an implicit process heavily reliant on vast training datasets. We contest that such implicit modeling approaches fail to manage more challenging situations. In this paper, we propose a new method that harnesses the facial structure and explicitly explores inter-dependencies among facial landmarks in an end-to-end fashion. We propose a Structural Dependency Learning Module (SDLM). It uses 3D face information to map facial features into a canonical UV space, in which the facial structure is explicitly 3D semantically aligned. Besides, to explore the global relationships between facial landmarks, we take advantage of the self-attention mechanism in the image and UV spaces. We name the proposed method Facial Structure-based Face Alignment (FSFA). FSFA reinforces the landmark structure, especially under challenging conditions. Extensive experiments demonstrate that FSFA achieves state-of-the-art performance on the WFLW, 300W, AFLW, and COFW68 datasets.

Maskrenderer: 3D-infused multi-mask realistic face reenactment

We present a novel end-to-end identity-agnostic face reenactment system, MaskRenderer, that can generate realistic, high fidelity frames in real-time. Although recent face reenactment works have shown promising results, there are still significant challenges such as identity leakage and imitating mouth movements, especially for large pose changes and occluded faces. MaskRenderer tackles these problems by using (i) a 3DMM to model 3D face structure to better handle pose changes, occlusion, and mouth movements compared to 2D representations; (ii) a triplet loss function to embed the cross-reenactment during training for better identity preservation; and (iii) multi-scale occlusion, improving inpainting and restoring missing areas. Comprehensive quantitative and qualitative experiments conducted on the VoxCeleb1 test set, demonstrate that MaskRenderer outperforms state-of-the-art models on unseen faces, especially when the Source and Driving identities are very different.

Anti Face Spoofing System

Face spoofing, the act of using deceptive techniques such as printed photos or digital images to deceive facial recognition systems, poses a significant threat to the security of various applications, including biometric authentication and access control systems. This paper presents a concise yet effective approach to address the challenge of anti-face spoofing using Python within a limited codebase of 50 lines. The proposed solution leverages a combination of image processing techniques and machine learning algorithms to detect and prevent face spoofing attempts. A pre-trained deep neural network model for facial recognition is employed to extract essential facial features. Subsequently, the system utilizes image manipulation detection methods to identify anomalies indicative of face spoofing attacks. The implementation showcases the simplicity and efficiency of the proposed anti-face spoofing technique, demonstrating the potential for integration into real- world applications with minimal computational overhead. The concise Python code enables easy adoption and adaptation for developers aiming to enhance the security of facial recognition systems against face spoofing threats. The experimental results demonstrate the effectiveness of the approach in accurately distinguishing between genuine and spoofed facial images, thereby contributing to the robustness of facial recognition systems in the presence of adversarial attacks. Keywords - Liveness Detection, Presentation Attack Detection, Spoof Detection Algorithms, Biometric Security, Face Recognition Robustness, 3D Face Modeling, Texture Analysis, Deep Learning Anti- Spoofing, Multi-Spectral Imaging, Behavioral Biometrics, Challenge-Response Mechanisms, Optical Flow Analysis, Pulse Detection, Surface Reflectance, Depth Sensing, Motion Analysis, Pattern Recognition, Anti-Spoofing Networks, Temporal Feature Extraction, Anomaly Detection.

PointSurFace: Discriminative point cloud surface feature extraction for 3D face recognition

Due to the geometric information in the 3D face data, the 3D face recognition methods exhibit better robustness against the physical attacks compared to the 2D recognition methods. In this paper, we propose PointSurFace, a 3D face point cloud recognition method integrated with a specially designed facial surface feature extraction (SurF) module. In PointSurFace, SurF comprehensively combines the coordinate positions, normal vectors, and angle relationships, etc., of each patch in the 3D face point cloud, to explore the explicit local geometric information of 3D faces and generate facial surface features. Subsequently, the surface features are fed into an encoder consisting of four set abstraction (SA) modules and three Inverted Residual MLP (InvResMLP) modules to extract more discriminative 3D facial features, where InvResMLP could suppress the overfitting and reduce information loss in the model training. In addition, PointSurFace utilizes channel fusion operations to integrate position information into the extracted 3D facial features, further enhancing the model performance. Experimental results in 3D facial recognition and verification across multiple datasets demonstrate that PointSurFace achieves the best 3D face recognition performance to date. For example, on Lock3DFace, it achieves the state-of-the-art recognition and verification accuracy of 90.03% and 80.01%, respectively, surpassing the previous methods by 2.02% and 3.19%. The ablation studies evaluate each module of PointSurFace, suggesting that the proposed surface feature extraction module significantly enhances the model performance.

MeshWGAN: Mesh-to-Mesh Wasserstein GAN With Multi-Task Gradient Penalty for 3D Facial Geometric Age Transformation.

As the metaverse develops rapidly, 3D facial age transformation is attracting increasing attention, which may bring many potential benefits to a wide variety of users, e.g., 3D aging figures creation, 3D facial data augmentation and editing. Compared with 2D methods, 3D face aging is an underexplored problem. To fill this gap, we propose a new mesh-to-mesh Wasserstein generative adversarial network (MeshWGAN) with a multi-task gradient penalty to model a continuous bi-directional 3D facial geometric aging process. To the best of our knowledge, this is the first architecture to achieve 3D facial geometric age transformation via real 3D scans. As previous image-to-image translation methods cannot be directly applied to the 3D facial mesh, which is totally different from 2D images, we built a mesh encoder, decoder, and multi-task discriminator to facilitate mesh-to-mesh transformations. To mitigate the lack of 3D datasets containing children's faces, we collected scans from 765 subjects aged 5-17 in combination with existing 3D face databases, which provided a large training dataset. Experiments have shown that our architecture can predict 3D facial aging geometries with better identity preservation and age closeness compared to 3D trivial baselines. We also demonstrated the advantages of our approach via various 3D face-related graphics applications.

SketchMetaFace: A Learning-Based Sketching Interface for High-Fidelity 3D Character Face Modeling.

Modeling 3D avatars benefits various application scenarios such as AR/VR, gaming, and filming. Character faces contribute significant diversity and vividity as a vital component of avatars. However, building 3D character face models usually requires a heavy workload with commercial tools, even for experienced artists. Various existing sketch-based tools fail to support amateurs in modeling diverse facial shapes and rich geometric details. In this article, we present SketchMetaFace - a sketching system targeting amateur users to model high-fidelity 3D faces in minutes. We carefully design both the user interface and the underlying algorithm. First, curvature-aware strokes are adopted to better support the controllability of carving facial details. Second, considering the key problem of mapping a 2D sketch map to a 3D model, we develop a novel learning-based method termed "Implicit and Depth Guided Mesh Modeling" (IDGMM). It fuses the advantages of mesh, implicit, and depth representations to achieve high-quality results with high efficiency. In addition, to further support usability, we present a coarse-to-fine 2D sketching interface design and a data-driven stroke suggestion tool. User studies demonstrate the superiority of our system over existing modeling tools in terms of the ease to use and visual quality of results. Experimental analyses also show that IDGMM reaches a better trade-off between accuracy and efficiency.

Self-supervised learning for fine-grained monocular 3D face reconstruction in the wild

Self-supervised learning for fine-grained monocular 3D face reconstruction in the wild