3D Face Reconstruction Research Articles

Transformer-Based High-Fidelity Facial Displacement Completion for Detailed 3D Face Reconstruction

In this paper, we tackle a special face completion task, facial displacement completion, which can offer a key component for many single image 3D face reconstruction systems. To produce a detailed 3D face with ear-to-ear complete displacement UV map, we propose a novel Displacement Completion method based on Transformer (DCT). Current transformer based image inpainting methods usually follow a two-stage scheme, which firstly recovers the masked pixels in low resolution with transformer, and then replenishes the inpainting result in high resolution with GAN. Although these methods have achieved great success, they suffer from information loss from two aspects when applied in face completion: 1) The downsampling operation makes transformer only produce a coarse appearance prior for GAN, incurring middle and low level information loss. 2) Some meaningful facial semantics can be well captured by transformer and further benefit the completion, but it's has not yet been explored. Motivated by the above considerations, we come up with three key designs in the proposed DCT: PCA tokenization, BERT-style learning, and style modulation. Firstly, we use PCA tokenization to replace the downsampling in transformer to preserve more meaningful structures. Secondly, we make transformer simulate the two tasks in BERT, Masked Language Model (MLM) and Next Sentence Prediction (NSP), for both masked pixels and facial attributes recovery. Thirdly, we encode the outcome of transformer as the latent code to guide an image translation network in the StyleGAN2 modulation way. Experments on both FaceScape dataset and in-the-wild data demonstrate DCT's better performance compared with other transformer based or GAN based completion methods.

ReenactArtFace: Artistic Face Image Reenactment.

Large-scale datasets and deep generative models have enabled impressive progress in human face reenactment. Existing solutions for face reenactment have focused on processing real face images through facial landmarks by generative models. Different from real human faces, artistic human faces (e.g., those in paintings, cartoons, etc.) often involve exaggerated shapes and various textures. Therefore, directly applying existing solutions to artistic faces often fails to preserve the characteristics of the original artistic faces (e.g., face identity and decorative lines along face contours) due to the domain gap between real and artistic faces. To address these issues, we present ReenactArtFace, the first effective solution for transferring the poses and expressions from human videos to various artistic face images. We achieve artistic face reenactment in a coarse-to-fine manner. First, we perform 3D artistic face reconstruction, which reconstructs a textured 3D artistic face through a 3D morphable model (3DMM) and a 2D parsing map from an input artistic image. The 3DMM can not only rig the expressions better than facial landmarks but also render images under different poses/expressions as coarse reenactment results robustly. However, these coarse results suffer from self-occlusions and lack contour lines. Second, we thus perform artistic face refinement by using a personalized conditional adversarial generative model (cGAN) fine-tuned on the input artistic image and the coarse reenactment results. For high-quality refinement, we propose a contour loss to supervise the cGAN to faithfully synthesize contour lines. Quantitative and qualitative experiments demonstrate that our method achieves better results than the existing solutions.

IMetaTown: A Metaverse System with Multiple Interactive Functions Based on Virtual Reality.

This work aims to pioneer the development of a real-time interactive and immersive Metaverse Human-Computer Interaction (HCI) system leveraging Virtual Reality (VR). The system incorporates a three-dimensional (3D) face reconstruction method, grounded in weakly supervised learning, to enhance player-player interactions within the Metaverse. The proposed method, two-dimensional (2D) face images, are effectively employed in a 2D Self-Supervised Learning (2DASL) approach, significantly optimizing 3D model learning outcomes and improving the quality of 3D face alignment in HCI systems. The work outlines the functional modules of the system, encompassing user interactions such as hugs and handshakes and communication through voice and text via blockchain. Solutions for managing multiple simultaneous online users are presented. Performance evaluation of the HCI system in a 3D reconstruction scene indicates that the 2DASL face reconstruction method achieves noteworthy results, enhancing the system's interaction capabilities by aiding 3D face modeling through 2D face images. The experimental system achieves a maximum processing speed of 18 frames of image data on a personal computer, meeting real-time processing requirements. User feedback regarding social acceptance, action interaction usability, emotions, and satisfaction with the VR interactive system reveals consistently high scores. The designed VR HCI system exhibits outstanding performance across diverse applications.

3D Face Reconstruction Based on a Single Image: A Review

3D Face Reconstruction Based on a Single Image: A Review

Learning 3D Face Reconstruction From the Cycle-Consistency of Dynamic Faces

Learning 3D Face Reconstruction From the Cycle-Consistency of Dynamic Faces

AU-aware Algorithm for 3D Facial Reconstruction

AU-aware Algorithm for 3D Facial Reconstruction

Disentangled representation transformer network for 3D face reconstruction and robust dense alignment

This paper proposes a disentangled representation transformer network (DRTN) for 3D dense face alignment and reconstruction. Unlike traditional 3DMM-based approaches, the target parameters, such as shape, expression, and pose, are individually estimated without considering their direct influences on one another and then jointly optimized. Hence, DRTN aims to enhance the representation of facial attributes in a semantic sense by learning the correlation of different 3D facial attribute parameters. To achieve this, we present a novel strategy to design disentangled 3D face attribute representation, which decomposes the given facial attributes into identity, expression, and poses. Specifically, the 3D face parameter estimation in the regression network depends on the correlation of other face attribute parameters rather than being independent. The branching of the identity component aims to reinforce learning the expression and pose attributes by preserving the overall face geometry structure and identity. Accordingly, the expression and pose parts of the branch preserve the consistency of expression and pose attributes, respectively. Moreover, DRTN helps refine the reconstruction and alignment of facial details in large poses, mainly by coupling other facial attribute parameters. Extensive qualitative and quantitative experimental results on widely evaluated benchmarking datasets demonstrate that our approach achieves competitive performance compared to state-of-the-art methods.

I Am an Earphone and I Can Hear My User’s Face: Facial Landmark Tracking Using Smart Earphones

This article presents EARFace , a system that shows the feasibility of tracking facial landmarks for 3D facial reconstruction using in-ear acoustic sensors embedded within smart earphones. This enables a number of applications in the areas of facial expression tracking, user interfaces, AR/VR applications, affective computing, and accessibility, among others. Although conventional vision-based solutions break down under poor lighting and occlusions, and also suffer from privacy concerns, earphone platforms are robust to ambient conditions while being privacy-preserving. In contrast to prior work on earable platforms that perform outer-ear sensing for facial motion tracking, EARFace shows the feasibility of completely in-ear sensing with a natural earphone form factor, thus enhancing the comfort levels of wearing. The core intuition exploited by EARFace is that the shape of the ear canal changes due to the movement of facial muscles during facial motion. EARFace tracks the changes in shape of the ear canal by measuring ultrasonic channel frequency response of the inner ear, ultimately resulting in tracking of the facial motion. A transformer-based machine learning model is designed to exploit spectral and temporal relationships in the ultrasonic channel frequency response data to predict the facial landmarks of the user with an accuracy of 1.83 mm. Using these predicted landmarks, a 3D graphical model of the face that replicates the precise facial motion of the user is then reconstructed. Domain adaptation is further performed by adapting the weights of layers using a group-wise and differential learning rate. This decreases the training overhead in EARFace . The transformer-based machine learning model runs on smart phone devices with a processing latency of 13 ms and an overall low power consumption profile. Finally, usability studies indicate higher levels of comforts of wearing EARFace ’s earphone platform in comparison with alternative form factors.

Enhancing polarization 3D facial imaging: overcoming azimuth ambiguity without extra depth devices.

Polarization 3D imaging has been a research hotspot in the field of 3D facial reconstruction because of its biosafety, high efficiency, and simplicity. However, the application of this technology is limited by the multi-valued problem of the azimuth angle of the normal vector. Currently, the most common method to overcome this limitation is to introduce additional depth techniques at the cost of reducing its applicability. This study presents a passive 3D polarization facial imaging method that does not require additional depth-capturing devices. It addresses the issue of azimuth ambiguity based on prior information about the target image's features. Specifically, by statistically analyzing the probability distribution of real azimuth angles, it is found that their quadrant distribution is closely related to the positions of facial feature points. Therefore, through facial feature detection, the polarized normal azimuth angle of each pixel can be accurately assigned to the corresponding quadrant, thus determining a precise unique normal vector and achieving accurate 3D facial reconstruction. Finally, our azimuth angle correction method was validated by simulated polarization imaging results, and it achieved accurate correction for over 75% of the global pixels without using additional depth techniques. Experimental results further indicate that this method can achieve polarization 3D facial imaging under natural conditions without extra depth devices, and the 3D results preserve edge details and texture information.

3D face reconstruction from a single image based on hybrid-level contextual information with weak supervision

Currently, deep learning-based 3D face reconstruction methods have shown promising results. However, they ignore the contextual information of the face, which is a topologically unified entirety. This paper proposes a 3D face reconstruction approach based on hybrid-level contextual information. Firstly, we suggest a regression network with contextual modeling capability at the feature level, PPR-CNet, which adopts a preferential parameter regression to regress the 3DMM parameters dynamically based on their various impacts on the reconstructed 3D face. Furthermore, we design a contextual landmark loss to constrain the face geometry at the landmark level. We introduce a differentiable renderer combined with multiple loss functions for weakly-supervised training. Quantitative experiments on two benchmarks show our method outperforms several SOTA methods. Extensive qualitative experiments indicate that our method performs efficiently in realism, facial proportion, and occlusion.

Decaf: Monocular Deformation Capture for Face and Hand Interactions

Existing methods for 3D tracking from monocular RGB videos predominantly consider articulated and rigid objects ( e.g. , two hands or humans interacting with rigid environments). Modelling dense non-rigid object deformations in this setting ( e.g. when hands are interacting with a face), remained largely unaddressed so far, although such effects can improve the realism of the downstream applications such as AR/VR, 3D virtual avatar communications, and character animations. This is due to the severe ill-posedness of the monocular view setting and the associated challenges ( e.g. , in acquiring a dataset for training and evaluation or obtaining the reasonable non-uniform stiffness of the deformable object). While it is possible to naïvely track multiple non-rigid objects independently using 3D templates or parametric 3D models, such an approach would suffer from multiple artefacts in the resulting 3D estimates such as depth ambiguity, unnatural intra-object collisions and missing or implausible deformations. Hence, this paper introduces the first method that addresses the fundamental challenges depicted above and that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system. As a pivotal step in its creation, we process the reconstructed raw 3D shapes with position-based dynamics and an approach for non-uniform stiffness estimation of the head tissues, which results in plausible annotations of the surface deformations, hand-face contact regions and head-hand positions. At the core of our neural approach are a variational auto-encoder supplying the hand-face depth prior and modules that guide the 3D tracking by estimating the contacts and the deformations. Our final 3D hand and face reconstructions are realistic and more plausible compared to several baselines applicable in our setting, both quantitatively and qualitatively. https://vcai.mpi-inf.mpg.de/projects/Decaf

Comprehensive Vulnerability Evaluation of Face Recognition Systems to Template Inversion Attacks Via 3DFace Reconstruction.

In this paper, we comprehensively evaluate the vulnerability of state-of-the-art face recognition systems to template inversion attacks using 3D face reconstruction. We propose a new method (called GaFaR) to reconstruct 3D faces from facial templates using a pretrained geometry-aware face generation network, and train a mapping from facial templates to the intermediate latent space of the face generator network. We train our mapping with a semi-supervised approach using real and synthetic face images. For real face images, we use a generative adversarial network (GAN)-based framework to learn the distribution of generator intermediate latent space. For synthetic face images, we directly learn the mapping from facial templates to the generator intermediate latent code. Furthermore, to improve the success attack rate, we use two optimization methods on the camera parameters of the GNeRF model. We propose our method in the whitebox and blackbox attacks against face recognition systems and compare the transferability of our attack with state-of-the-art methods across other face recognition systems on the MOBIO and LFW datasets. We also perform practical presentation attacks on face recognition systems using the digital screen replay and printed photographs, and evaluate the vulnerability of face recognition systems to different template inversion attacks. The project page is available at https://www.idiap.ch/paper/gafar.

Multi-dimensional stereo face reconstruction for psychological assistant diagnosis in medical meta-universe

Virtual reality/augmented reality (VR/AR) technology is very important for meta-cosmic medicine. It is the technical link between the material world and the virtual world. The realization of face-to-face psychological diagnosis with three-dimensional holography is a difficult problem in meta-cosmic medicine. Because psychological diagnosis needs to observe rapidly changing microexpressions, it is difficult to capture them. At present, most of the research is based on two dimensions, which cannot meet the real-time reconstruction requirements of meta-universe medicine. To solve these problems, this paper introduces a 3D facial reconstruction algorithm composed of multiresolution feature fusion and attention mechanisms. It can be used for psychological assistant diagnosis of meta-cosmic medicine. First, face alignment and 3D coordinate construction of the feature point cloud are carried out through a 3D convolution coding network. After that, multiresolution cost convolution is used to extract and match the color, facial bones, and depth features of texture images. Finally, the transposed three-dimensional convolution is used to decode and output. To achieve the real-time reconstruction of psychotherapy, we designed a three-dimensional channel attention mechanism module for acceleration. This makes this method suitable for the construction of virtual doctors and virtual patients in the virtual platform of the medical universe of psychological medicine and can also complete the construction of the virtual teaching platform of the medical universe.

3D Face Reconstruction: The Road to Forensics

3D face reconstruction algorithms from images and videos are applied to many fields, from plastic surgery to the entertainment sector, thanks to their advantageous features. However, when looking at forensic applications, 3D face reconstruction must observe strict requirements that still make its possible role in bringing evidence to a lawsuit unclear. An extensive investigation of the constraints, potential, and limits of its application in forensics is still missing. Shedding some light on this matter is the goal of the present survey, which starts by clarifying the relation between forensic applications and biometrics, with a focus on face recognition. Therefore, it provides an analysis of the achievements of 3D face reconstruction algorithms from surveillance videos and mugshot images and discusses the current obstacles that separate 3D face reconstruction from an active role in forensic applications. Finally, it examines the underlying datasets, with their advantages and limitations, while proposing alternatives that could substitute or complement them.

Revealing the face of Ramesses II through computed tomography, digital 3D facial reconstruction and computer-generated Imagery

King Ramesses II (c. 1279 BCE – 1213 BCE), also known as Ramesses the Great, was the third ruler of the 19th Dynasty of the New Kingdom of ancient Egypt. He lived to around 90 years of age and is considered as one of the most powerful rulers of ancient Egypt. From scientific analysis of the mummified remains of the pharaoh using Computed Tomography (CT) scan data and scrutiny of available historical records, 3D facial depictions of the Pharaoh at two different ages were produced: one at 90 years old (age of death) and one at age 45 years when he was at the peak of his military activities. This article discusses the methods and decisions involved in the creation of these facial depictions.

The Florence multi-resolution 3D facial expression dataset

In the literature, several 3D face datasets have been collected, aiming at advancing the field of 3D face analysis from different perspectives. Data collection generally follows specific research needs, and the existing 3D face datasets all have different characteristics that are tailored for investigating different tasks, encompassing face recognition, facial expressions and emotions analysis, 3D face reconstruction. However, the majority of these datasets are either collected with high-resolution scanners, or consumer level devices, such as the Kinect, the latter being motivated by the burdensome and costly process of collecting high-quality scans. Differently from 2D imagery, the difference in resolution in 3D data represents a non negligible problem that is under-investigated, and still prevents the successful development of methods that can work in real scenarios. In this paper, we propose a new 3D face dataset, named “Florence Multi-Resolution 3D Facial Expression” (Florence 3DMRE), which aims at bridging the gap between high- and low-resolution 3D face datasets. Its peculiarity consists in (1) including high-resolution (HR) models obtained with a HR scanner, and paired samples collected with a Kinect sensor, (2) LR and HR scans are synchronized and capture extreme and asymmetric facial deformations as used in facial rehabilitation exercises. In total, our dataset consists of 14 subjects, each performing 19 complex and asymmetric expressions. For each of them, we collected a high-resolution scan, and an RGB-D sequence. Finally, to highlight the value of the dataset and the challenges it introduces, we use the collected data to perform baseline experiments for cross-resolution 3D face recognition and reconstruction. The dataset is released for research purposes only, and complies to GDPR for data treatment. The dataset can be found at this link.

A Perceptual Shape Loss for Monocular 3D Face Reconstruction

AbstractMonocular 3D face reconstruction is a wide‐spread topic, and existing approaches tackle the problem either through fast neural network inference or offline iterative reconstruction of face geometry. In either case carefully‐designed energy functions are minimized, commonly including loss terms like a photometric loss, a landmark reprojection loss, and others. In this work we propose a new loss function for monocular face capture, inspired by how humans would perceive the quality of a 3D face reconstruction given a particular image. It is widely known that shading provides a strong indicator for 3D shape in the human visual system. As such, our new ‘perceptual’ shape loss aims to judge the quality of a 3D face estimate using only shading cues. Our loss is implemented as a discriminator‐style neural network that takes an input face image and a shaded render of the geometry estimate, and then predicts a score that perceptually evaluates how well the shaded render matches the given image. This ‘critic’ network operates on the RGB image and geometry render alone, without requiring an estimate of the albedo or illumination in the scene. Furthermore, our loss operates entirely in image space and is thus agnostic to mesh topology. We show how our new perceptual shape loss can be combined with traditional energy terms for monocular 3D face optimization and deep neural network regression, improving upon current state‐of‐the‐art results.

Meta-evaluation for 3D Face Reconstruction Via Synthetic Data.

The standard benchmark metric for 3D face reconstruction is the geometric error between reconstructed meshes and the ground truth. Nearly all recent reconstruction methods are validated on real ground truth scans, in which case one needs to establish point correspondence prior to error computation, which is typically done with the Chamfer (i.e., nearest neighbor) criterion. However, a simple yet fundamental question have not been asked: Is the Chamfer error an appropriate and fair benchmark metric for 3D face reconstruction? More generally, how can we determine which error estimator is a better benchmark metric? We present a meta-evaluation framework that uses synthetic data to evaluate the quality of a geometric error estimator as a benchmark metric for face reconstruction. Further, we use this framework to experimentally compare four geometric error estimators. Results show that the standard approach not only severely underestimates the error, but also does so inconsistently across reconstruction methods, to the point of even altering the ranking of the compared methods. Moreover, although non-rigid ICP leads to a metric with smaller estimation bias, it could still not correctly rank all compared reconstruction methods, and is significantly more time consuming than Chamfer. In sum, we show several issues present in the current benchmarking and propose a procedure using synthetic data to address these issues.

The presentation of a semi-supervised deep learning platform for 3D face reconstruction from 2D images

The presentation of a semi-supervised deep learning platform for 3D face reconstruction from 2D images

Enhancing eID card mobile-based authentication through 3D facial reconstruction

In recent years the number of credentials and digital identities needed to access online services is exploding, increasing the risk of fraud and unauthorized access to personal data.Biometric verification techniques can help to build more secure authentication systems; in particular face recognition algorithms have reached near perfect accuracy under ideal conditions. In Europe 60% of population owns an electronic identification (eID) document which contains biometric data, and almost all member states provide tools to verify the identity of a user who holds an eID card; however, the adoption of biometric authentication schemes is still limited. Reasons for this low adoption are privacy concerns, about the storage or transmission of personal images, and the degradation of performance of facial recognition algorithms with 2D images acquired "in the wild" (real world condition with inconsistency in lighting and positioning, unclear or occluded facial features, noise from the background). In this paper a new authentication scheme is proposed, exploiting images acquired "in the wild" and 3D facial reconstruction techniques which can help in improving the quality of the probed images, as changes in lighting or position have limited impact on facial shape. Several experiments have been conducted in order to investigate the performance of the proposed method, using the Bosphorus and the NIST Face Recognition Grand Challenge (FRGC) datasets. In particular, one of the most accurate ML face recognition algorithms has been selected to compare the performance of 2D snapshot data, obtained from 3D back-projection, against 2D uncontrolled images acquired "in the wild". We show that using 3D facial reconstructions significantly improve face recognition performance, allowing for a more secure and robust authentication system.