3D Face Reconstruction Research Articles

MPF-GAN: an enhanced architecture for 3D face reconstruction

MPF-GAN: an enhanced architecture for 3D face reconstruction

Advanced 3D Face Reconstruction from Single 2D Images Using Enhanced Adversarial Neural Networks and Graph Neural Networks.

This paper presents a novel framework for 3D face reconstruction from single 2D images and addresses critical limitations in existing methods. Our approach integrates modified adversarial neural networks with graph neural networks to achieve state-of-the-art performance. Key innovations include (1) a generator architecture based on Graph Convolutional Networks (GCNs) with a novel loss function and identity blocks, mitigating mode collapse and instability; (2) the integration of facial landmarks and a non-parametric efficient-net decoder for enhanced feature capture; and (3) a lightweight GCN-based discriminator for improved accuracy and stability. Evaluated on the 300W-LP and AFLW2000-3D datasets, our method outperforms existing approaches, reducing Chamfer Distance by 62.7% and Earth Mover's Distance by 57.1% on 300W-LP. Moreover, our framework demonstrates superior robustness to variations in head positioning, occlusion, noise, and lighting conditions while achieving significantly faster processing times.

Weakly Supervised 3D Face Reconstruction with Joint Spatial and Frequency Domain Information

3D face reconstruction is an important research direction in computer vision, and its goal is to recover a 3D face model from a single face picture. In the absence of real 3D face data, how to reconstruct a 3D face with a high degree of realism has become a hot research topic in recent years. Existing reconstruction algorithms usually rely on 3D labels generated from a large number of 2D face images as training data, however, inaccurate data will seriously affect the reconstruction quality. For this reason, the paper proposes a joint spatial-frequency domain decoupled weak supervision to achieve 3D face reconstruction, the main idea of which is to construct a multi-level loss function by using the weakly supervised information extracted from the spatial domain, and separating the frequency-domain information between the input and the rendered image in the frequency domain, and minimizing the difference between the two by difference computation. The method combines deep learning with 3D deformable models to reconstruct 3D models with high quality texture and shape from only a single face image. Quantitative experiments on the AFLW2000-3D and MICC Florence datasets show that the normalized average error in the small pose interval is as low as 2.42%, and the face reconstruction accuracy in the outdoor scene is 0.98 0.22 mm. Qualitative experiments on the MoFa-test, MICA datasets show that when faced with reconstruction with different poses, lighting, and expressions , our method outperforms other state-of-the-art reconstruction methods.

2D/3D Expression Generation Using Advanced Learning Techniques and the Emotion Wheel

Facial expression analysis is a critical component in numerous applications, ranging from human- computer interaction to digital character animation. Despite the availability of extensive datasets, most focus predominantly on basic emotions, limiting the expressiveness and applicability of models trained on them. This article introduces a novel approach to generating compound emotional expressions by leveraging the Emotion Wheel, a principle that captures the complex interrelations between basic emotions. Our method integrates the EMOCA (Emotion Driven Monocular Face Capture and Animation) [1] framework, which enhances 3D facial reconstruction by incorporating emotion recognition, stacking models [2, 54] with a sophisticated expression blending algorithm to synthesize nuanced 2D and 3D facial animations. Utilizing the VKIST dataset, which includes high-resolution facial images of Vietnamese individuals, we build a comprehensive database of emotion parameters. Through principal component analysis (PCA) and correlation-driven blending, our approach not only enhances the realism of generated facial expressions but also preserves the subtle nuances that are characteristic of human emotions. Experimental results demonstrate that our method consistently outperforms traditional linear interpolation techniques, producing more distinct and recognizable blended expressions. A user study further validates the naturalness and quality of the generated expressions, with average ratings indicating a strong preference for the proposed method over existing approaches. These findings suggest significant potential for improving emotional expressiveness in both static and dynamic digital character representations.

Self-supervised learning for fine-grained monocular 3D face reconstruction in the wild

Self-supervised learning for fine-grained monocular 3D face reconstruction in the wild

3D Face Reconstruction Based on ResNet Feature Extraction and CBAM

In view of the scarcity, high cost and lack of diversity of three-dimensional (3D) face datasets, this paper designs an end-to-end self-supervised learning 3D face reconstruction network, which uses single 2D face image as input. The model bypasses the 3D face datasets and only uses the 2D face datasets for training to achieve high-precision 3D face reconstruction without any 3D face prior. First, the improved ResNet50 feature extraction module is introduced to extract and characterize the input image by deep convolutional network. Then, a lightweight convolutional block attention module is added to the face prediction subnetwork. On the one hand, channel attention extracts different information included in the image, and on the other hand spatial attention finds the location of the information. So, the serialized attention operation could accurately find the features required for different parameter predictions, further improving face reconstruction parameters’ prediction accuracy. Finally, training, ablation and comparison experiments were conducted on CelebFaces Attributes, basel face model and Photoface datasets, and the combined loss function of pixel loss and perception loss was selected. The pixel loss function was calculated at the pixel microscopic level, and the perception loss function was calculated at the image macroscopic convolution level. The combination of the two could complement each other. Compared with the historical optimal results of the same network structure, the scale-invariant depth error and mean angle deviation of the proposed algorithm are improved by 5.2% and 8.2%, respectively. Experimental results strongly prove the effectiveness of the algorithm.

Personalizing human avatars based on realistic 3D facial reconstruction

Personalizing human avatars based on realistic 3D facial reconstruction

Face Recognition Based on Deep Learning: A Comprehensive Review

Face recognition technology has undergone transformative changes with the advent of deep learning techniques. This review paper provides a comprehensive examination of the development and current state of face recognition techniques influenced by deep learning. We begin by discussing the fundamental deep learning models that have dramatically enhanced the accuracy and efficiency of face recognition, highlighting pivotal architectures such as convolutional neural networks (CNNs) and autoencoders. Subsequent sections delve into the application of these models in various environments and challenges, such as different lighting conditions, occlusion, and facial expressions. We also address the integration of deep learning with emerging technologies such as 3D facial reconstruction and multimodal biometrics. Furthermore, the review explores the ethical, privacy, and bias concerns inherent in facial recognition systems, focusing on the need for responsible and fair practices in AI. Finally, future directions are suggested, focusing on the need for robust, adaptable, and ethical face recognition systems. This paper aims to provide an important resource for researchers and practitioners in the field of computer vision, providing insight into the technological advances and ongoing challenges in deep learning-based face recognition.

High Fidelity Makeup via 2D and 3D Identity Preservation Net

In this article, we address the challenging makeup transfer task, aiming to transfer makeup from a reference image to a source image while preserving facial geometry and background consistency. Existing deep neural network-based methods have shown promising results in aligning facial parts and transferring makeup textures. However, they often neglect the facial geometry of the source image, leading to two adverse effects: (1) alterations in geometrically relevant facial features, causing face flattening and loss of personality, and (2) difficulties in maintaining background consistency, as networks cannot clearly determine the face-background boundary. To jointly tackle these issues, we propose the High Fidelity Makeup via two-dimensional (2D) and 3D Identity Preservation Network (IP23-Net), to the best of our knowledge, a novel framework that leverages facial geometry information to generate more realistic results. Our method comprises a 3D Shape Identity Encoder, which extracts identity and 3D shape features. We incorporate a 3D face reconstruction model to ensure the three-dimensional effect of face makeup, thereby preserving the characters’ depth and natural appearance. To preserve background consistency, our Background Correction Decoder automatically predicts an adaptive mask for the source image, distinguishing the foreground and background. In addition to popular benchmarks, we introduce a new large-scale High Resolution Synthetic Makeup Dataset containing 335,230 diverse high-resolution face images to evaluate our method’s generalization ability. Experiments demonstrate that IP23-Net achieves high-fidelity makeup transfer while effectively preserving background consistency. The code will be made publicly available.

3D facial reconstruction and 3D printing of women skull from Bronze age, Slovakia

Facial reconstruction of individuals from historical periods can provide more information about them. The main aim of this study was to create an anthropological facial reconstruction of a Bronze Age skull based on a new soft tissue thickness database created from a sample of the Slovak female population. Using ultrasound, we evaluated 27 facial landmarks on 50 Slovak women, aged 18–30 years with Slovak genealogical history. The Bronze Age skull was anthropologically estimated as a Caucasoid female aged 24–30 years at the time of death. The ultrasound results were analysed and compared with nine European populations. Statistical analysis showed that there is no statistical difference between these populations in the values of facial soft tissue thicknesses. Using CT and Blender software with add-ons, we created a facial approximation of the skull. We 3D printed the virtual model, which can be used for educational and scientific purposes.

Improved 3D human face reconstruction from 2D images using blended hard edges

This study reports an effective and robust edge-based scheme for the reconstruction of 3D human faces from input of single images, addressing drawbacks of existing methods in case of large face pose angles or noisy input images. Accurate 3D face reconstruction from 2D images is important, as it can enable a wide range of applications, such as face recognition, animations, games and AR/VR systems. Edge features extracted from 2D images contain wealthy and robust 3D geometric information, which were used together with landmarks for face reconstruction purpose. However, the accurate reconstruction of 3D faces from contour features is a challenging task, since traditional edge or contour detection algorithms introduce a great deal of noise, which would adversely affect the reconstruction. This paper reports on the use of a hard-blended face contour feature from a neural network and a Canny edge extractor for face reconstruction. The quantitative results indicate that our method achieves a notable improvement in face reconstruction with a Euclidean distance error of 1.64 mm and a normal vector distance error of 1.27 mm when compared to the ground truth, outperforming both traditional and other deep learning-based methods. These metrics show particularly significant advancements, especially in face shape reconstruction under large pose angles. The method also achieved higher accuracy and robustness on in-the-wild images under conditions of blurring, makeup, occlusion and poor illumination.

PR3D: Precise and realistic 3D face reconstruction from a single image

AbstractReconstructing the three‐dimensional (3D) shape and texture of the face from a single image is a significant and challenging task in computer vision and graphics. In recent years, learning‐based reconstruction methods have exhibited outstanding performance, but their effectiveness is severely constrained by the scarcity of available training data with 3D annotations. To address this issue, we present the PR3D (Precise and Realistic 3D face reconstruction) method, which consists of high‐precision shape reconstruction based on semi‐supervised learning and high‐fidelity texture reconstruction based on StyleGAN2. In shape reconstruction, we use in‐the‐wild face images and 3D annotated datasets to train the auxiliary encoder and the identity encoder, encoding the input image into parameters of FLAME (a parametric 3D face model). Simultaneously, a novel semi‐supervised hybrid landmark loss is designed to more effectively learn from in‐the‐wild face images and 3D annotated datasets. Furthermore, to meet the real‐time requirements in practical applications, a lightweight shape reconstruction model called fast‐PR3D is distilled through teacher–student learning. In texture reconstruction, we propose a texture extraction method based on face reenactment in StyleGAN2 style space, extracting texture from the source and reenacted face images to constitute a facial texture map. Extensive experiments have demonstrated the state‐of‐the‐art performance of our method.

EMOVA: Emotion-driven neural volumetric avatar

3D facial reconstruction is essential for metaverse applications. Traditional mesh-based methods have difficulty rendering photorealistic faces and complex objects. Recent advancements in Neural Radiance Fields (NeRFs) have excelled in representing complex objects. However, they struggle with capturing subtle facial differences, particularly around the eyes and mouth, due to their reliance on RGB value comparisons. To address this, we propose an EMOtion-driven Volumetric Avatar (EMOVA) that utilizes emotional stimuli from images and voices to enhance facial precision. Visual emotional features ensure that the reconstructed face aligns with the input emotion, while auditory features enhance facial details from various viewpoints. Through an attention-based fusion of these features, EMOVA accurately captures and reconstructs faces, even in self-occlusion. EMOVA outperforms the state-of-the-art methods by more than 5.93% in terms of LPIPS for face reconstruction.

Self-supervised single-image 3D face reconstruction method based on attention mechanism and attribute refinement

Self-supervised single-image 3D face reconstruction method based on attention mechanism and attribute refinement

A proposed reconstruction method of a 3D animation scene based on a fuzzy long and short-term memory algorithm.

With the development of computer technology leading to a broad range of virtual technology implementations, the construction of virtual tasks has become highly demanded and has increased rapidly, especially in animation scenes. Constructing three-dimensional (3D) animation characters utilizing properties of actual characters could provide users with immersive experiences. However, a 3D face reconstruction (3DFR) utilizing a single image is a very demanding operation in computer graphics and vision. In addition, limited 3D face data sets reduce the performance improvement of the proposed approaches, causing a lack of robustness. When datasets are large, face recognition, transformation, and animation implementations are relatively practical. However, some reconstruction methods only consider the one-to-one processes without considering the correlations or differences in the input images, resulting in models lacking information related to face identity or being overly sensitive to face pose. A face model composed of a convolutional neural network (CNN) regresses 3D deformable model coefficients for 3DFR and alignment tasks. The manuscript proposes a reconstruction method for 3D animation scenes employing fuzzy LSMT-CNN (FLSMT-CNN). Multiple collected images are employed to reconstruct 3D animation characters. First, the serialized images are processed by the proposed method to extract the features of face parameters and then improve the conventional deformable face modeling (3DFDM). Afterward, the 3DFDM is utilized to reconstruct animation characters, and finally, high-precision reconstructions of 3D faces are achieved. The FLSMT-CNN has enhanced both the precision and strength of the reconstructed 3D animation characters, which provides more opportunities to be applied to other animation scenes.

Research on 3D Face Reconstruction Algorithm Based on ResNet and Transformer

In view of the problems of high production cost, scarcity and lack of diversity of 3D face datasets, this paper designs an end-to-end self-supervised learning 3D face reconstruction algorithm with a single 2D face image as input, which only uses 2D face datasets to complete model training. First, the improved ResNet module is introduced to preprocess the input face image. The deep residual neural network has strong feature extraction and characterization ability for the image, which can provide rich high-level semantic feature maps for the subsequent subnetwork. Then, add transformer module completely based on self-attention mechanism to the parameter prediction subnetwork, which can make different parameters of the subnetwork focus on self-related feature map information and avoid interference from invalid feature map information, so as to further improve the parameter prediction accuracy of the subnetwork. Next, training, ablation and comparison experiments were conducted on CelebA, BFM and Photoface datasets, and the combined function of pixel loss function and perceptual loss function was selected as the loss function. The experimental results show that: compared with the historical optimal results of the same network structure, the scale-invariant depth error (SIDE) and mean angle deviation (MAD) are improved by 5.9% and 10.8%, respectively, which strongly proves the effectiveness of the algorithm. Finally, in order to verify the actual effect of the 3D face reconstruction algorithm, examples are selected in this paper for reconstruction. The 3D faces generated by the algorithm all have a good sense of reality, which intuitively and effectively proves the advancement of the algorithm.

Detecting Facial Action Units From Global-Local Fine-Grained Expressions

Since Facial Action Unit (AU) annotations require domain expertise, common AU datasets only contain a limited number of subjects. As a result, a crucial challenge for AU detection is addressing identity overfitting. We find that AUs and facial expressions are highly associated, and existing facial expression datasets often contain a large number of identities. In this paper, we aim to utilize the expression datasets without AU labels to facilitate AU detection. Specifically, we develop a novel AU detection framework aided by the Global-Local facial Expressions Embedding, dubbed GLEE-Net. Our GLEE-Net consists of three branches to extract identity-independent expression features for AU detection. We introduce a global branch for modeling the overall facial expression while eliminating the impacts of identities. We also design a local branch focusing on specific local face regions. The combined output of global and local branches is firstly pre-trained on an expression dataset as an identity-independent expression embedding, and then finetuned on AU datasets. Therefore, we significantly alleviate the issue of limited identities. Furthermore, we introduce a 3D global branch that extracts expression coefficients through 3D face reconstruction to consolidate 2D AU descriptions. Finally, a Transformer-based multi-label classifier is employed to fuse all the representations for AU detection. Extensive experiments demonstrate that our method significantly outperforms the state-of-the-art on the widely-used DISFA, BP4D and BP4D+ datasets.

Exploiting global and instance-level perceived feature relationship matrices for 3D face reconstruction and dense alignment

3D face reconstruction and dense alignment play a key role in digital humans. Learning-based approaches have been developed by employing cascaded convolutional neural networks for feature extraction and spatial-semantic relationship construction. However, these networks often struggle with accurately capturing the complex relationships among various facial components and attributes, particularly in unconstrained environments. Furthermore, the mappings modeled by convolutional weights are at low levels, which are usually implicit and local, and lack global discrimination. In this paper, we propose a global and instance-level perceived relationship matrices-based network (PRMNet) to recover high-fidelity 3D faces and perform accurate dense alignment in unconstrained environments. Specifically, a Key Information Extraction Module (KIEM) extracts crucial features from global and local feature banks. This reduces reasoning costs while maintaining high perceptual quality. Learnable global and instance-level perceived feature relationship matrices are then integrated into the Feature Relationship Reasoning Module (FRRM). This calibration combines key information from both the global macroscopic and sample-specific microscopic views, allowing accurate construction of spatial-semantic relationships to harvest both global discrimination and local relevance. Finally, we introduce a Spatial Reasoning and Guidance Module (SRGM), designed to recalibrate the joint weight responses of feature extraction and its enhancement paths using various attention mechanisms, thereby further enhancing global discrimination ability. Extensive quantitative and qualitative experiments on the benchmark datasets show that our PRMNet outperforms the state-of-the-art. Codes and all resources will be publicly available at https://github.com/Ray-tju/PRMNet.

Single depth image 3D face reconstruction via domain adaptive learning

Single depth image 3D face reconstruction via domain adaptive learning

Artificial intelligence in cosmetic dermatology: An update on current trends

The use of artificial intelligence (AI) will soon be commonplace within the field of cosmetic dermatology. Current uses for AI in the discipline have focused on empowering patients to be more involved in treatment decisions with customizable skin care, augmented-reality applications, and at-home skin analysis tools. AI-driven skin analysis tools are also included in many dermatology practices with the development of three-dimensional facial reconstruction, including models for predicting clinical outcomes. We highlight current and developing applications of AI in cosmetic dermatology and provide insight into future modalities in this field. Dermatologists need to be well-informed about emerging technologies to better educate patients and enhance their clinical practices.