Facial Mesh Research Articles

Audio-Driven Facial Animation with Deep Learning: A Survey

Audio-driven facial animation is a rapidly evolving field that aims to generate realistic facial expressions and lip movements synchronized with a given audio input. This survey provides a comprehensive review of deep learning techniques applied to audio-driven facial animation, with a focus on both audio-driven facial image animation and audio-driven facial mesh animation. These approaches employ deep learning to map audio inputs directly onto 3D facial meshes or 2D images, enabling the creation of highly realistic and synchronized animations. This survey also explores evaluation metrics, available datasets, and the challenges that remain, such as disentangling lip synchronization and emotions, generalization across speakers, and dataset limitations. Lastly, we discuss future directions, including multi-modal integration, personalized models, and facial attribute modification in animations, all of which are critical for the continued development and application of this technology.

Robotic frameless brain biopsy system enhanced by facial mesh registration

AbstractIn this study, a new approach is presented that eliminates stereotactic frameworks and the use of markers, offering an alternative to traditional brain biopsy systems. The classical biopsy operation involves the registration of magnetic resonance (MR) and computed tomography (CT) information taken from the patient at different times. Typically, the surgeon's planning information, which takes an average of 4 h on MR, is transferred to CT, and the surgical operation commences. However, this approach necessitates two separate acquisitions (MR and CT), adversely affecting patient comfort and increasing the workload. In the proposed system, it is recommended to register MR‐Depth camera data instead of MR‐CT registration. To achieve this, a 3D face pattern is obtained from the data received from the depth camera attached to the robot arm and overlapped with the mesh obtained by segmentation of the MR. It was observed that registration with sub‐millimeter precision was achieved using the CMFreg surface registration technique.

Enhancing Automative Safety System for Driver Fatigue Detection by Using Mediapipe’s Framework

Traffic accidents caused by tiredness, exhaustion, and distracted driving are a big concern on a global scale. As per global Road Safety reports drowsy driving was reason for 91,000 road accidents in Tamil Nadu in the year 2023. In this study, I suggest a collective deep learning architecture for automatic driver sleepiness detection while some earlier works are suggested extracting the lips and eyes movements to identify sleepiness, more contemporary computer vision-based systems have performed only somewhat well. This is because they either use extremely large deep learning models with still-poor performance or they use hand-crafted features with traditional methods like K-NN Algotithm and openCV. To overcome such issues in this project I am planing to create a Driver Drowsiness Detection and Alerting System using Mediapipe‘s Face Mesh solution in Python. These systems assess the driver‘s alertness and warn the driver if needed. Continuous driving can be tedious and exhausting. A motorist may get droopy and perhaps nod off due to inactivity. we will create a drowsy driver detection system to address such an issue. For this, we will use Mediapipe‘s Face Mesh solution in python and the Eye Aspect ratio formula using by Navie Bayes Algorithm. My goal is to create a robust and easy-to-use application that detects and alerts users if their eyes are closed for a long time. Keyword: Navie Baye’s, Mediapipe Framework.

Dynamic 4D facial capture pipeline with appearance driven progressive retopology based on optical flow

This paper presents a production-oriented 4D facial reconstruction pipeline designed to produce high-fidelity facial mesh sequences with a consistently structured topology, while preserving the wireframe structure specified by artists. We have designed and developed a compact, efficient, and fast optical capture system based on synchronized camera arrays for high-precision dynamic 3D facial imaging. Unlike prevailing methods that primarily concentrate on single-frame reconstruction, often reliant on labor-intensive manual annotation, our framework exploits the constraint of appearance consistency to autonomously establish feature correspondence and uphold temporal coherence within the mesh. Consequently, our approach eliminates mesh drifting and jitter, enabling full parallelization for dynamic facial expression capture. The proposed pipeline decouples the non-linear deformation of facial expressions from the rigid movements of the skull through a stable external device. Leveraging progressive retopology, our methodology employs artist-guided templates as priors, ensuring the preservation of wireframe structures across the result sequence. Progressive retopology is achieved by constraining different fine-grained features of 3D landmarks, scan surface shapes, and appearance textures. The results of our study showcase facial mesh sequences with production-quality topology, adept at faithfully reproducing character expressions from photographs while achieving artist-friendly stable facial movements.

MeshWGAN: Mesh-to-Mesh Wasserstein GAN With Multi-Task Gradient Penalty for 3D Facial Geometric Age Transformation.

As the metaverse develops rapidly, 3D facial age transformation is attracting increasing attention, which may bring many potential benefits to a wide variety of users, e.g., 3D aging figures creation, 3D facial data augmentation and editing. Compared with 2D methods, 3D face aging is an underexplored problem. To fill this gap, we propose a new mesh-to-mesh Wasserstein generative adversarial network (MeshWGAN) with a multi-task gradient penalty to model a continuous bi-directional 3D facial geometric aging process. To the best of our knowledge, this is the first architecture to achieve 3D facial geometric age transformation via real 3D scans. As previous image-to-image translation methods cannot be directly applied to the 3D facial mesh, which is totally different from 2D images, we built a mesh encoder, decoder, and multi-task discriminator to facilitate mesh-to-mesh transformations. To mitigate the lack of 3D datasets containing children's faces, we collected scans from 765 subjects aged 5-17 in combination with existing 3D face databases, which provided a large training dataset. Experiments have shown that our architecture can predict 3D facial aging geometries with better identity preservation and age closeness compared to 3D trivial baselines. We also demonstrated the advantages of our approach via various 3D face-related graphics applications.

Formulating facial mesh tracking as a differentiable optimization problem: a backpropagation-based solution

Facial mesh tracking enables the production of topologically consistent 3D facial meshes from stereo video input captured by calibrated cameras. This technology is an integral part of many digital human applications, such as personalized avatar creation, audio-driven 3D facial animation, and talking face video generation. Currently, most facial mesh tracking methods are built on computer graphics techniques, which involve complex procedures and often necessitate human annotation within pipelines. As a result, these approaches are difficult to implement and hard to generalize across various scenarios. We propose a backpropagation-based solution that formulates facial mesh tracking as a differentiable optimization problem called the BPMT. Our solution leverages visual clues extracted from the stereo input to estimate vertex-wise geometry and texture information. The BPMT is composed of two steps: automatic face analysis and mesh tracking. In the first step, a range of visual clues are automatically extracted from the input, including facial point clouds, multi-view 2D landmarks, 3D landmarks in the world coordinate system, motion fields, and image masks. The second step can be viewed as a differentiable optimization problem, with constraints comprising stereo video input and facial clues. The primary objective is to achieve topologically consistent 3D facial meshes across frames. Additionally, the parameters to be optimized encompass the positions of free-form deformed vertices and a shared texture UV map. Furthermore, the 3D morphable model (3DMM) is introduced as a form of regularization to enhance the convergence of the optimization process. Leveraging the fully developed backpropagation software, we progressively register the facial meshes to the recorded object, generating high-quality 3D faces with consistent topologies. The BPMT requires no manual labeling within the pipeline, making it suitable for producing large-scale stereo facial data. Moreover, our method exhibits a high degree of flexibility and extensibility, positioning it as a promising platform for future research in the community.

GeoSparseNet: A Multi-Source Geometry-Aware CNN for Urban Scene Analysis

The convolutional neural networks (CNNs) functioning on geometric learning for the urban large-scale 3D meshes are indispensable because of their substantial, complex, and deformed shape constitutions. To address this issue, we proposed a novel Geometry-Aware Multi-Source Sparse-Attention CNN (GeoSparseNet) for the urban large-scale triangular mesh classification task. GeoSparseNet leverages the non-uniformity of 3D meshes to depict both broad flat areas and finely detailed features by adopting the multi-scale convolutional kernels. By operating on the mesh edges to prepare for subsequent convolutions, our method exploits the inherent geodesic connections by utilizing the Large Kernel Attention (LKA) based Pooling and Unpooling layers to maintain the shape topology for accurate classification predictions. Learning which edges in a mesh face to collapse, GeoSparseNet establishes a task-oriented process where the network highlights and enhances crucial features while eliminating unnecessary ones. Compared to previous methods, our innovative approach outperforms them significantly by directly processing extensive 3D mesh data, resulting in more discerning feature maps. We achieved an accuracy rate of 87.5% when testing on an urban large-scale model dataset of the Australian city of Adelaide.

Prismatic-element SBPML coupled with SBFEM for 3D infinite transient wave problems

In this paper, an enhanced prismatic-element scaled boundary perfectly matched layer (SBPML) is developed, which is a novel time-domain artificial boundary method for 3D infinite wave problems. The SBPML permits the utilization of an artificial boundary with general geometry and can consider planar physical surfaces and interfaces extending to infinity. Moreover, this enhancement enables the seamless integration with surface meshes of arbitrary polygonal faces (such as triangles, quadrangles, pentagons, heptagons, octagons, etc.) at the truncation boundaries. The interior domain can be modelled by polyhedrons formulated by the scaled boundary finite element method (SBFEM). The geometric shape of the SBPML elements in the proposed method, within the local coordinate system, takes the form of a regular prism with regular polygonal top and bottom faces. Consequently, it is referred to as the Prismatic-element SBPML. Local scaled boundary coordinates based on the polygonal shape function are firstly introduced into the truncated infinite domain to describe its general geometry properties and provide the coupled capabilities with arbitrary polygonal face meshes at the truncation boundaries of numerical models. Then, a complex stretching function from PML is applied to radial direction of the local scaled boundary coordinates to map the physical space onto the complex space, resulting in a SBPML domain. Upon spatial discretization, the proposed method is formulated using a 2nd-order mixed displacement-stress unsplit-field formulation, allowing seamless coupling with FEM and SBFEM originally designed for the interior computational domain. Finally, three benchmark examples of wave propagation problems in unbounded domains and two soil-structure interaction problems of engineering problems are presented to demonstrate the accuracy and robustness of the Prismatic-element SBPML.

Learning to Stabilize Faces

AbstractNowadays, it is possible to scan faces and automatically register them with high quality. However, the resulting face meshes often need further processing: we need to stabilize them to remove unwanted head movement. Stabilization is important for tasks like game development or movie making which require facial expressions to be cleanly separated from rigid head motion. Since manual stabilization is labor‐intensive, there have been attempts to automate it. However, previous methods remain impractical: they either still require some manual input, produce imprecise alignments, rely on dubious heuristics and slow optimization, or assume a temporally ordered input. Instead, we present a new learning‐based approach that is simple and fully automatic. We treat stabilization as a regression problem: given two face meshes, our network directly predicts the rigid transform between them that brings their skulls into alignment. We generate synthetic training data using a 3D Morphable Model (3DMM), exploiting the fact that 3DMM parameters separate skull motion from facial skin motion. Through extensive experiments we show that our approach outperforms the state‐of‐the‐art both quantitatively and qualitatively on the tasks of stabilizing discrete sets of facial expressions as well as dynamic facial performances. Furthermore, we provide an ablation study detailing the design choices and best practices to help others adopt our approach for their own uses.

ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular Region

AbstractEye gaze and expressions are crucial non‐verbal signals in face‐to‐face communication. Visual effects and telepresence demand significant improvements in personalized tracking, animation, and synthesis of the eye region to achieve true immersion. Morphable face models, in combination with coordinate‐based neural volumetric representations, show promise in solving the difficult problem of reconstructing intricate geometry (eyelashes) and synthesizing photorealistic appearance variations (wrinkles and specularities) of eye performances. We propose a novel hybrid representation ‐ ShellNeRF ‐ that builds a discretized volume around a 3DMM face mesh using concentric surfaces to model the deformable ‘periocular’ region. We define a canonical space using the UV layout of the shells that constrains the space of dense correspondence search. Combined with an explicit eyeball mesh for modeling corneal light‐transport, our model allows for animatable photorealistic 3D synthesis of the whole eye region. Using multi‐view video input, we demonstrate significant improvements over state‐of‐the‐art in expression re‐enactment and transfer for high‐resolution close‐up views of the eye region.

Method for Human Ear Localization in Controlled and Uncontrolled Environments

One of the fundamental stages in recognizing people by their ears, which most works omit, is locating the area of interest. The sets of images used for experiments generally contain only the ear, which is not appropriate for application in a real environment, where the visual field may contain part of or the entire face, a human body, or objects other than the ear. Therefore, determining the exact area where the ear is located is complicated, mainly in uncontrolled environments. This paper proposes a method for ear localization in controlled and uncontrolled environments using MediaPipe, a tool for face localization, and YOLOv5s architecture for detecting the ear. The proposed method first determines whether there are cues that indicate that a face exists in an image, and then, using the MediaPipe facial mesh, the points where an ear potentially exists are obtained. The extracted points are employed to determine the ear length based on the proportions of the human body proposed by Leonardo Da Vinci. Once the dimensions of the ear are obtained, the delimitation of the area of interest is carried out. If the required elements are not found, the model uses the YOLOv5s architecture module, trained to recognize ears in controlled environments. We employed four datasets for testing (i) In-the-wild Ear Database, (ii) IIT Delhi Ear Database, (iii) AMI Ear Database, and (iv) EarVN1.0. Also, we used images from the Internet and some acquired using a Redmi Note 11 cell phone camera. An accuracy of 97% with an error of 3% was obtained with the proposed method, which is a competitive measure considering that tests were conducted in controlled and uncontrolled environments, unlike state-of-the-art methods.

4D Facial Expression Diffusion Model

Facial expression generation is one of the most challenging and long-sought aspects of character animation, with many interesting applications. The challenging task, traditionally having relied heavily on digital craftspersons, remains yet to be explored. In this paper, we introduce a generative framework for generating 3D facial expression sequences (i.e. 4D faces) that can be conditioned on different inputs to animate an arbitrary 3D face mesh. It is composed of two tasks: (1) Learning the generative model that is trained over a set of 3D landmark sequences, and (2) Generating 3D mesh sequences of an input facial mesh driven by the generated landmark sequences. The generative model is based on a Denoising Diffusion Probabilistic Model (DDPM), which has achieved remarkable success in generative tasks of other domains. While it can be trained unconditionally, its reverse process can still be conditioned by various condition signals. This allows us to efficiently develop several downstream tasks involving various conditional generation, by using expression labels, text, partial sequences, or simply a facial geometry. To obtain the full mesh deformation, we then develop a landmark-guided encoder-decoder to apply the geometrical deformation embedded in landmarks on a given facial mesh. Experiments show that our model has learned to generate realistic, quality expressions solely from the dataset of relatively small size, improving over the state-of-the-art methods. Videos and qualitative comparisons with other methods can be found at https://github.com/ZOUKaifeng/4DFM. Code and models will be made available upon acceptance.

Three-Dimensional Virtual Reconstruction of External Nasal Defects Based on Facial Mesh Generation Network.

(1) Background: In digital-technology-assisted nasal defect reconstruction methods, a crucial step involves utilizing computer-aided design to virtually reconstruct the nasal defect's complete morphology. However, current digital methods for virtual nasal defect reconstruction have yet to achieve efficient, precise, and personalized outcomes. In this research paper, we propose a novel approach for reconstructing external nasal defects based on the Facial Mesh Generation Network (FMGen-Net), aiming to enhance the levels of automation and personalization in virtual reconstruction. (2) Methods: We collected data from 400 3D scans of faces with normal morphology and combined the structured 3D face template and the Meshmonk non-rigid registration algorithm to construct a structured 3D facial dataset for training FMGen-Net. Guided by defective facial data, the trained FMGen-Net automatically generated an intact 3D face that was similar to the defective face, and maintained a consistent spatial position. This intact 3D face served as the 3D target reference face (3D-TRF) for nasal defect reconstruction. The reconstructed nasal data were extracted from the 3D-TRF based on the defective area using reverse engineering software. The '3D surface deviation' between the reconstructed nose and the original nose was calculated to evaluate the effect of 3D morphological restoration of the nasal defects. (3) Results: In the simulation experiment of 20 cases involving full nasal defect reconstruction, the '3D surface deviation' between the reconstructed nasal data and the original nasal data was 1.45 ± 0.24 mm. The reconstructed nasal data, constructed from the personalized 3D-TRF, accurately reconstructed the anatomical morphology of nasal defects. (4) Conclusions: This paper proposes a novel method for the virtual reconstruction of external nasal defects based on the FMGen-Net model, achieving the automated and personalized construction of the 3D-TRF and preliminarily demonstrating promising clinical application potential.

Deep Spectral Meshes: Multi-Frequency Facial Mesh Processing with Graph Neural Networks

With the rising popularity of virtual worlds, the importance of data-driven parametric models of 3D meshes has grown rapidly. Numerous applications, such as computer vision, procedural generation, and mesh editing, vastly rely on these models. However, current approaches do not allow for independent editing of deformations at different frequency levels. They also do not benefit from representing deformations at different frequencies with dedicated representations, which would better expose their properties and improve the generated meshes’ geometric and perceptual quality. In this work, spectral meshes are introduced as a method to decompose mesh deformations into low-frequency and high-frequency deformations. These features of low- and high-frequency deformations are used for representation learning with graph convolutional networks. A parametric model for 3D facial mesh synthesis is built upon the proposed framework, exposing user parameters that control disentangled high- and low-frequency deformations. Independent control of deformations at different frequencies and generation of plausible synthetic examples are mutually exclusive objectives. A Conditioning Factor is introduced to leverage these objectives. Our model takes further advantage of spectral partitioning by representing different frequency levels with disparate, more suitable representations. Low frequencies are represented with standardised Euclidean coordinates, and high frequencies with a normalised deformation representation (DR). This paper investigates applications of our proposed approach in mesh reconstruction, mesh interpolation, and multi-frequency editing. It is demonstrated that our method improves the overall quality of generated meshes on most datasets when considering both the L1 norm and perceptual Dihedral Angle Mesh Error (DAME) metrics.

Indonesian Lip‐Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long‐Term Recurrent Convolutional Network

Indonesian Lip‐Reading Detection and Recognition Based on Lip Shape Using Face Mesh and Long‐Term Recurrent Convolutional Network

Experimental and simulated studies on the distribution law of rotational–perforated fluid in tridimensional rotational flow sieve tray

The distribution law of the rotational and perforated fluid in tridimensional rotational flow sieve trays (TRST) is the key point for revealing its working mechanism and engineering application. Due to complex issues such as overlapping blades, rotational fluid accumulation, and centrifugal force within TRST, it is very difficult to accurately calculate the flow ratio. In this paper, the flowable interior mesh faces were constructed at all sieve holes on a blade, and the velocity distribution on the interior faces was simulated as the fluid perforated through the different sieves, thus calculating the volume flow rate and ratio of the flow through perforations. And the three-blade unit experiment was designed to calculate the perforated ratio of the entire blade. Compared with the simulated values, the relative differences of the gas and liquid phases were within 25% and 10%, respectively, which were in good agreement.

Modified Characteristic Finite Element Method with Second-Order Spatial Accuracy for Solving Convection-Dominated Problem on Surfaces.

We present a modified characteristic finite element method that exhibits second-order spatial accuracy for solving convection-reaction-diffusion equations on surfaces. The temporal direction adopted the backward-Euler method, while the spatial direction employed the surface finite element method. In contrast to regular domains, it is observed that the point in the characteristic direction traverses the surface only once within a brief time. Thus, good approximation of the solution in the characteristic direction holds significant importance for the numerical scheme. In this regard, Taylor expansion is employed to reconstruct the solution beyond the surface in the characteristic direction. The stability of our scheme is then proved. A comparison is carried out with an existing characteristic finite element method based on face mesh. Numerical examples are provided to validate the effectiveness of our proposed method.

Numerical simulation of the extrusion process of viscoplastic materials using a radial point interpolation method

PurposeFused Filament Fabrication (FFF) is an extrusion-based manufacturing process using fused thermoplastics. Despite its low cost, the FFF is not extensively used in high-value industrial sectors mainly due to parts' anisotropy (related to the deposition strategy) and residual stresses (caused by successive heating cycles). Thus, this study aims to investigate the process improvement and the optimization of the printed parts.Design/methodology/approachIn this work, a meshless technique – the Radial Point Interpolation Method (RPIM) – is used to numerically simulate the viscoplastic extrusion process – the initial phase of the FFF. Unlike the FEM, in meshless methods, there is no pre-established relationship between the nodes so the nodal mesh will not face mesh distortions and the discretization can easily be modified by adding or removing nodes from the initial nodal mesh. The accuracy of the obtained results highlights the importance of using meshless techniques in this field.FindingsMeshless methods show particular relevance in this topic since the nodes can be distributed to match the layer-by-layer growing condition of the printing process.Originality/valueUsing the flow formulation combined with the heat transfer formulation presented here for the first time within an in-house RPIM code, an algorithm is proposed, implemented and validated for benchmark examples.

PSXI-8 Comparison of Spectral and Motion Sensing for Grazing Behavior Determination in Extensive Systems

Abstract There is a need to classify eating behaviors of grazing animals to better understand time budgets and intake levels of extensively managed livestock. The objective of this work was to compare spectral and motion-based sensing for classification of animal grazing versus standing behaviors within extensive production systems. The sensor was an open-source setup with a TTGO-T-Beam microprocessor (Shenzhen Xin Yuan Electronic Technology Co, China), soldered with an AS7265x Spectral Triad (SparkFun Electronics, Niwot, CO) and generic 9 degree of freedom inertial motion unit. The sensors were attached to a mesh face mask and placed on 5 horses for 10 minutes of data collection. Within the data collection period, animals were allowed to graze for 5 minutes, and required to stand for 5 minutes. The order of grazing and standing was randomized among animals. Data were collected by the microprocessor at 100 hertz and averaged over a 20 second period. Average data were transmitted via long range radio (LoRa) to a base station. Data were matched by timestamp to ground truth observations of animal standing vs grazing behaviors. A random forest algorithm was used to explore predicting animal grazing versus standing behaviors based on motion data (average pitch, yaw, and roll of animals over the experimental period) individually, based on the top features selected from the spectral data individually, or based on all collected data together. The model using the motion data only returned an out of bag estimated error rate of 26.5%. The model using the top features selected from the spectral data yielded a much lower out of bag estimated error rate (8.82%), which was not improved by leveraging all data together. By comparing the spectral data with GPS coordinates attached to each observation, we were able to clearly distinguish the spatial area used for grazing compared with that used for standing. One potential impact of this preliminary exploration is in low-cost, low-power, more precise behavioral classification for grazing livestock. Because the spectral sensed data generated more reliable ability to distinguish grazing and standing behaviors, we may be able to rely on lower frequency observations, or lower data acquisition rates to achieve acceptable classification accuracies. Similarly, given the success of NIR in analyzing feed composition values, there is likelihood that spectral sensing could be used not only to identify grazing, but to character forage quality grazed.

Research on point cloud hole filling and 3D reconstruction in reflective area

3D reconstruction is the process of obtaining the three-dimensional shape or surface structure of an object, which is widely used in advanced manufacturing fields such as automotive, aerospace, industrial inspection, and reverse engineering. However, due to the structural characteristics of the component itself, the reflective properties of the coating material, and other factors, there may be specular reflection during image acquisition, making it difficult to achieve complete 3D reconstruction of the component. This paper proposes a method to address the problem of incomplete 3D reconstruction of strongly reflective objects by recognizing outlier points and filling point cloud holes. The proposed View-Transform-PointNet outlier point recognition network improves the alignment of the initial point cloud plane and implements secondary alignment of the point cloud based on the perpendicularity between the outlier plane in mixed reflection and the point cloud plane. The point cloud hole-filling method is based on the principle of outlier formation and approximates a local Gaussian distribution to linear variation. The distance between the end of each outlier plane and the real surface is calculated to repair the depth information of outlier points. The proposed method achieves a 39.4% increase in the number of point cloud filling, a 45.2% increase in the number of triangular mesh faces, a 46.9% increase in surface area, and a chamfer distance (CD) of 0.4471009, which is better than existing geometric repair methods in terms of standard deviation and smoothness. The method improves the alignment of initial point cloud planes and enhances the accuracy of outlier point recognition, which are the main innovative points of this study. The 3D reconstruction of the repaired point cloud model is achieved through Poisson equation and parameter adjustment. The proposed method reduces the error caused by large curvature in the boundary region and improves the smoothness and accuracy of the reconstructed model.