3D Landmarks Research Articles

Automated 3D Perioral Landmark Detection Using High-Resolution Network: Artificial Intelligence-based Anthropometric Analysis.

Three-dimensional facial stereophotogrammetry, a convenient, noninvasive and highly reliable evaluation tool, has in recent years shown great potential in plastic surgery for preoperative planning and evaluating treatment efficacy. However, it requires manual identification of facial landmarks by trained evaluators to obtain anthropometric data, which takes much time and effort. Automatic 3D facial landmark localization has the potential to facilitate fast data acquisition and eliminate evaluator error. The aim of this work was to describe a novel deep-learning method based on dimension transformation and key-point detection for automated 3D perioral landmark annotation. After transforming a 3D facial model into 2D images, High-Resolution Network is implemented for key-point detection. The 2D coordinates of key points are then mapped back to the 3D model using mathematical methods to obtain the 3D landmark coordinates. This program was trained with 120 facial models and validated in 50 facial models. Our approach achieved a satisfactory mean [standard deviation] accuracy of 1.30 [0.68] mm error in landmark detection with a mean processing time of 5.2 [0.21] seconds per model. Subsequent analysis based on these landmarks showed mean errors of 0.87 [1.02] mm for linear measurements and 5.62° [6.61°] for angular measurements. This automated 3D perioral landmarking method could serve as an effective tool that enables fast and accurate anthropometric analysis of lip morphology for plastic surgery and aesthetic procedures.

Audio-to-Deep-Lip: Speaking lip synthesis based on 3D landmarks

Generating talking lips in sync with input speech has the potential to enhance speech communication and enable novel applications. This paper presents a system that can generate accurate 3D talking lips, readily applicable to unseen subjects and different languages. The developed head-mounted facial acquisition device and automated data processing pipeline can generate precise landmarks while mitigating the difficulty of acquiring 3D facial data. Our system consists of three stages to generate accurate lip movements. In the first stage, the fine-tuned Wav2Vec2.0+Transformer captures long-range audio context dependencies. In the second stage, we propose the Viseme Fixing method, which significantly improves lip accuracy at/b//p//m//f/ phonemes. In the last stage, we innovatively use the structural relationship between the inner and outer lips and learn to map the outer lip landmarks to the inner lip landmarks. Subjective evaluations show that the generated talking lips match the input audio significantly. We demonstrate two applications that animate 2D face videos and 3D face models using our landmarks. The precise lip landmarks allow the generated animations to exceed the results of state-of-the-art methods.

Intra-rater and inter-rater reliability of 3D facial measurements

Three-dimensional (3D) body scanning technology has applications for obtaining anthropometric data in human-centered and product development fields. The reliability of 3D measurements gathered from 3D scans must be assessed to understand the degree to which this technology is appropriate for use in place of manual anthropometric methods. The intra- and inter-rater reliabilities of 3D facial measurements were assessed among four novice raters using 3D landmarking. Intraclass correlation coefficient (ICC) statistics were calculated for the 3D measurement data collected in three phases to assess baseline reliabilities and improvements in reliabilities as the result of additional training and experience. Based on the results of this study, the researchers found that the collection of 3D measurement data, by multiple raters and using 3D landmarking methods, yielded a high percentage of ICC statistics in the good to excellent (>0.75 ICC) reliability range. Rater training and experience were important considerations in improving intra- and inter-rater reliabilities.

Double Reference Guided Interactive 2D and 3D Caricature Generation

In this paper, we propose the first geometry and texture (double) referenced interactive 2D and 3D caricature generating and editing method. The main challenge of caricature generation lies in the fact that it not only exaggerates the facial geometry but also refreshes the facial texture. We address this challenge by utilizing the semantic segmentation maps as an intermediary domain, removing the influence of photo texture while preserving the person-specific geometry features. Specifically, our proposed method consists of two main components: 3D-CariNet and CariMaskGAN. 3D-CariNet uses sketches or caricatures to exaggerate the input photo into several types of 3D caricatures. To generate a CariMask, we geometrically exaggerate the photos using the projection of exaggerated 3D landmarks, after which CariMask is converted into a caricature by CariMaskGAN. In this step, users can edit and adjust the geometry of caricatures freely. Moreover, we propose a semantic detail preprocessing approach that considerably increases the details of generated caricatures and allows modification of hair strands, wrinkles, and beards. By rendering high-quality 2D caricatures as textures, we produce 3D caricatures with a variety of texture styles. Extensive experimental results have demonstrated that our method can produce higher-quality caricatures as well as support interactive modification with ease.

Generating Multiple 4D Expression Transitions by Learning Face Landmark Trajectories

In this work, we address the problem of 4D facial expressions generation. This is usually addressed by animating a neutral 3D face to reach an expression peak, and then get back to the neutral state. In the real world though, people show more complex expressions, and switch from one expression to another. We thus propose a new model that generates transitions between different expressions, and synthesizes long and composed 4D expressions. This involves three sub-problems: (i) modeling the temporal dynamics of expressions, (ii) learning transitions between them, and (iii) deforming a generic mesh. We propose to encode the temporal evolution of expressions using the motion of a set of 3D landmarks, that we learn to generate by training a manifold-valued GAN (Motion3DGAN). To allow the generation of composed expressions, this model accepts two labels encoding the starting and the ending expressions. The final sequence of meshes is generated by a Sparse2Dense mesh Decoder (S2D-Dec) that maps the landmark displacements to a dense, per-vertex displacement of a known mesh topology. By explicitly working with motion trajectories, the model is totally independent from the identity. Extensive experiments on five public datasets show that our proposed approach brings significant improvements with respect to previous solutions, while retaining good generalization to unseen data.

4D Facial Expression Diffusion Model

Facial expression generation is one of the most challenging and long-sought aspects of character animation, with many interesting applications. The challenging task, traditionally having relied heavily on digital craftspersons, remains yet to be explored. In this paper, we introduce a generative framework for generating 3D facial expression sequences (i.e. 4D faces) that can be conditioned on different inputs to animate an arbitrary 3D face mesh. It is composed of two tasks: (1) Learning the generative model that is trained over a set of 3D landmark sequences, and (2) Generating 3D mesh sequences of an input facial mesh driven by the generated landmark sequences. The generative model is based on a Denoising Diffusion Probabilistic Model (DDPM), which has achieved remarkable success in generative tasks of other domains. While it can be trained unconditionally, its reverse process can still be conditioned by various condition signals. This allows us to efficiently develop several downstream tasks involving various conditional generation, by using expression labels, text, partial sequences, or simply a facial geometry. To obtain the full mesh deformation, we then develop a landmark-guided encoder-decoder to apply the geometrical deformation embedded in landmarks on a given facial mesh. Experiments show that our model has learned to generate realistic, quality expressions solely from the dataset of relatively small size, improving over the state-of-the-art methods. Videos and qualitative comparisons with other methods can be found at https://github.com/ZOUKaifeng/4DFM. Code and models will be made available upon acceptance.

Automatic labeling of 3D facial acupoint landmarks

<p>As special marks on a human face, facial landmarks reflect the facial features of various parts of the face, which is crucial in biomedicine and medical imaging. In addition, facial landmarks are also important features in computer vision such as face detection, face recognition, facial pose estimation, and facial animation. In this paper, we construct a 3D facial acupoint annotated dataset by labeling 37 facial acupoints on 846 neutral face triangle mesh on the FaceScape dataset. Based on these annotated data, we use a feature template matching method to realize the automatic annotation of 37 acupoints on triangle meshes. We used 40 meshes as the training set to extract the geometric patterns of 3D acupoints and then measured the performance of the automatic labeling algorithm on 20 meshes and 806 meshes as the test sets. In the training process, we extract the tangent plane for each landmark, project the neighbor vertices of the landmark to the tangent plane, and construct the feature image with <em>R</em> × <em>R</em> resolution through the bounding box of the projected points. In the testing process, we use the pattern images extracted during training to find the average features and use them as a guide to optimize the predicted landmarks. The experimental results show that our automatic acupoint labeling method has achieved good results.<strong></strong></p>

3D anatomy of the supraorbital and greater occipital nerve trajectories.

This research aims to enhance understanding of the anatomy of the supraorbital nerve (SON) and greater occipital nerve (GON), focusing on their exit points, distal trajectories, and variability, utilizing a novel 3D representation. Ten cadaveric specimens underwent meticulous dissection, and 3D landmarks were registered. Models were generated from CT scans, and a custom 3D method was employed to visualize nerve trajectories. Measurements, including lengths and distances, were obtained for the SON and GON. The SON exhibited varied exit points, with the lateral branches being the longest. The GON showed distinct branching patterns, which are described relative to various anatomical reference points and planes. No systematic left-right differences were observed for either nerve. 3D analysis revealed significant interindividual variability in nerve trajectories. The closest approximation between the SON and GON occurred between lateral branches. The study introduces a novel 3D methodology for analyzing the SON and GON, highlighting considerable anatomical variation. Understanding this variability is crucial for clinical applications and tools targeting the skull innervation. The findings serve as a valuable reference for future research, emphasizing the necessity for personalized approaches in innervation-related interventions.

The influence of landmark visualization style on task performance, visual attention, and spatial learning in a real-world navigation task

ABSTRACT Depicting landmarks on mobile maps is an increasingly popular countermeasure to the negative effect that navigation aids have on spatial learning – landmarks guide visual attention and facilitate map-to-environment information matching. However, the most effective method to visualize landmarks on mobile map aids remains an open question. We conducted a real-world navigation study outdoors to evaluate the influence of realistic vs. abstract 3D landmark visualization styles on wayfinders’ navigation performance, visual attention, and spatial learning. While navigating with realistic landmarks, low-spatial-ability wayfinders focused more on the landmarks in the environment and demonstrated improved knowledge of directions between landmarks. Our findings emphasize the importance of visual realism when enriching navigation aids with landmarks to guide attention and enhance spatial learning for low-spatial-ability wayfinders.

BMP2 rs1005464 is associated with mandibular condyle size variation

This study aimed to evaluate the association between single nucleotide polymorphisms (SNPs) in endochondral development-related genes and mandibular condyle shape, size, volume, and symmetry traits. Cone-beam Computed Tomographies and genomic DNA from 118 individuals were evaluated (age range: 15–66 years). Data from twelve 3D landmarks on mandibular condyles were submitted to morphometric analyses including Procrustes fit, principal component analysis, and estimation of centroid sizes and fluctuating asymmetry scores. Condylar volumes were additionally measured. Seven SNPs across BMP2, BMP4, RUNX2 and SMAD6 were genotyped. Linear models were fit to evaluate the effect of the SNPs on the mandibular condyles’ quantitative traits. Only the association between BMP2 rs1005464 and centroid size remained significant after adjusting to account for the false discovery rate due to multiple testing. Individuals carrying at least one A allele for this SNP showed larger condylar size than common homozygotes GG (β = 0.043; 95% CI: 0.014—0.071; P value = 0.028). The model including BMP2 rs1005464, age and sex of the participants explained 17% of the variation in condylar size. Shape, volume, and symmetry were not associated with the evaluated SNPs. These results suggest that BMP2 rs1005464 might be associated with variation in the mandibular condyles size.

CGOF++: Controllable 3D Face Synthesis With Conditional Generative Occupancy Fields.

Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images. However, previous methods focus on controllable 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes. In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors. At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh, built on top of EG3D (Chan et al. 2022), a recent tri-plane-based generative model. To achieve accurate control over fine-grained 3D face shapes of the synthesized images, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis framework. Experiments validate the effectiveness of the proposed method, which is able to generate high-fidelity face images and shows more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.

Inequality-Constrained 3D Morphable Face Model Fitting.

3D morphable model (3DMM) fitting on 2D data is traditionally done via unconstrained optimization with regularization terms to ensure that the result is a plausible face shape and is consistent with a set of 2D landmarks. This paper presents inequality-constrained 3DMM fitting as the first alternative to regularization in optimization-based 3DMM fitting. Inequality constraints on the 3DMM's shape coefficients ensure face-like shapes without modifying the objective function for smoothness, thus allowing for more flexibility to capture person-specific shape details. Moreover, inequality constraints on landmarks increase robustness in a way that does not require per-image tuning. We show that the proposed method stands out with its ability to estimate person-specific face shapes by jointly fitting a 3DMM to multiple frames of a person. Further, when used with a robust objective function, namely gradient correlation, the method can work "in-the-wild" even with a 3DMM constructed from controlled data. Lastly, we show how to use the log-barrier method to efficiently implement the method. To our knowledge, we present the first 3DMM fitting framework that requires no learning yet is accurate, robust, and efficient. The absence of learning enables a generic solution that allows flexibility in the input image size, interchangeable morphable models, and incorporation of camera matrix.

A Novel Method Based on Topological Perception Theory for 3D Landmark Building Model Retrieval

A Novel Method Based on Topological Perception Theory for 3D Landmark Building Model Retrieval

Two-Dimensional Geometric Morphometric Method on Frontal Sinus for Race Estimation: A Lateral Skull Radiograph Study

Introduction: Race estimation of unknown individual is essential in forensic investigation. The resiliency of frontal sinus makes it a potential tool for biological profiling, particularly in cases where fragmented skeleton persists. Geometric morphometrics is an efficient way to characterise shape. However, the use of frontal sinus to identify race of Malaysians is yet to be investigated. This research employed a two-dimensional (2D) geometric morphometric to examine the morphological differences of the frontal sinus among the major races in Malaysia. Methods: Lateral skull radiographs which comprising of 453 adult Malaysian (151 Malays, Chinese and Indian respectively) were used. The 2D landmarks of eight were placed on the digitalized radiographs and 2D geometric morphometric analysis was performed using MorphoJ software. Results: Procrustes ANOVA revealed a significantly different frontal sinus shape (p-value < 0.05) between races. Canonical variate analysis showed significantly different frontal sinus morphology (p-value < 0.05) between Malay and Indian as well as Chinese and Indian. Discriminant function analysis with cross-validation demonstrated a 57.4% accuracy rate. Conclusion: This population-specific study based on frontal sinus of Malaysians using the 2D geometric morphometric, though less reliable, sheds new light on the potential applicability of this method for race estimation purpose.

Inner ear morphology in wild versus laboratory house mice.

The semicircular canals of the inner ear are involved in balance and velocity control. Being crucial to ensure efficient mobility, their morphology exhibits an evolutionary conservatism attributed to stabilizing selection. Release of selection in slow-moving animals has been argued to lead to morphological divergence and increased inter-individual variation. In its natural habitat, the house mouse Mus musculus moves in a tridimensional space where efficient balance is required. In contrast, laboratory mice in standard cages are severely restricted in their ability to move, which possibly reduces selection on the inner ear morphology. This effect was tested by comparing four groups of mice: several populations of wild mice trapped in commensal habitats in France; their second-generation laboratory offspring, to assess plastic effects related to breeding conditions; a standard laboratory strain (Swiss) that evolved for many generations in a regime of mobility reduction; and hybrids between wild offspring and Swiss mice. The morphology of the semicircular canals was quantified using a set of 3D landmarks and semi-landmarks analyzed using geometric morphometric protocols. Levels of inter-population, inter-individual (disparity) and intra-individual (asymmetry) variation were compared. All wild mice shared a similar inner ear morphology, in contrast to the important divergence of the Swiss strain. The release of selection in the laboratory strain obviously allowed for an important and rapid drift in the otherwise conserved structure. Shared traits between the inner ear of the lab strain and domestic pigs suggested a common response to mobility reduction in captivity. The lab-bred offspring of wild mice also differed from their wild relatives, suggesting plastic response related to maternal locomotory behavior, since inner ear morphology matures before birth in mammals. The signature observed in lab-bred wild mice and the lab strain was however not congruent, suggesting that plasticity did not participate to the divergence of the laboratory strain. However, contrary to the expectation, wild mice displayed slightly higher levels of inter-individual variation than laboratory mice, possibly due to the higher levels of genetic variance within and among wild populations compared to the lab strain. Differences in fluctuating asymmetry levels were detected, with the laboratory strain occasionally displaying higher asymmetry scores than its wild relatives. This suggests that there may indeed be a release of selection and/or a decrease in developmental stability in the laboratory strain.

Computational Approaches and Observer Variation in the 3D Musculoskeletal Modeling of the Heads of Anolis.

High-resolution imaging, 3D modeling, and quantitative analyses are equipping evolutionary biologists with new approaches to understanding the variation and evolution of the musculoskeletal system. However, challenges with interpreting DiceCT data and higher order use of modeled muscles have not yet been fully explored, and the error in and accuracy of some digital methods remain unclear. West Indian Anolis lizards are a model clade for exploring patterns in functional adaptation, ecomorphology, and sexual size dimorphism in vertebrates. These lizards possess numerous jaw muscles with potentially different anatomies that sculpt the adductor chamber of the skull. Here we test approaches to quantifying the musculoskeletal shape of the heads of two species of Anolis: A. pulchellus and A. sagrei. We employ comparative approaches such as DiceCT segmentation of jaw muscles, 3D surface attachment mapping, and 3D landmarking with the aim of exploring muscle volumes, 3D muscle fiber architecture, and sexual dimorphism of the skull. We then compare sources of measurement error in these 3D analyses while also presenting new 3D musculoskeletal data from the Anolis feeding apparatus. These findings demonstrate the accessibility and repeatability of these emerging techniques as well as provide details regarding the musculoskeletal anatomy of the heads of A. pulchellus and A. sagrei which show potential for further research of comparative biomechanics and evolution in the clade.

Multi-Angle Facial Recognition: Enhancing Biometric Security with a Broadly Positioned Stereo-Camera System

This study addresses the vulnerabilities of traditional monocular camera-based face recognition systems, emphasizing the need for improved security and reliability in biometric authentication under varying environmental conditions, lighting, and human poses. To counteract the risk of spoofing attacks using masks or static images, we introduce a multi-angle stereo camera system. This system is strategically designed to capture facial imagery from multiple perspectives, thereby enhancing depth perception and spatial accuracy, crucial for high-security authentication. Employing a novel image processing approach, the study integrates a Convolutional Neural Networks (CNN) with a simple Boolean operation to differentiate the landmarks detected on each camera. This method exploits CNN’s robust feature extraction capabilities and the effective usage of stereo camera, enabling precise detection and analysis of 3D facial landmarks. Such an approach significantly bolsters the system’s ability to differentiate between genuine faces and deceptive representations like masks or static images. Empirical results demonstrate that the stereo camera configuration substantially improves recognition accuracy, reducing both false positives and negatives, especially in controlled spoofing scenarios. The advanced 3D facial landmark detection further reinforces the system’s security. With its enhanced robustness and security, the developed system shows great potential for applications in areas requiring stringent identity verification, such as banking, public facilities, and smart home technologies.

Clinical validity and precision of deep learning-based cone-beam computed tomography automatic landmarking algorithm.

This study was performed to assess the clinical validity and accuracy of a deep learning-based automatic landmarking algorithm for cone-beam computed tomography (CBCT). Three-dimensional (3D) CBCT head measurements obtained through manual and automatic landmarking were compared. A total of 80 CBCT scans were divided into 3 groups: non-surgical (39 cases); surgical without hardware, namely surgical plates and mini-screws (9 cases); and surgical with hardware (32 cases). Each CBCT scan was analyzed to obtain 53 measurements, comprising 27 lengths, 21 angles, and 5 ratios, which were determined based on 65 landmarks identified using either a manual or a 3D automatic landmark detection method. In comparing measurement values derived from manual and artificial intelligence landmarking, 6 items displayed significant differences: R U6CP-L U6CP, R L3CP-L L3CP, S-N, Or_R-R U3CP, L1L to Me-GoL, and GoR-Gn/S-N (P<0.05). Of the 3 groups, the surgical scans without hardware exhibited the lowest error, reflecting the smallest difference in measurements between human- and artificial intelligence-based landmarking. The time required to identify 65 landmarks was approximately 40-60 minutes per CBCT volume when done manually, compared to 10.9 seconds for the artificial intelligence method (PC specifications: GeForce 2080Ti, 64GB RAM, and an Intel i7 CPU at 3.6 GHz). Measurements obtained with a deep learning-based CBCT automatic landmarking algorithm were similar in accuracy to values derived from manually determined points. By decreasing the time required to calculate these measurements, the efficiency of diagnosis and treatment may be improved.

Geo-Referencing and Analysis of Entities Extracted from Old Drawings and Photos Using Computer Vision and Deep Learning Algorithms

This study offers a quantitative solution that automates the creation of a historical timeline starting with old drawings from the beginning of the 18th century and ending with present-day photographs of the Old City of Jerusalem. This is performed using GIScience approaches, computer vision, and deep learning. The motivation to select the Old City of Jerusalem is the substantial availability of old archival drawings and photographs, owing to the area’s significance throughout the years. This task is challenging, as drawings, old photographs, and new photographs exhibit distinct characteristics. Our method encompasses several key components for the analyses: a 2D location recommendation engine, which detects an approximate location in the image of 3D landmarks; 2D landmarks to 3D conversion; and 2D polygonal areas to 3D GIS polylines conversion. This is applied to the segmentation of built areas. To achieve more accurate results, Meta’s Segment Anything model was utilized, which eliminates the need for extensive data preparation, training, and validation, thus optimizing the process. Using such techniques enabled us to examine the landscape development throughout the last three centuries and gain deeper insights concerning the evolution of prominent landmarks and features such as built area over time.

Eye-Tracking System with Low-End Hardware: Development and Evaluation

Eye-tracking systems have emerged as valuable tools in various research fields, including psychology, medicine, marketing, car safety, and advertising. However, the high costs of the necessary specialized hardware prevent the widespread adoption of these systems. Appearance-based gaze estimation techniques offer a cost-effective alternative that can rely solely on RGB cameras, albeit with reduced accuracy. Therefore, the aim of our work was to present a real-time eye-tracking system with low-end hardware that leverages appearance-based techniques while overcoming their drawbacks to make reliable gaze data accessible to more users. Our system employs fast and light machine learning algorithms from an external library called MediaPipe to identify 3D facial landmarks. Additionally, it uses a series of widely recognized computer vision techniques, like morphological transformations, to effectively track eye movements. The precision and accuracy of the developed system in recognizing saccades and fixations when the eye movements are mainly horizontal were tested through a quantitative comparison with the EyeLink 1000 Plus, a professional eye tracker. Based on the encouraging registered results, we think that it is possible to adopt the presented system as a tool to quickly retrieve reliable gaze information.