3D Facial Landmark Research Articles

A Survey Study of the 3D Facial Landmark Detection Techniques Used as a Screening Tool for Diagnosis of the Obstructive Sleep Apnea Syndrome.

Obstructive Sleep Apnea (OSA) is a common disorder affecting both adults and children. It is characterized by repeated episodes of apnea (stopped breathing) and hypopnea (reduced breathing), which result in intermittent hypoxia. We recognize pediatric and adult OSA, and this paper focuses on pediatric OSA. While adults often suffer from daytime sleepiness, children are more likely to develop behavioral abnormalities. Early diagnosis and treatment are important to prevent negative effects on children's development. Without the treatment, children may be at increased risk of developing high blood pressure or other heart problems. The gold standard for OSA diagnosis is the polysomnography (sleep study) PSG performed at a sleep center. Not only is it an expensive procedure, but it can also be very stressful, especially for children. Patients have to stay at the sleep center during the night. Therefore, screening tools are very important. Multiple studies have shown that OSA screening tools can be based on facial anatomical landmarks. Anatomical landmarks are landmarks located at specific anatomical locations. For the purpose of the screening tool, a specific list of anatomical locations needs to be identified. We are presenting a survey study of the automatic identification of these landmarks on 3D scans of the patient's head. We are considering and comparing both knowledge-based and AI-based identification techniques, with a focus on the development of the automatic OSA screening tool.

Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection

AbstractIn this paper, we examine three important issues in the practical use of state‐of‐the‐art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many facial landmark detectors require a face normalization step as a pre‐process, often accomplished by a separately trained neural network that crops and resizes the face in the input image. There is no guarantee that this pre‐trained network performs optimal face normalization for the task of landmark detection. Thus, we instead analyse the use of a spatial transformer network that is trained alongside the landmark detector in an unsupervised manner, jointly learning an optimal face normalization and landmark detection by a single neural network. Second, we show that modifying the output head of the landmark predictor to infer landmarks in a canonical 3D space rather than directly in 2D can further improve accuracy. To convert the predicted 3D landmarks into screen‐space, we additionally predict the camera intrinsics and head pose from the input image. As a side benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful in determining landmark visibility among other things. Third, when training a landmark detector on multiple datasets at the same time, annotation inconsistencies across datasets forces the network to produce a sub‐optimal average. We propose to add a semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark detectors, we specifically target a recently proposed continuous 2D landmark detector to demonstrate how each of our additions leads to meaningful improvements over the state‐of‐the‐art on standard benchmarks.

Automated 3D Perioral Landmark Detection Using High-Resolution Network: Artificial Intelligence-based Anthropometric Analysis.

Three-dimensional facial stereophotogrammetry, a convenient, noninvasive and highly reliable evaluation tool, has in recent years shown great potential in plastic surgery for preoperative planning and evaluating treatment efficacy. However, it requires manual identification of facial landmarks by trained evaluators to obtain anthropometric data, which takes much time and effort. Automatic 3D facial landmark localization has the potential to facilitate fast data acquisition and eliminate evaluator error. The aim of this work was to describe a novel deep-learning method based on dimension transformation and key-point detection for automated 3D perioral landmark annotation. After transforming a 3D facial model into 2D images, High-Resolution Network is implemented for key-point detection. The 2D coordinates of key points are then mapped back to the 3D model using mathematical methods to obtain the 3D landmark coordinates. This program was trained with 120 facial models and validated in 50 facial models. Our approach achieved a satisfactory mean [standard deviation] accuracy of 1.30 [0.68] mm error in landmark detection with a mean processing time of 5.2 [0.21] seconds per model. Subsequent analysis based on these landmarks showed mean errors of 0.87 [1.02] mm for linear measurements and 5.62° [6.61°] for angular measurements. This automated 3D perioral landmarking method could serve as an effective tool that enables fast and accurate anthropometric analysis of lip morphology for plastic surgery and aesthetic procedures.

Automatic labeling of 3D facial acupoint landmarks

<p>As special marks on a human face, facial landmarks reflect the facial features of various parts of the face, which is crucial in biomedicine and medical imaging. In addition, facial landmarks are also important features in computer vision such as face detection, face recognition, facial pose estimation, and facial animation. In this paper, we construct a 3D facial acupoint annotated dataset by labeling 37 facial acupoints on 846 neutral face triangle mesh on the FaceScape dataset. Based on these annotated data, we use a feature template matching method to realize the automatic annotation of 37 acupoints on triangle meshes. We used 40 meshes as the training set to extract the geometric patterns of 3D acupoints and then measured the performance of the automatic labeling algorithm on 20 meshes and 806 meshes as the test sets. In the training process, we extract the tangent plane for each landmark, project the neighbor vertices of the landmark to the tangent plane, and construct the feature image with <em>R</em> × <em>R</em> resolution through the bounding box of the projected points. In the testing process, we use the pattern images extracted during training to find the average features and use them as a guide to optimize the predicted landmarks. The experimental results show that our automatic acupoint labeling method has achieved good results.<strong></strong></p>

Multi-Angle Facial Recognition: Enhancing Biometric Security with a Broadly Positioned Stereo-Camera System

This study addresses the vulnerabilities of traditional monocular camera-based face recognition systems, emphasizing the need for improved security and reliability in biometric authentication under varying environmental conditions, lighting, and human poses. To counteract the risk of spoofing attacks using masks or static images, we introduce a multi-angle stereo camera system. This system is strategically designed to capture facial imagery from multiple perspectives, thereby enhancing depth perception and spatial accuracy, crucial for high-security authentication. Employing a novel image processing approach, the study integrates a Convolutional Neural Networks (CNN) with a simple Boolean operation to differentiate the landmarks detected on each camera. This method exploits CNN’s robust feature extraction capabilities and the effective usage of stereo camera, enabling precise detection and analysis of 3D facial landmarks. Such an approach significantly bolsters the system’s ability to differentiate between genuine faces and deceptive representations like masks or static images. Empirical results demonstrate that the stereo camera configuration substantially improves recognition accuracy, reducing both false positives and negatives, especially in controlled spoofing scenarios. The advanced 3D facial landmark detection further reinforces the system’s security. With its enhanced robustness and security, the developed system shows great potential for applications in areas requiring stringent identity verification, such as banking, public facilities, and smart home technologies.

Facial Similarity Measure for Recognizing Monozygotic Twins Utilizing 3D Facial Landmarks, Efficient Geodesic Distance Computation, and Machine Learning Algorithms

Facial Similarity Measure for Recognizing Monozygotic Twins Utilizing 3D Facial Landmarks, Efficient Geodesic Distance Computation, and Machine Learning Algorithms

Eye-Tracking System with Low-End Hardware: Development and Evaluation

Eye-tracking systems have emerged as valuable tools in various research fields, including psychology, medicine, marketing, car safety, and advertising. However, the high costs of the necessary specialized hardware prevent the widespread adoption of these systems. Appearance-based gaze estimation techniques offer a cost-effective alternative that can rely solely on RGB cameras, albeit with reduced accuracy. Therefore, the aim of our work was to present a real-time eye-tracking system with low-end hardware that leverages appearance-based techniques while overcoming their drawbacks to make reliable gaze data accessible to more users. Our system employs fast and light machine learning algorithms from an external library called MediaPipe to identify 3D facial landmarks. Additionally, it uses a series of widely recognized computer vision techniques, like morphological transformations, to effectively track eye movements. The precision and accuracy of the developed system in recognizing saccades and fixations when the eye movements are mainly horizontal were tested through a quantitative comparison with the EyeLink 1000 Plus, a professional eye tracker. Based on the encouraging registered results, we think that it is possible to adopt the presented system as a tool to quickly retrieve reliable gaze information.

In Cabin Driver Monitoring and Alerting System For Passenger cars using Machine Learning.

The primary causes of road accidents are driver errors. Such as excessive speeding, driver distraction, and driver sleepiness. Diver sleepiness and distraction can be controlled by implementing advanced technology such as an in-cabin driver monitoring system. This is a technological solution that uses a machine learning algorithm to improve road safety. This paper describes a new real-time driver monitoring and alerting system that solely monitors driver Sleepiness, diver distraction, and seatbelt wearing status in order to prevent road accidents. It employs the YOLOv5 deep learning algorithm to detect various types of distraction and For seatbelt wearing status and computer vision’s cvzone framework is used to detect sleepiness using 3D facial landmarking. In this system, The camera is mounted in the centre of the vehicle’s dashboard for data collection and real-time implementation. The Raspberry Pi 3b module is used for the Realtime implementation. According to the findings of this study, the system is capable of real-time implementation, and the machine learning model used in the study achieves an accuracy of more than 90%.

Free-HeadGAN: Neural Talking Head Synthesis With Explicit Gaze Control

We present Free-HeadGAN, a person-generic neural talking head synthesis system. We show that modeling faces with sparse 3D facial landmarks is sufficient for achieving state-of-the-art generative performance, without relying on strong statistical priors of the face, such as 3D Morphable Models. Apart from 3D pose and facial expressions, our method is capable of fully transferring the eye gaze, from a driving actor to a source identity. Our complete pipeline consists of three components: a canonical 3D key-point estimator that regresses 3D pose and expression-related deformations, a gaze estimation network and a generator that is built upon the architecture of HeadGAN. We further experiment with an extension of our generator to accommodate few-shot learning using an attention mechanism, in case multiple source images are available. Compared to recent methods for reenactment and motion transfer, our system achieves higher photo-realism combined with superior identity preservation, while offering explicit gaze control.

JR2Net: Joint Monocular 3D Face Reconstruction and Reenactment

Face reenactment and reconstruction benefit various applications in self-media, VR, etc. Recent face reenactment methods use 2D facial landmarks to implicitly retarget facial expressions and poses from driving videos to source images, while they suffer from pose and expression preservation issues for cross-identity scenarios, i.e., when the source and the driving subjects are different. Current self-supervised face reconstruction methods also demonstrate impressive results. However, these methods do not handle large expressions well, since their training data lacks samples of large expressions, and 2D facial attributes are inaccurate on such samples. To mitigate the above problems, we propose to explore the inner connection between the two tasks, i.e., using face reconstruction to provide sufficient 3D information for reenactment, and synthesizing videos paired with captured face model parameters through face reenactment to enhance the expression module of face reconstruction. In particular, we propose a novel cascade framework named JR2Net for Joint Face Reconstruction and Reenactment, which begins with the training of a coarse reconstruction network, followed by a 3D-aware face reenactment network based on the coarse reconstruction results. In the end, we train an expression tracking network based on our synthesized videos composed by image-face model parameter pairs. Such an expression tracking network can further enhance the coarse face reconstruction. Extensive experiments show that our JR2Net outperforms the state-of-the-art methods on several face reconstruction and reenactment benchmarks.

An Automated Method of 3D Facial Soft Tissue Landmark Prediction Based on Object Detection and Deep Learning.

Three-dimensional facial soft tissue landmark prediction is an important tool in dentistry, for which several methods have been developed in recent years, including a deep learning algorithm which relies on converting 3D models into 2D maps, which results in the loss of information and precision. This study proposes a neural network architecture capable of directly predicting landmarks from a 3D facial soft tissue model. Firstly, the range of each organ is obtained by an object detection network. Secondly, the prediction networks obtain landmarks from the 3D models of different organs. The mean error of this method in local experiments is 2.62±2.39, which is lower than that in other machine learning algorithms or geometric information algorithms. Additionally, over 72% of the mean error of test data falls within ±2.5 mm, and 100% falls within 3 mm. Moreover, this method can predict 32 landmarks, which is higher than any other machine learning-based algorithm. According to the results, the proposed method can precisely predict a large number of 3D facial soft tissue landmarks, which gives the feasibility of directly using 3D models for prediction.

Accuracy and reliability of automated three-dimensional facial landmarking in medical and biological studies. A systematic review.

3D facial landmarking is becoming a fundamental part of clinical and biological applications. Manual landmarking is time consuming and prone to cumulative errors, so attempts have been made to automate 3D facial landmarking. However, data in the literature are sparse. The objectives of this study are to investigate current evidence for the accuracy and reliability of various 3D facial automated landmarking methods used in medical and biological studies and evaluate their performance against the manual annotation method. Electronic and manual searches of the literature were performed in April 2021. Only studies that were published in English and evaluated the accuracy of automated landmarking algorithms in 3D facial images for medical or biological settings were included. Two authors independently screened the articles for eligibility. The QUADAS-2 tool was used for the quality analysis of the included studies. Due to the heterogeneity of the selected studies, a meta-analysis was not possible, so a narrative synthesis of the findings was performed. From 1002 identified records, after applying the inclusion and exclusion criteria, 14 articles were ultimately selected, read, and critically analysed. Different algorithms were used for the automated 3D landmarking of various numbers of facial landmarks ranging from 10 to 29 landmarks. The average difference between the manual and automated methods ranged from 0.67 to 4.73 mm, and the best performance was achieved using deep learning models. Poor study design and inadequate reporting were found in the implementation of the reference standards and population selection for the intended studies, which could have led to overfitting of the tested algorithm. This systematic review was limited by the quality of the included studies and uncovered several methodological limitations evident in the corresponding literature. Compared to manual landmarking, automated Landmark localization of individual facial landmarks reported in the literature is not accurate enough to allow their use for clinical purposes. This result indicates that automatic facial landmarking is still developing, and further studies are required to develop a system that could match or exceed the performance of the current gold standard. PROSPERO: CRD42021241531.

Evaluation of Camera Pose Estimation Using Human Head Pose Estimation

We introduce and evaluate a novel camera pose estimation framework that uses the human head as a calibration object. The proposed method facilitates extrinsic calibration from 2D input images (NIR and/or RGB), while merely relying on the detected human head, without the need for depth information. The approach is applicable to single cameras or multi-camera networks. Our implementation uses a fine-tuned deep learning-based 2D human facial landmark detector to estimate the 3D human head pose by fitting a 3D head model to the detected 2D facial landmarks. Our work focuses on an evaluation of the proposed approach on real multi-camera recordings and synthetic renderings to determine the accuracy of the pose estimation results and their applicability. We assess the robustness of our method against different input parameters, such as varying relative camera positions, variations of head models, face occlusions (by masks, sun glasses, etc.), potential biases and variance among humans. Based on the experimental results, we expect our approach to be effective for numerous use cases including automotive attention monitoring, robotics, VR/AR and other scenarios where ease of handling outweighs accuracy.

SqueezExpNet: Dual-stage convolutional neural network for accurate facial expression recognition with attention mechanism

Facial expression recognition (FER) using a deep convolutional neural network (DCNN) is important and challenging. Although a substantial effort is made to increase FER accuracy through DCNN, previous studies are still not sufficiently generalisable for real-world applications. Traditional FER studies are mainly limited to controlled lab-posed frontal facial images, which lack the challenges of motion blur, head poses, occlusions, face deformations and lighting under uncontrolled conditions. In this work, we proposed a SqueezExpNet architecture that can take advantage of local and global facial information for a highly accurate FER system that can handle environmental variations. Our network was divided into two stages: a geometrical attention stage that possesses a SqueezeNet-like architecture to obtain local highlight information and a spatial texture stage comprising several squeezed and expanded layers to exploit high-level global features. In particular, we created a weighted mask of 3D face landmarks and used element-wise multiplication with a spatial feature in the first stage to draw attention to important local facial regions. Next, we input the face spatial image and its augmentations into the second stage of the network. Finally, like a classifier, a recurrent neural network was designed to collaborate the highlighted information from dual stages rather than simply using the SoftMax function, thereby aiding in overcoming the uncertainties. Experiments covering basic and compound FER tasks were performed using the three leading facial expression datasets. Our strategy outperformed the existing DCNN methods and achieved state-of-the-art results. The developed architecture, adopted research methodology and reported findings may find potential applications of real-time FER in surveillance, health and feedback systems.

Comparison Study of Extraction Accuracy of 3D Facial Anatomical Landmarks Based on Non-Rigid Registration of Face Template

(1) Background: Three-dimensional (3D) facial anatomical landmarks are the premise and foundation of facial morphology analysis. At present, there is no ideal automatic determination method for 3D facial anatomical landmarks. This research aims to realize the automatic determination of 3D facial anatomical landmarks based on the non-rigid registration algorithm developed by our research team and to evaluate its landmark localization accuracy. (2) Methods: A 3D facial scanner, Face Scan, was used to collect 3D facial data of 20 adult males without significant facial deformities. Using the radial basis function optimized non-rigid registration algorithm, TH-OCR, developed by our research team (experimental group: TH group) and the non-rigid registration algorithm, MeshMonk (control group: MM group), a 3D face template constructed in our previous research was deformed and registered to each participant's data. The automatic determination of 3D facial anatomical landmarks was realized according to the index of 32 facial anatomical landmarks determined on the 3D face template. Considering these 32 facial anatomical landmarks manually selected by experts on the 3D facial data as the gold standard, the distance between the automatically determined and the corresponding manually selected facial anatomical landmarks was calculated as the "landmark localization error" to evaluate the effect and feasibility of the automatic determination method (template method). (3) Results: The mean landmark localization error of all facial anatomical landmarks in the TH and MM groups was 2.34 ± 1.76 mm and 2.16 ± 1.97 mm, respectively. The automatic determination of the anatomical landmarks in the middle face was better than that in the upper and lower face in both groups. Further, the automatic determination of anatomical landmarks in the center of the face was better than in the marginal part. (4) Conclusions: In this study, the automatic determination of 3D facial anatomical landmarks was realized based on non-rigid registration algorithms. There is no significant difference in the automatic landmark localization accuracy between the TH-OCR algorithm and the MeshMonk algorithm, and both can meet the needs of oral clinical applications to a certain extent.

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews.

Previous research has shown that eyebrow movement during speech exhibits a systematic relationship with intonation: brow raises tend to be aligned with pitch accents, typically preceding them. The present study approaches the question of temporal alignment between brow movement and intonation from a new angle. The study makes use of footage from the Late Night Show with David Letterman, processed with 3D facial landmark detection. Pitch is modeled as a sinusoidal function whose parameters are correlated with the maximum height of the eyebrows in a brow raise. The results confirm some previous findings on audiovisual prosody but lead to new insights as well. First, the shape of the pitch signal in a region of approx. 630 ms before the brow raise is not random and tends to display a specific shape. Second, while being less informative than the post-peak pitch, the pitch signal in the pre-peak region also exhibits correlations with the magnitude of the associated brow raises. Both of these results point to early preparatory action in the speech signal, calling into question the visual-precedes-acoustic assumption. The results are interpreted as supporting a unified view of gesture/speech co-production that regards both signals as manifestations of a single communicative act.

DEPTH MEASUREMENT OF HUMANS USING WEB CAM

Artificial Intelligence has gradually become the mainstream in the detection algorithms because of its powerful feature extraction ability and adaptive ability. However, how to guarantee the accuracy and speed is still a huge challenge in the field of depth measurement. We implement face mesh from media pipe (module) which is developed by google. Media Pipe Face Mesh is a face geometry solution that estimates 468 3D face landmarks in real-time even on mobile devices. It employs machine learning (ML) to infer the 3D surface geometry, requiring a single camera input without the need for a dedicated depth sensor. Using face mesh, we take points from face and depth is measured. This makes project more accurate and speed. In our project we measure real time 3D depth, using these we make our text flexible with depth measured from webcam. When we move closer to our webcam text size reduces, and when we go far from webcam, text size is enhanced, which is accordance to the convenience to our end users. When we implement this in real life, end users can see the captions comfortably from any place they are watching the screen only. Keywords: OpenCV, Mediapipe , FaceMesh, Windows, Python.

Geometric Deep Neural Network Using Rigid and Non-Rigid Transformations for Landmark-Based Human Behavior Analysis.

Deep learning architectures, albeit successful in most computer vision tasks, were designed for data with an underlying Euclidean structure, which is not usually fulfilled since pre-processed data may lie on a non-linear space. In this article, we propose a geometric deep learning approach using rigid and non-rigid transformations, named KShapenet, for 2D and 3D landmark-based human motion analysis. Landmark configuration sequences are first modeled as trajectories on Kendall's shape space and then mapped to a linear tangent space. The resulting structured data are then input to a deep learning architecture, which includes a layer that optimizes over rigid and non-rigid transformations of landmark configurations, followed by a CNN-LSTM network. We apply KShapenet to 3D human landmark sequences for action and gait recognition, and 2D facial landmark sequences for expression recognition, and demonstrate the competitiveness of the proposed approach with respect to state-of-the-art.

Analyzing important statistical features from facial behavior in human depression using XGBoost

Major Depressive Disorder (MDD) has been known as one of the most prevalent mental disorders whose symptoms can be observed from changes in facial behaviors. Previous studies had attempted to build Machine Learning (ML) models to assess depression severity using such features but few have utilized these models to determine key facial behaviors for MDD. In this study, we used video data to assess the severity of MDD and determine important features based on three approaches (XGBoost, Spearman’s correlation, and t-test). In addition, there is the Facial Action Coding System (FACS) framework that allows visual data such as changes in facial behavior to be modeled as time series data. The results show that the XGBoost model obtained the best results when trained using features selected through the t-test statistical method with 5.387 MAE, 6.266 RMSE, and 0.042 R2 . The majority of the important features consist of Action Unit (AU) and Features 3D around the regions of the left eye, right cheek, and lip area. However, the majority of the important features discovered from the three approaches, are the first derivatives of the 3D facial landmark coordinates of the cheeks, eyes, and lips, especially along the z-axis. However, the variables used in this research are limited to the first derivatives, which meant that usages of wider variations of facial behavior data may further be studied so that Computer-Aided Diagnosis (CAD) systems for mental disorders may be realized in the future.

FACIAL LANDMARK DETECTION WITH MEDIAPIPE & CREATING ANIMATED SNAPCHAT FILTERS

MediaPipe Face Mesh is a face geometry solution that estimates 468 3D face landmarks in real-time even on mobile devices. It employs machine learning (ML) to infer the 3D surface geometry, requiring only a single camera input without the need for a dedicated depth sensor. Utilizing lightweight model architectures together with GPU acceleration throughout the pipeline, the solution delivers real-time performance critical for live experiences. There are two models and they are face detection and face landmark model.Face landmark screen coordinates are converted into the Metric 3D space coordinates; face pose transformation matrix is estimated as a rigid linear mapping from the canonical face metric landmark set into the runtime face metric landmark set in a way that minimizes a difference between the two; a face mesh is created using the runtime face metric landmarks as the vertex positions (XYZ), while both the vertex texture coordinates (UV) and the triangular topology are inherited from the canonical face model.Snapchat Filter Controlled by Facial Expressions: Currently, our face expression recognizer can check whether the eyes and mouth are open or not so to get the most out of it, we can overlay scalable eyes images on top of the eyes of the user when his eyes are open and a video of fire coming out of the mouth of the user when the mouth is open.