Facial Landmark Detection Research Articles

Region-Aware Deep Feature-Fused Network for Robust Facial Landmark Localization

In facial landmark localization, facial region initialization usually plays an important role in guiding the model to learn critical face features. Most facial landmark detectors assume a well-cropped face as input and may underperform in real applications if the input is unexpected. To alleviate this problem, we present a region-aware deep feature-fused network (RDFN). The RDFN consists of a region detection subnetwork and a region-wise landmark localization subnetwork to explicitly solve the input initialization problem and derive the landmark score maps, respectively. To exploit the association between tasks, we develop a cross-task feature fusion scheme to extract multi-semantic region features while trading off their importance in different dimensions via global channel attention and global spatial attention. Furthermore, we design a within-task feature fusion scheme to capture the multi-scale context and improve the gradient flow for the landmark localization subnetwork. At the inference stage, a location reweighting strategy is employed to transform the score maps into 2D landmark coordinates. Extensive experimental results demonstrate that our method has competitive performance compared to recent state-of-the-art methods, achieving NMEs of 3.28%, 1.48%, and 3.43% on the 300W, AFLW, and COFW datasets, respectively.

AUTOMATIC DETECTION OF FACIAL LOCATIONS TO MEASURE FACIAL ASYMMETRY AFTER PAEDIATRIC RADIOTHERAPY

Abstract AIMS Children receiving radiotherapy for head and neck tumours, such as Head and Neck Rhabdomyosarcoma (HN- RMS), often experience facial asymmetry later in life. Here, we present a convolutional neural network (CNN) that will enable the automatic detection of facial anatomical landmarks on MR images taken during follow-up examinations. This model will facilitate quantitative tracking of facial asymmetry, thus revealing the centres of growth that are most affected by radiation, with the ultimate goal of defining precise dose tolerance levels. METHOD A dataset of 61 paediatric MRI images was used to train separate models that infer the locations of four facial landmarks. These locations had previously been manually labelled by two independent observers. Before training, the images were normalised and registered to optimise model performance. The model was judged to be of acceptable accuracy if the distances between the observed landmark (gold-standard location) and the location predicted by the trained models were smaller than the inter-observer distances of each facial landmark. RESULTS The model predicted landmarks which were closer to the gold-standard location than the inter-observer locations. In detail: • Optic chiasm: 0.81 ± 0.45 mm (inter-observer 1.49 mm). • Nasion: 0.70 ± 0.62 mm (inter-observer 3.24 mm). • Left Ectoconchion: 1.22 ± 0.44 mm (inter-observer 3.58 mm). • Right Ectoconchion: 0.80 ± 0.82 mm (inter-observer 4.04 mm). CONCLUSIONS This research shows promising potential for a CNN to accurately locate head and neck landmarks in paediatric MRI images. This tool can label craniofacial landmarks in large clinical datasets quicker and more accurately than can be achieved through manual techniques.

Angular Photogrammetric Analysis of Facial Soft Tissue by Image Processing Algorithms.

The main aim of this study was to present an automatic method based on image processing algorithms for facial anatomical landmark localization and angular photogrammetric analysis applicable for rhinoplasty surgery. We studied and measured color profile photographs of 100 patients before and after rhinoplasty surgery. In facial anthropometry analysis, anatomical landmarks are often defined by specialists, manually. This process is time-consuming and requires training and skill. The Cascade Regression Method (CRM) was utilized for facial landmark detection to overcome the mentioned problem. In this study, 11 anatomical landmarks were used to measure 9 facial angular metrics. Finally, a t-test (with the significance level set at a p-value of 0.05) was applied to analyze before surgery versus after surgery comparisons. Experimental results dedicated that there is a significance difference (p < 0.001) in nasofrontal, nasolabial, mentolabial, nasomental, facial convexity including nose, facial convexity excluding nose, projection of the upper lip to chin, and H angles before and after surgery. Also, results showed that there is not a significance difference in nose tip angle. We believe that the presented system can aim to reduce the personal errors made by manual measurement and to facilitate facial anthropometry analysis before and after surgery with high accuracy. Also, the normative data for Iranian women can be used as a guide for the diagnosis and planning of oral and maxillofacial, ENT, and plastic surgeries. This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Thermal comfort prediction based on automated extraction of skin temperature of face component on thermal image

This paper proposes a framework for predicting thermal comfort based on the automated extraction of skin temperature features from thermal images. This aims to non-invasively collect thermal comfort information for occupant-centric control (OCC), significantly impacting energy consumption, health, and overall well-being. The proposed framework adopts 68-point face landmarks to identify regions of interest (ROIs) of face components in thermal images, and subsequently extracts skin temperature features from those ROIs to predict thermal comfort. To assess its performance, the face landmark detection performance was evaluated using various colormaps on thermal images. Furthermore, the validity of the proposed skin temperature features extraction from ROIs of face components was evaluated, determining which skin temperature features are effectively entered into machine learning models. Additionally, the reliability of the framework for predicting thermal comfort under different head pose conditions was evaluated to ensure its validity. The results revealed that the proposed framework achieved an accuracy rate of 90.26% and showed robustness even in the extreme head pose. The study's findings suggest that the proposed framework can make OCC more effective based on more accurate thermal comfort prediction using a single thermal camera device.

AFLFP: A Database With Annotated Facial Landmarks for Facial Palsy

Facial landmark detection is a crucial step for the task of computer-aided facial palsy diagnosis, which enables to focus on the affected facial regions for learning asymmetry, shape, and texture features of facial palsy. However, it is still very challenging to accurately detect salient landmarks on facial palsy images due to the unavailability of sufficient training databases providing annotated facial landmark images of facial palsy. To this end, we present a database in this article named annotated facial landmarks for facial palsy (AFLFP). AFLFP is a diverse, and reliable database that contains facial images with 16-class facial expressions of asymmetric facial expressions from 88 subjects. Each facial image is independently and manually annotated with 68 facial landmarks. This database is the first public manually annotated facial landmark database for facial palsy so far. Furthermore, to establish the benchmark results for the proposed database, we propose a deep neural network (DNN) baseline with a two-stage cascaded fully convolutional network (FCN), which can detect facial landmarks in facial palsy from coarse to fine. The comprehensive experiments show that the proposed method performs better than the mainstream methods of machine-and deep-learning. And we have also compared the performance using normal-and palsy-faces, respectively, as the training data. The comparison results show that there are significant differences between them in terms of facial landmark detection, which further confirms the necessity to develop a facial landmark database specifically for facial palsy.

Neonatal Face and Facial Landmark Detection from Video Recordings.

This paper explores automated face and facial landmark detection of neonates, which is an important first step in many video-based neonatal health applications, such as vital sign estimation, pain assessment, sleep-wake classification, and jaundice detection. Utilising three publicly available datasets of neonates in the clinical environment, 366 images (258 subjects) and 89 (66 subjects) were annotated for training and testing, respectively. Transfer learning was applied to two YOLO-based models, with input training images augmented with random horizontal flipping, photo-metric colour distortion, translation and scaling during each training epoch. Additionally, the re-orientation of input images and fusion of trained deep learning models was explored. Our proposed model based on YOLOv7Face outperformed existing methods with a mean average precision of 84.8% for face detection, and a normalised mean error of 0.072 for facial landmark detection. Overall, this will assist in the development of fully automated neonatal health assessment algorithms.Clinical relevance- Accurate face and facial landmark detection provides an automated and non-contact option to assist in video-based neonatal health applications.

An optimized deep convolutional neural network for adaptive learning using feature fusion in multimodal data

The outbreak of COVID-19 has caused an unprecedented increase in the usage of e-Learning platforms. The closure of educational institutions globally has significantly impacted the traditional education system. Without physical interaction, engaging students in an e-Learning environment has become a major challenge for teachers and e-Learning platform providers. Unlike a face-to-face classroom, where teachers can easily monitor students’ behaviour and adapt the learning content according to their needs, this is impossible in an e-Learning environment. This study introduces an optimized deep learning-based approach for detecting student engagement levels to ensure students remain connected to the learning process. The proposed optimized model uses facial emotion recognition and head movement detection to track real-time engagement levels for big data as real-time-based face and head datasets are analysed. The system leverages facial landmark detection to monitor head movements and deep learning models such as VGG19 and ResNet50 for facial emotion recognition. The system combines the output of both approaches to accurately predict the student engagement state as either ‘engaged’ or ‘disengaged’ with an accuracy rate of 91.67%.

A Smartphone Application for Personalized Tooth Shade Determination.

Tooth shade determination methods for evaluating the effectiveness of whitening products at home are limited. In this study, an iPhone app for personalized tooth shade determination was developed. While capturing dental photographs in selfie mode before and after whitening, the app can maintain consistent illumination and tooth appearance conditions that affect tooth color measurement. An ambient light sensor was used to standardize the illumination conditions. To maintain consistent tooth appearance conditions determined by appropriately opening the mouth, facial landmark detection, an artificial intelligence technique that estimates key face parts and outlines, was used. The effectiveness of the app in ensuring uniform tooth appearance was investigated through color measurements of the upper incisors of seven participants via photographs captured in succession. The coefficients of variation for incisors L*, a*, and b* were less than 0.0256 (95% CI, 0.0173-0.0338), 0.2748 (0.1596-0.3899), and 0.1053 (0.0078-0.2028), respectively. To examine the feasibility of the app for tooth shade determination, gel whitening after pseudo-staining by coffee and grape juice was performed. Consequently, whitening results were evaluated by monitoring the ∆Eab color difference values (1.3 unit minimum). Although tooth shade determination remains a relative quantification method, the proposed method can support evidence-based selection of whitening products.

Robust and Precise Facial Landmark Detection by Self-Calibrated Pose Attention Network.

Current fully supervised facial landmark detection methods have progressed rapidly and achieved remarkable performance. However, they still suffer when coping with faces under large poses and heavy occlusions for inaccurate facial shape constraints and insufficient labeled training samples. In this article, we propose a semisupervised framework, that is, a self-calibrated pose attention network (SCPAN) to achieve more robust and precise facial landmark detection in challenging scenarios. To be specific, a boundary-aware landmark intensity (BALI) field is proposed to model more effective facial shape constraints by fusing boundary and landmark intensity field information. Moreover, a self-calibrated pose attention (SCPA) model is designed to provide a self-learned objective function that enforces intermediate supervision without label information by introducing a self-calibrated mechanism and a pose attention mask. We show that by integrating the BALI fields and SCPA model into a novel SCPAN, more facial prior knowledge can be learned and the detection accuracy and robustness of our method for faces with large poses and heavy occlusions have been improved. The experimental results obtained for challenging benchmark datasets demonstrate that our approach outperforms state-of-the-art methods in the literature.

Gaze Assisted Transaction

Abstract: Automated Teller Machine (ATM) was introduced in the year 1959 for encouraging self service in retail banking. The single smart card ATM for multiple bank accounts is used for security applications for withdrawing cash from ATM machines. Eye tracking has been introduced as a way of direct interaction which could pave the way for new technologies and devices to be introduced for end-users. It is becoming a popular way of interaction. Gaze coordinates could be used to pinpoint the target selected by the user on the screen, then proceed to do a command more efficiently. We hypothesize that the limitations and disadvantages mentioned for touch screen interfaces can be improved by adding gaze capabilities. It is worth mentioning that gaze along with the touch method. Therefore, the targeting can be done via gaze and the selection by touch, which will improve the efficiency and also the screen is not obscured by hand during targeting and even increases the security. The proposed project is based on Event based architecture. CNN algorithm has been chosen to improve the efficiency of the project. The model is easily adaptable to complex, often chaotic environments and the system is easily extendable when new event types appear. The User interface will be designed to be user friendly and comfortable. The design uses a toolkit DLIB for making real world machine learning and data analysis applications. Dlib is majorly used for face detection and facial landmark detection. The system can be divided into two parts. The first part consists of eye typing implementation and the second part is text entry into the system.

Attendance Compilation by Facial Recognition Methods of Image Processing: A Review

Abstract: In recent years, researchers have continued to refine and improve deep learning-based approaches to image processing, as well as exploring new areas such as generative adversarial networks (GANs) and reinforcement learning. This paper provides a comprehensive survey of deep learning-based methods for face recognition, including CNN-based models, auto encoder models, and hybrid models. These papers demonstrate the effectiveness of deep convolutional neural networks (CNNs) in face detection, recognition, and attendance compilation, achieving state-of-the-art accuracy on several benchmark datasets, including LFW, YouTube Faces, and YTF datasets. The Efficient Net model is a family of CNNs that achieves state of the art accuracy on multiple image recognition benchmarks while being significantly smaller and faster than previous models. The Arc Face loss function is used for facial landmark detection and gender classification in facial images.

Combining CNNs and Markov-like Models for Facial Landmark Detection with Spatial Consistency Estimates.

The accurate localization of facial landmarks is essential for several tasks, including face recognition, head pose estimation, facial region extraction, and emotion detection. Although the number of required landmarks is task-specific, models are typically trained on all available landmarks in the datasets, limiting efficiency. Furthermore, model performance is strongly influenced by scale-dependent local appearance information around landmarks and the global shape information generated by them. To account for this, we propose a lightweight hybrid model for facial landmark detection designed specifically for pupil region extraction. Our design combines a convolutional neural network (CNN) with a Markov random field (MRF)-like process trained on only 17 carefully selected landmarks. The advantage of our model is the ability to run different image scales on the same convolutional layers, resulting in a significant reduction in model size. In addition, we employ an approximation of the MRF that is run on a subset of landmarks to validate the spatial consistency of the generated shape. This validation process is performed against a learned conditional distribution, expressing the location of one landmark relative to its neighbor. Experimental results on popular facial landmark localization datasets such as 300 w, WFLW, and HELEN demonstrate the accuracy of our proposed model. Furthermore, our model achieves state-of-the-art performance on a well-defined robustness metric. In conclusion, the results demonstrate the ability of our lightweight model to filter out spatially inconsistent predictions, even with significantly fewer training landmarks.

Blink-To-Live eye-based communication system for users with speech impairments

Eye-based communication languages such as Blink-To-Speak play a key role in expressing the needs and emotions of patients with motor neuron disorders. Most invented eye-based tracking systems are complex and not affordable in low-income countries. Blink-To-Live is an eye-tracking system based on a modified Blink-To-Speak language and computer vision for patients with speech impairments. A mobile phone camera tracks the patient’s eyes by sending real-time video frames to computer vision modules for facial landmarks detection, eye identification and tracking. There are four defined key alphabets in the Blink-To-Live eye-based communication language: Left, Right, Up, and Blink. These eye gestures encode more than 60 daily life commands expressed by a sequence of three eye movement states. Once the eye gestures encoded sentences are generated, the translation module will display the phrases in the patient’s native speech on the phone screen, and the synthesized voice can be heard. A prototype of the Blink-To-Live system is evaluated using normal cases with different demographic characteristics. Unlike the other sensor-based eye-tracking systems, Blink-To-Live is simple, flexible, and cost-efficient, with no dependency on specific software or hardware requirements. The software and its source are available from the GitHub repository (https://github.com/ZW01f/Blink-To-Live).

Heatmap-Guided Selective Feature Attention for Robust Cascaded Face Alignment

Face alignment methods have been actively studied using coordinate and heatmap regression tasks. Although these regression tasks have the same objective for facial landmark detection, each task requires different valid feature maps. Therefore, it is not easy to simultaneously train two kinds of tasks with a multi-task learning network structure. Some studies have proposed multi-task learning networks with two kinds of tasks, but they do not suggest an efficient network that can train them simultaneously because of the shared noisy feature maps. In this paper, we propose a heatmap-guided selective feature attention for robust cascaded face alignment based on multi-task learning, which improves the performance of face alignment by efficiently training coordinate regression and heatmap regression. The proposed network improves the performance of face alignment by selecting valid feature maps for heatmap and coordinate regression and using the background propagation connection for tasks. This study also uses a refinement strategy that detects global landmarks through a heatmap regression task and then localizes landmarks through cascaded coordinate regression tasks. To evaluate the proposed network, we tested it on the 300W, AFLW, COFW, and WFLW datasets and obtained results that outperformed other state-of-the-art networks.

Robust facial landmark detection by probability‐guided hourglass network

AbstractThe absence of local features and global shape constraints severely limits the performance of the hourglass network for facial landmark detection in unconstrained environments. Moreover, diverse feature types and scales may result in low accuracy. This paper proposes a probability‐guided hourglass network to enhance the shape constraints for robust facial landmark detection. Firstly, a multi‐scale pre‐processing module is designed to extract features at different scales. Secondly, based on the heatmaps generated by the stacked hourglass network, the coarse localizations are obtained, while the probability maps are generated with local features. Finally, a probability‐based boundary regression method is proposed and the hausdorff distance is modified as the loss function to constrain the feature shape. Adaptive weights are also added to the loss function, which can help relieve the data imbalance problem. Subjective and objective experimental results on the challenging datasets show that this method outperforms the state‐of‐the‐arts on unconstrained conditions.

Deep Learning Algorithms to Identify Autism Spectrum Disorder in Children-Based Facial Landmarks

People with autistic spectrum disorders (ASDs) have difficulty recognizing and engaging with others. The symptoms of ASD may occur in a wide range of situations. There are numerous different types of functions for people with an ASD. Although it may be possible to reduce the symptoms of ASD and enhance the quality of life with appropriate treatment and support, there is no cure. Developing expert systems for identifying ASD based on the facial landmarks of children is the main contribution for improvements in the healthcare system in Saudi Arabia for detecting ASD at an early stage. However, deep learning algorithms have provided outstanding performances in a variety of pattern-recognition studies. The use of techniques based on convolutional neural networks (CNNs) has been proposed by several scholars to use in investigations of ASD. At present, there is no diagnostic test available for ASD, making this diagnosis challenging. Clinicians focus on a patient’s behavior and developmental history. Therefore, using the facial landmarks of children has become very important for detecting ASDs as the face is thought to be a reflection of the brain; it has the potential to be used as a diagnostic biomarker, in addition to being an easy-to-use and practical tool for the early detection of ASDs. This study uses a variety of transfer learning approaches observed in deep CNNs to recognize autistic children based on facial landmark detection. An empirical study is conducted to discover the ideal settings for the optimizer and hyperparameters in the CNN model so that its prediction accuracy can be improved. A transfer learning approach, such as MobileNetV2 and hybrid VGG19, is used with different machine learning programs, such as logistic regression, a linear support vector machine (linear SVC), random forest, decision tree, gradient boosting, MLPClassifier, and K-nearest neighbors. The deep learning models are examined using a standard research dataset from Kaggle, which contains 2940 images of autistic and non-autistic children. The MobileNetV2 model achieved an accuracy of 92% on the test set. The results of the proposed research indicate that MobileNetV2 transfer learning strategies are better than those developed in existing systems. The updated version of our model has the potential to assist physicians in verifying the accuracy of their first screening for ASDs in child patients.

Dual Learning for Joint Facial Landmark Detection and Action Unit Recognition

Facial landmark detection and action unit~(AU) recognition are two essential tasks in facial analysis. Previous works rarely consider the relationship between these complementary tasks. In this paper, we introduce a novel multi-task dual learning framework to exploit the relationship between facial landmark detection and AU recognition while simultaneously addressing both tasks. When both tasks share middle-level features, common patterns can be exploited and middle- and high-level features can be used to perform facial landmark detection and AU recognition, respectively. In addition, a dual learning mechanism is designed to convert the predicted landmarks and AUs of the label space to the corresponding facial image of the image space, further exploring the strong correlations between the tasks. By jointly training the proposed method at both the feature and label levels, each task improves the other. Experiments on two benchmark databases demonstrate that the proposed method can leverage dependencies to boost the generalization of both tasks.

Evaluation of Camera Pose Estimation Using Human Head Pose Estimation

We introduce and evaluate a novel camera pose estimation framework that uses the human head as a calibration object. The proposed method facilitates extrinsic calibration from 2D input images (NIR and/or RGB), while merely relying on the detected human head, without the need for depth information. The approach is applicable to single cameras or multi-camera networks. Our implementation uses a fine-tuned deep learning-based 2D human facial landmark detector to estimate the 3D human head pose by fitting a 3D head model to the detected 2D facial landmarks. Our work focuses on an evaluation of the proposed approach on real multi-camera recordings and synthetic renderings to determine the accuracy of the pose estimation results and their applicability. We assess the robustness of our method against different input parameters, such as varying relative camera positions, variations of head models, face occlusions (by masks, sun glasses, etc.), potential biases and variance among humans. Based on the experimental results, we expect our approach to be effective for numerous use cases including automotive attention monitoring, robotics, VR/AR and other scenarios where ease of handling outweighs accuracy.

3D Video Tracking Technology in the Assessment of Orofacial Impairments in Neurological Disease: Clinical Validation.

This study sought to determine whether clinically interpretable kinematic features extracted automatically from three-dimensional (3D) videos were correlated with corresponding perceptual clinical orofacial ratings in individuals with orofacial impairments due to neurological disorders. 45 participants (19 diagnosed with motor neuron diseases [MNDs] and 26 poststroke) performed two nonspeech tasks (mouth opening and lip spreading) and one speech task (repetition of a sentence "Buy Bobby a Puppy") while being video-recorded in a standardized lab setting. The color video recordings of participants were assessed by an expert clinician-a speech language pathologist-on the severity of three orofacial measures: symmetry, range of motion (ROM), and speed. Clinically interpretable 3D kinematic features, linked to symmetry, ROM, and speed, were automatically extracted from video recordings, using a deep facial landmark detection and tracking algorithm for each of the three tasks. Spearman correlations were used to identify features that were significantly correlated (p value < .05) with their corresponding clinical scores. Clinically significant kinematic features were then used in the subsequent multivariate regression models to predict the overall orofacial impairment severity score. Several kinematic features extracted from 3D video recordings were associated with their corresponding perceptual clinical scores, indicating clinical validity of these automatically derived measures. Different patterns of significant features were observed between MND and poststroke groups; these differences were aligned with clinical expectations in both cases. The results show that kinematic features extracted automatically from simple clinical tasks can capture characteristics used by clinicians during assessments. These findings support the clinical validity of video-based automatic extraction of kinematic features.

Facial Photo-Guided Head Anatomy Modeling Based on Deep Learning and 2D/3D Shape Prior Model Registration

Three-dimensional (3D) models of individualized head anatomy are often used in the preoperative planning of plastic surgery. Most current methods use Computed Tomography (CT) to acquire the head anatomy, but CT scan exposes the patients to X-ray radiation. Alternatively, artificial intelligence-based estimation of 3D head anatomy from the facial photo is a safer and more efficient choice, especially for scenarios that do not require high modeling accuracy of internal structures (e.g., doctor-patient communication and surgery procedure demonstration). We develop a method for constructing a personalized 3D model of complete head anatomy (including skin and internal structures) from the front and side view photos. We first detect key facial landmarks from the photos with a convolution neural network (CNN), then perform 2D/3D registration to morph a statistical shape model (SSM) of complete head anatomy to match the dual-view photos. Our method uses deep learning to achieve accurate facial landmark detection and employ the anatomy shape prior to yielding reasonable internal structure estimation. We evaluated the method based on 12 subjects (including 7 males and 5 females). The facial surface reconstruction error was assessed using the 3D surface scan of the subjects as the ground truth. Our method’s root-mean-square-error (RMSE) was 3.60±0.49 mm, which was 1.06±0.39 mm lower than the state-of-the-art (SOTA) CNN-based face reconstruction method. Our method also predicts all the internal head structures (bones, muscles, vessels, nerve fibers, fat, glands, and brain structures) which are not provided by the SOTA method. To the best of our knowledge, this is the first study modeling complete head anatomy from facial photos toward clinical applications.