Facial Landmark Detection Research Articles

Detecting Facial Region and Landmarks at Once via Deep Network

For accurate and fast detection of facial landmarks, we propose a new facial landmark detection method. Previous facial landmark detection models generally perform a face detection step before landmark detection. This greatly affects landmark detection performance depending on which face detection model is used. Therefore, we propose a model that can simultaneously detect a face region and a landmark without performing the face detection step before landmark detection. The proposed single-shot detection model is based on the framework of YOLOv3, a one-stage object detection method, and the loss function and structure are altered to learn faces and landmarks at the same time. In addition, EfficientNet-B0 was utilized as the backbone network to increase processing speed and accuracy. The learned database used 300W-LP with 64 facial landmarks. The average normalized error of the proposed model was 2.32 pixels. The processing time per frame was about 15 milliseconds, and the average precision of face detection was about 99%. As a result of the evaluation, it was confirmed that the single-shot detection model has better performance and speed than the previous methods. In addition, as a result of using the COFW database, which has 29 landmarks instead of 64 to verify the proposed method, the average normalization error was 2.56 pixels, which was also confirmed to show promising performance.

Head tracker using webcam for auralization

Binaural rendering is a technique that seeks to generate virtual auditory environments that replicate the natural listening experience, including the three-dimensional perception of spatialized sound sources. As such, real-time knowledge of the listener's position, or more specifically, their head and ear orientations allow the transfer of movement from the real world to virtual spaces, which consequently enables a richer immersion and interaction with the virtual scene. This study presents the use of a simple laptop integrated camera (webcam) as a head tracker sensor, disregarding the necessity to mount any hardware to the listener's head. The software was built on top of a state-of-the-art face landmark detection model, from Google's MediaPipe library for Python. Manipulations to the coordinate system are performed, in order to translate the origin from the camera to the center of the subject's head and adequately extract rotation matrices and Euler angles. Low-latency communication is enabled via User Datagram Protocol (UDP), allowing the head tracker to run in parallel and asynchronous with the main application. Empirical experiments have demonstrated reasonable accuracy and quick response, indicating suitability to real-time applications that do not necessarily require methodical precision.

A hybrid technique for intelligent bank security system based on blink gesture recognition

Enhancement of bank security system plays an important role at the present time due to increasing attempts to expose it to many infringements by criminals. This paper proposes a hybrid technique of amplitude and duration to recognize its blink gestures to increase the accuracy of the human security system’s interaction for intelligent bank security, as this hybrid technique implemented analyzing of face, facial landmark detection, and the duration of the eye aspect ratio (EAR) for blink processes. The proposed hybrid technique is evaluated by experimenting a number of blink gesture with two eyes obtained from web camera and the results shows better and more effective analysis and provide high security bank system.

Application of Deep Learning on Student Engagement in e-learning environments

The drastic impact of COVID-19 pandemic is visible in all aspects of our lives including education. With a distinctive rise in e-learning, teaching methods are being undertaken remotely on digital platforms due to COVID-19. To reduce the effect of this pandemic on the education sector, most of the educational institutions are already conducting online classes. However, to make these digital learning sessions interactive and comparable to the traditional offline classrooms, it is essential to ensure that students are properly engaged during online classes. In this paper, we have presented novel deep learning based algorithms that monitor the student’s emotions in real-time such as anger, disgust, fear, happiness, sadness, and surprise. This is done by the proposed novel state-of-the-art algorithms which compute the Mean Engagement Score (MES) by analyzing the obtained results from facial landmark detection, emotional recognition and the weights from a survey conducted on students over an hour-long class. The proposed automated approach will certainly help educational institutions in achieving an improved and innovative digital learning method.

AnchorFace: An Anchor-based Facial Landmark Detector Across Large Poses

Facial landmark localization aims to detect the predefined points of human faces, and the topic has been rapidly improved with the recent development of neural network based methods. However, it remains a challenging task when dealing with faces in unconstrained scenarios, especially with large pose variations. In this paper, we target the problem of facial landmark localization across large poses and address this task based on a split-and-aggregate strategy. To split the search space, we propose a set of anchor templates as references for regression, which well addresses the large variations of face poses. Based on the prediction of each anchor template, we propose to aggregate the results, which can reduce the landmark uncertainty due to the large poses. Overall, our proposed approach, named AnchorFace, obtains state-of-the-art results with extremely efficient inference speed on four challenging benchmarks, i.e. AFLW, 300W, Menpo, and WFLW dataset. Code will be available soon.

Sample-Specific Output Constraints for Neural Networks

It is common practice to constrain the output space of a neural network with the final layer to a problem-specific value range. However, for many tasks it is desired to restrict the output space for each input independently to a different subdomain with a non-trivial geometry, e.g. in safety-critical applications, to exclude hazardous outputs sample-wise. We propose ConstraintNet—a scalable neural network architecture which constrains the output space in each forward pass independently. Contrary to prior approaches, which perform a projection in the final layer, ConstraintNet applies an input-dependent parametrization of the constrained output space. Thereby, the complete interior of the constrained region is covered and computational costs are reduced significantly. For constraints in form of convex polytopes, we leverage the vertex representation to specify the parametrization. The second modification consists of adding an auxiliary input in form of a tensor description of the constraint to enable the handling of multiple constraints for the same sample. Finally, ConstraintNet is end-to-end trainable with almost no overhead in the forward and backward pass. We demonstrate ConstraintNet on two regression tasks: First, we modify a CNN and construct several constraints for facial landmark detection tasks. Second, we demonstrate the application to a follow object controller for vehicles and accomplish safe reinforcement learning in this case. In both experiments, ConstraintNet improves performance and we conclude that our approach is promising for applying neural networks in safety-critical environments.

Analysis of surgical outcome after upper eyelid surgery by computer vision algorithm using face and facial landmark detection.

To evaluate the postoperative changes with a computer vision algorithm for anterior full-face photographs of patients who have undergone upper eyelid blepharoplasty surgery with, or without, a Müller's muscle-conjunctival resection (MMCR). All patients who underwent upper eyelid blepharoplasty surgery (Group I), or upper eyelid blepharoplasty with MMCR (Group II) were included. Both preoperative and 6-month postoperative anterior full-face photographs of 55 patients were analyzed. Computer vision and image processing technologies were used to measure the palpebral distance (PD), eye-opening area (EA), and average eyebrow height (AEBH) for both eyes. Preoperative and postoperative measurements were calculated and compared between the two groups. In Group II, change in postoperative Right PD, Left PD, Right EA, Left EA was significantly higher than in Group I (p = 0.004 for REPD; p = 0.001 for LEPD; p = 0.004 for REA; p = 0.002 for LEA, p < 0.05). In Group II, the postoperative change in Right AEBH, Left AEBH was significantly higher than in Group I (p = 0.001 for RABH and LABH, p < 0.05). Eyelid surgery for esthetic purposes requires artistic judgment and objective evaluation. Because of the slight differences in photograph sizes and dynamic factors of the face due to head movements and facial expressions, it is hard to compare and make a truly objective evaluation of the eyelid operations. With a computer vision algorithm, using the face and facial landmark detection system, the photographs are normalized and calibrated. This system offers a simple, standardized, objective, and repeatable method of patient assessment. This can be the first step of Artificial Intelligence algorithm to evaluate the patients who had undergone eyelid operations.

CAT-CAD: A Computer-Aided Diagnosis Tool for Cataplexy

Narcolepsy with cataplexy is a severe lifelong disorder characterized, among others, by sudden loss of bilateral face muscle tone triggered by emotions (cataplexy). A recent approach for the diagnosis of the disease is based on a completely manual analysis of video recordings of patients undergoing emotional stimulation made on-site by medical specialists, looking for specific facial behavior motor phenomena. We present here the CAT-CAD tool for automatic detection of cataplexy symptoms, with the double aim of (1) supporting neurologists in the diagnosis/monitoring of the disease and (2) facilitating the experience of patients, allowing them to conduct video recordings at home. CAT-CAD includes a front-end medical interface (for the playback/inspection of patient recordings and the retrieval of videos relevant to the one currently played) and a back-end AI-based video analyzer (able to automatically detect the presence of disease symptoms in the patient recording). Analysis of patients’ videos for discovering disease symptoms is based on the detection of facial landmarks, and an alternative implementation of the video analyzer, exploiting deep-learning techniques, is introduced. Performance of both approaches is experimentally evaluated using a benchmark of real patients’ recordings, demonstrating the effectiveness of the proposed solutions.

Gaze dynamics with spatiotemporal guided feature descriptor for prediction of driver’s maneuver behavior

At different levels of driving automation, driver’s gaze maintains great indispensable importance on semantic perception of the surround. In this work, we model gaze dynamics and clarify its relationship with driver’s maneuver behaviors from personalized driving style. Firstly, this paper proposes an Occlusion-immune Face Detector (OFD) for facial landmark detection, which can adaptively solve the facial occlusion introduced by the body and glasses frame in the real-world driving scenarios. Meanwhile, an Eye-head Coordination Model is brought up to bridge the error gap in gaze direction through determining eye pose and head pose fused pattern. Then, a vectorized spatiotemporal guidance feature (STGF) descriptor combining gaze accumulation and gaze transition frequency is proposed to construct gaze dynamics within a time window. Finally, we predict driver’s maneuver behavior through STGF descriptor considering different driving styles to clarify the relationship between gaze dynamics and driving maneuver task. Natural driving data are sampled in a dedicated instrumented vehicle testbed, on which 15 drivers with three kind of driving styles participated. Experimental results show that the prediction model achieves the best performance, estimating driver’s behavior an average of 1 s ahead of actual behavior with 83.6% accuracy considering driving style, compared with other approaches.

2D Wasserstein loss for robust facial landmark detection

The recent performance of facial landmark detection has been significantly improved by using deep Convolutional Neural Networks (CNNs), especially the Heatmap Regression Models (HRMs). Although their performance on common benchmark datasets has reached a high level, the robustness of these models still remains a challenging problem in the practical use under noisy conditions of realistic environments. Contrary to most existing work focusing on the design of new models, we argue that improving the robustness requires rethinking many other aspects, including the use of datasets, the format of landmark annotation, the evaluation metric as well as the training and detection algorithm itself. In this paper, we propose a novel method for robust facial landmark detection, using a loss function based on the 2D Wasserstein distance combined with a new landmark coordinate sampling relying on the barycenter of the individual probability distributions. Our method can be plugged-and-play on most state-of-the-art HRMs with neither additional complexity nor structural modifications of the models. Further, with the large performance increase, we found that current evaluation metrics can no longer fully reflect the robustness of these models. Therefore, we propose several improvements to the standard evaluation protocol. Extensive experimental results on both traditional evaluation metrics and our evaluation metrics demonstrate that our approach significantly improves the robustness of state-of-the-art facial landmark detection models.

Detection of facial landmarks by a convolutional neural network in patients with oral and maxillofacial disease

Facial nerve dysfunction is common in patients with Bell’s palsy, trauma, tumour, or iatrogenic injuries. Imaging assessment is the most convenient method for patients and their treating physician. With developments in artificial intelligence (AI), manual work will be replaced. In this study, a database of facial images of patients with oral and maxillofacial diseases was set up to develop a facial nerve functional assessment system based on AI. This database was then used to evaluate the accuracy of a state-of-the-art algorithm for facial landmark detection named ‘HRNet’. Utilizing this database and with appropriate human intervention, HRNet was used in facial annotation. The accuracy of annotations was evaluated through the normalized mean error. A total of 912 images were collected from 300 people; 546 of these images had abnormal features including defects, swelling, scars, or facial paralysis. The accuracy for the abnormal group was lower than that for the normal group before and after training, but improvements in accuracy were identified in both groups post-training. In conclusion, this new database demonstrates the ability of HRNet to localize facial landmarks in patients with oral and maxillofacial diseases. More images for training should be added to this database to diversify it in the future.

Victim identification application with face recognition based on android

The increase in the population causes the problems faced are also increasing. Problems that arise include that, the number of accident victims who have recently often occurred on the highway, where the victims of the accident suffered minor injuries to severe and some even have to take lives due to accidents on the highway. And the second is crime, like a murder case that can take someone’s life. Often victims do not carry identities, making it difficult for the police or related parties to identify victims of accidents and killings. This system is made in the hope that it can help the police or related parties to be able to recognize the face of the victim without an identity just by using the android application. So that it can save more time in identifying the identity of the victim. Victim face recognition algorithm using Facial Landmark Detector and for its implementation using the Dlib library. To do facial recognition, it is done by taking Figures of victims using an application on android which is then sent to the server for the recognition process. After the image is sent to the server, the recognition process will be carried out, the results of the introduction process are sent back to android to display the results. The level of accuracy of the victim’s face recognition with a wound if using a tolerance of 0.35 is 100%, but if using tolerance 0.2 the level of accuracy is only 25%.

Implement of the transformation for markerless facial motion capture in 3D animation

Motion capture is a motion assisting technology that can resemble the motion of objects being captured and is now widely used for film development, especially in visual effects or animation. From that, the author has the idea, namely to use a Blender application to implement a markerless facial motion capture on motion of object 3D modeling based on webcam because it didn’t have a menu or feature for markerless motion capture. Markerless is meant to process the assignment of annotation points to objects with a computer system so that this method is very simple and saves time in its operation, especially in facial motion capture. In this research, the face detection method uses haar cascade and to provide facial landmark annotation using detector landmark lbf model. We are using an example of 3D modeling a human face that has a bone on the neck, head, mouth, eyes, and eyebrows that will follow the movement of the human face. The combination of face landmark detection and rigging in 3D modeling a human face uses the transformation geometry method to produce real-time motion. From this research study, 3D animated face objects can follow motion on real face targets but there is still noise.

Facial imaging and landmark detection technique for objective assessment of unilateral peripheral facial paralysis

ABSTRACT In this paper, we propose a hypothesis that the facial landmark detection methods constructed by a private UPFP facial dataset can perform better than the model on a healthy facial dataset in the task of UPFP facial landmark detection. For proving this hypothesis, a customized UPFP facial dataset with 68 facial landmark annotations was built. A state-of-the-art facial landmark detection method was employed on the three evaluation datasets to exploit and prove the hypothesis. The mean error of validation dataset is 3.15, 56% lower than 7.42 that of the healthy dataset, which proves the hypothesis is true.

Visible-to-Thermal Transfer Learning for Facial Landmark Detection

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.

Design of a Facial Landmark Detection System Using a Dynamic Optical Flow Approach

Many facial landmark methods based on convolutional neural networks (CNN) have been proposed to achieve favorable detection results. However, the instability landmarks that occur in video frames due to CNNs are extremely sensitive to input image noise. To solve this problem of landmark shaking, this study proposes a simple and effective facial landmark detection method comprising a lightweight U-Net model and a dynamic optical flow (DOF). The DOF uses the fast optical flow to obtain the optical flow vector of the landmark and uses dynamic routing to improve landmark stabilization. A lightweight U-Net model is designed to predict facial landmarks with a smaller model size and less computational complexity. The predicted facial landmarks are further fed to the DOF approach to deal with the unstable shaking. Finally, a comparison of several common methods and the proposed detection method is made on several benchmark datasets. Experimental evaluations and analyses show that not only can the lightweight U-Net model achieve favorable landmark prediction but also the DOF stabilizing method can improve the robustness of landmark prediction in both static images and video frames. It should be emphasized that the proposed detection system exhibits better performance than others without requiring heavy computational loadings.

Robust Real-time Head Pose Estimation for 10 Watt SBC

Head Pose Estimation has always been an essential part for many applications such as autonomous driving and driv-ing assist systems and hence performance optimization provides better performance as well as lower computing and power needs that allows us to run such applications over embedded devices inside these systems. In this article we present an implementation over a Single board computer for a new system of 3D Head pose estimation that estimates the Head pose of a person in real-time for applications such as Driver monitoring systems, Drones, Gesture recognition and tracking devices. The system is developed over a single board computer (SBC) that is suitable for very low powered applications, it only utilizes the data provided through the IR camera sensor to estimate both the Head and camera pose without any need for external sensors. This system will combine methods that include traditional image processing techniques for image projection, feature detection, key point description and 3D pose estimation along with Machine Learning techniques for face detection and facial landmarks detection.

Real-Time Eye Tracking for Bare and Sunglasses-Wearing Faces for Augmented Reality 3D Head-Up Displays

Eye pupil tracking is important for augmented reality (AR) three-dimensional (3D) head-up displays (HUDs). Accurate and fast eye tracking is still challenging due to multiple driving conditions with eye occlusions, such as wearing sunglasses. In this paper, we propose a system for commercial use that can handle practical driving conditions. Our system classifies human faces into bare faces and sunglasses faces, which are treated differently. For bare faces, our eye tracker regresses the pupil area in a coarse-to-fine manner based on a revised Supervised Descent Method based eye-nose alignment. For sunglasses faces, because the eyes are occluded, our eye tracker uses whole face alignment with a revised Practical Facial Landmark Detector for pupil center tracking. Furthermore, we propose a structural inference-based re-weight network to predict eye position from non-occluded areas, such as the nose and mouth. The proposed re-weight sub-network revises the importance of different feature map positions and predicts the occluded eye positions by non-occluded parts. The proposed eye tracker is robust via a tracker-checker and a small model size. Experiments show that our method achieves high accuracy and speed, approximately 1.5 and 6.5 mm error for bare and sunglasses faces, respectively, at less than 10 ms on a 2.0GHz CPU. The evaluation dataset was captured indoors and outdoors to reflect multiple sunlight conditions. Our proposed method, combined with AR 3D HUDs, shows promising results for commercialization with low crosstalk 3D images.

Attention Span Prediction Using Head-Pose Estimation With Deep Neural Networks

Automated human pose estimation is evolving as an exciting research area in human activity detection. It includes sophisticated applications such as malpractice detection in the examination, distracted driving, gesture detection, etc., and requires robust and reliable pose estimation techniques. These applications help to map the attention of the user with head pose estimation (HPE) metrics supported by emotion and gaze analysis. This paper solves the problem of attention score estimation with HPE. The proposed method ensures ease of implementation while addressing head pose estimation using 68 facial features. Further, to attain reliability and precision, head pose estimation has been implemented as a regression task. The coordinate pair angle method (CPAM) with deep neural network (DNN) regression and elastic net regression is carried out. The use of DNN ensures precision on low lighting, distorted or occluded images. CPAM methodology leverages facial landmark detection and angular difference to estimate head pose. Experimentation results showed that the proposed model could handle large datasets, real-time data processing, significant pose variations, partial occlusions, and diverse facial expressions with a mean absolute error (MAE) of 3° and less. The proposed system was evaluated on three standard databases: the 300W across large poses (300W-LP) dataset, annotated facial landmarks in the wild (AFLW2000) dataset, and the national institute of mental health child emotional faces picture set (NIMH - ChEFS ) dataset. The results achieved are on par with recent state-of-the-art methodologies such as anisotropic angle distribution learning (AADL), joint head pose estimation and face alignment algorithm (JFA), rotation axis focused attention network (RAFA-Net), and propose an MAE ranging up to 6°. The paper could achieve remarkable results for attention span prediction using head pose estimation and for many possible future applications.

Filter selection for image processing before the landmark detection stage for micro-expression analysis

The automated estimation of the psycho-emotional state of the human and their emotional reactions to the different influences using the video image analysis is an urgent task in different fields, such as: safeguarding in manufacturing, aviation, transportation, prevention of the crimes and terroristic threats, marketing researches etc. A promising direction is the facial micro-expression analysis. The facial micro-expressions are not under conscious control and reflect the objective emotional reaction. One of the key stages of the procedure of the automatic emotion estimation by the facial micro-expressions is the correct facial landmark detection. It is a complex task because of the presence of the different noise in the consecutive frames. Purpose – the investigation of the ways of increasing the performance of the facial micro-expression analysis pipeline by using preliminary video image processing procedures. It is shown that, as the preliminary stage of the micro-expression analysis pipeline, it is reasonable to perform the blurring of the original images to obtain the more stable results. The determined filtering parameters provide the MediaPipe framework a performance increase for the micro-expression analysis problems. It is shown that the video image blurring by the Gaussian filter with a size of 15×15 pixels allows to reduce the noise influence and to decrease the incorrect shifts of the facial landmarks from frame to frame induced by this noise. The proposed procedure of preliminary video image processing can be used for increasing the facial micro-expression recognition performance in emotion recognition systems based on the video sequence analysis.