Facial Landmark Points Research Articles

86‐3: Binocular Camera Eye Tracking Algorithm for Naked Eye 3D Display

Through real‐time acquisition of eye position using eye tracking, viewers can experience stereoscopic scenes from the naked eye 3D display. This paper proposes binocular camera eye tracking method that utilizes disparity to perceive the actual inter‐pupillary distance of different users. This method estimates the head posture based on facial landmark points and exploits camera projection to determine the spatial coordinate of the pupil. Furthermore, collaborative filtering approach is introduced to mitigate the jitter in pupil position. The experimental results demonstrate that the proposed algorithm enhances the performance in eye tracking compared to monocular method.

Deep neural network-based emotion recognition using facial landmark features and particle swarm optimization

Relating specifically to human–computer interaction (HCI), computer vision research has placed a substantial emphasis on intelligent emotion recognition in recent years. The primary emphasis lies in investigating speech aspects and bodily motions, while the knowledge of recognizing emotions from facial expressions remains relatively unexplored. Automated facial emotion detection allows a machine to assess and understand a person's emotional state, allowing the system to predict intent by analyzing facial expressions. Therefore, this research provides a novel parameter selection strategy using swarm intelligence and a fitness function for intelligent recognition of micro emotions. This paper presents a novel method based on geometric visual representation obtained from facial landmark points. We employ the Deep Neural Networks (DNN) model to analyze the input features from the normalized angle and distance values derived from these landmarks. The results of the experiments show that Particle Swarm Optimization (PSO) worked very well by using only a few carefully chosen features. The method achieved a recognition success rate of 98.76% on the MUG dataset and 97.79% on the GEMEP datasets.

Exploring Convolutional Neural Networks for Facial Expression Recognition: A Comprehensive Survey

Facial emotion recognition is a very sought-after field in recent times as it has many important applications such as effective calculations, video game testing and motion capture in video games, human-computer interaction through machine vision, computer research etc. Facial expression is considered a nonverbal form of communication, as it reveals an individual's internal sentiments and emotional states through changes in multiple facial landmark points. Facial identification provides a more comprehensive insight into the person's thoughts and these expressions are analyzed using deep learning methods, such as CNN. The accuracy rates achieved are compared to other methods. In this paper, a concise exploration of diverse applications within Facial Expression Recognition (FER) fields and the publicly accessible data sets employed in FER studies is outlined. FER using multiple different CNN algorithms is also presented. Finally, through comparing multiple different studies of various CNN algorithms, a table and a chart are provided for a better understanding of the rate of accuracy achieved throughout the use of different datasets.

Designing Effective Visual Feedback for Facial Rehabilitation Exercises: Investigating the Role of Shape, Transparency, and Age on User Experience.

Facial expression recognition technology has been utilized both for entertainment purposes and as a valuable aid in rehabilitation and facial exercise assistance. This technology leverages artificial intelligence models to predict facial landmark points and provide visual feedback, thereby facilitating users' facial movements. However, feedback designs that disregard user preferences may cause discomfort and diminish the benefits of exercise. This study aimed to develop a feedback design guide for facial rehabilitation exercises by investigating user responses to various feedback design methods. We created a facial recognition mobile application and designed six feedback variations based on shape and transparency. To evaluate user experience, we conducted a usability test involving 48 participants (24 subjects in their 20s and 24 over 60 years of age), assessing factors such as feedback, assistance, disturbance, aesthetics, cognitive ease, and appropriateness. The experimental results revealed significant differences in transparency, age, and the interaction between transparency and age. Consequently, it is essential to consider both transparency and user age when designing facial recognition feedback. The findings of this study could potentially inform the design of more effective and personalized visual feedback for facial motion, ultimately benefiting users in rehabilitation and exercise contexts.

Facial Expression Animation by Landmark Guided Residual Module

We study the problem of facial expression animation from a still image according to a driving video. This is a challenging task as expression motions are non-rigid and very subtle to be captured. Existing methods mostly fail to model these subtle expression motions, leading to the lack of details in their animation results. In this paper, we propose a novel facial expression animation method based on generative adversarial learning. To capture the subtle expression motions, Landmark guided Residual Module (LRM) is proposed to model detailed facial expression features. Specifically, residual learning is conducted at both coarse and fine levels conditioned on facial landmark heatmaps and landmark points respectively. Furthermore, we employ a consistency discriminator to ensure the temporal consistency of the generated video sequence. In addition, a novel metric named Emotion Consistency Metric is proposed to evaluate the consistency of facial expressions in the generated sequences with those in the driving videos. Experiments on MUG-Face, Oulu-CASIA and CAER datasets show that the proposed method can generate arbitrary expression motions on the source still image effectively, which are more photo-realistic and consistent with the driving video compared with results of state-of-the-art methods.

Abstract WMP120: Development Of Smartphone Enabled Machine Learning Algorithms For Autonomous Stroke Detection

Background: We developed an automated smart phone application for detection of acute stroke using machine learning (ML) algorithms for recognition of facial asymmetry, arm weakness, and speech changes. Methods: We analysed prospectively collected data from patients admitted to 4 major metropolitan stroke centers with confirmed diagnosis of acute stroke. Speech and facial data were captured via video recording and arm data was captured via device sensors. A. Face. This module extracts 68 facial landmark points that are passed through a dimensionality reduction step and an asymmetry classifier. We implemented and compared 26 classification methods with neurologists' clinical impression and determined Quadratic Discriminative Analysis as the best one in terms of accuracy and interpretability. B. Arm. Using data extracted from 3D accelerometer, gyroscope, and magnetometer , we designed a grasp agnostic classifier based on AdaBoost to process motion trajectories and detect arm weakness.C. Speech. We developed an algorithm based on frequency analysis and Mel Frequency Cepstral Coefficients (MFCC) to detect abnormal/slurred speech. All tests were conducted within 72 hours of admission. Each of the three ML outputs was correlated with neurologists’ clinical impression. Results: Among the 269 analysed patients, 41% were female, the median age was 71, % had hemorrhagic and % had ischemic stroke. Final analyses of 18311 facial images revealed 99.42% sensitivity, 93.67% specificity, and 97.11% accuracy in detection of facial asymmetry. The results for 43 arm trajectories revealed 71.42% sensitivity, 72.41% specificity, and 72.09% accuracy in detection of arm weakness. Preliminary analysis of MFCC algorithms confirmed adequate features for abnormal speech detection Conclusions: Our preliminary results confirm that smartphone enabled ML-algorithms can reliably identify acute stroke features with accuracy comparable to neurologists’ clinical impression.

Image-based Driver Alert System for Prevention of Fatigue-related Accidents

The objective of this project is to design a driver unconsciousness detection system using image processing to detect drowsiness and unconsciousness in drivers, thereby preventing accidents resulting from driver fatigue. Driver fatigue is a serious road safety issue, with approximately 20% of all road accidents attributed to this cause. Conventional drowsiness detection systems rely on physiological monitoring, which can be unreliable, expensive, and challenging to implement and maintain. In contrast, the proposed system monitors a sequence of images to identify facial and behavioral patterns indicative of drowsiness or unconsciousness. By detecting facial landmark points and analyzing the duration of eye closure, the system can accurately classify the driver’s state and take appropriate measures such as reducing the vehicle’s speed and alerting emergency services of the driver’s geo-location. The successful implementation of this system holds immense potential for substantially reducing the number of accidents resulting from driver fatigue, thereby mitigating the loss of lives and injuries.

Evaluation of soft tissue prediction accuracy for orthognathic surgery with skeletal class III malocclusion using maxillofacial regional aesthetic units.

This study aimed to evaluate the soft tissue prediction accuracy of patients undergoing orthognathic surgery to correct skeletal class III malocclusion using maxillofacial regional aesthetic units. Pre- and postoperative cone-beam computed tomography (CBCT) and 3D facial scans were taken for 58 patients who had undergone orthognathic surgery. The preoperative 3D facial scan was integrated with the preoperative CBCT using ProPlan CMF software. The software simulated the surgery and generated postoperative soft tissue prediction. The simulated 3D facial scan was then compared with the actual 3D facial scan obtained at least 6months after the surgery by the maxillofacial regional aesthetic units and the facial soft tissue landmark points. The anatomical regions of the upper lip, lower lip, chin, right external buccal and left external buccal prediction were above 2.0mm. As for the soft tissue landmarks, at chl, chr, ls, stm and li, the position of predicted scan was higher than that of the actual postoperative scan. The accuracy of 3D soft tissue predictions using ProPlan CMF software in Skeletal III patients was clinically satisfactory according to maxillofacial regional aesthetic units combined with facial soft tissue landmark points. However, the accuracy of prediction still needed improvement in some areas. The accuracy of soft tissue prediction can be analyzed more clearly through maxillofacial regional aesthetic units so that clinicians have a deeper understanding of the use of the software to predict soft tissue change after orthognathic surgery.

Uncertain and biased facial expression recognition based on depthwise separable convolutional neural network with embedded attention mechanism

Datasets play a crucial role in the development of facial expression recognition (FER), but most of the datasets suffer from obvious uncertainties and biases caused by different cultures and collection conditions. To look deeper into these issues, this paper first conducts two sets of experiments, face detection, and facial expression (FE) classification. They are based on three datasets (CK+, FER2013, and RAF–DB), which are collected from lab and wild environments. This paper proposes a network, depthwise separable convolutional neural network (CNN) with an embedded attention mechanism (DSA–CNN) for expression recognition. First, at the preprocessing stage, we obtain the maximum expression range clipping, which is calculated from 81 facial landmark points to filter nonface interferences. Then, we use DSA–CNN, which is based on a coordinate squeeze-and-excitation (CSE) attention for feature extraction. Finally, to further deal with imbalanced class biases and uncertainties issues, this paper proposes a class-weighted cross-entropy loss (CCE-loss) to alleviate the imbalance among seven emotional classes. Then, we combine CCE-loss with ranking regularization loss (RR-loss) and self-importance weighting cross-entropy loss (SCE-loss) at label amend stage, to jointly guide the training of the network. Extensive experiments on three FER datasets demonstrate that our proposed method outperforms most of the state-of-the-art methods eventually.

Deteksi Wajah Berbasis Facial Landmark Menggunakan OpenCV Dan Dlib

Computer science and information technology have advanced in a variety of sectors that were previously unachievable due to constraints such as hardware. Computer vision can be used to recognize an object using computer science. Objects can be recognized by taking or recording photos or videos and then processing them using specific tools and methodologies. The goal of the facial landmark-based face detection research using OpenCV and Dlib is to perform face detection in people so that it can be used for a variety of purposes in the future. The strategy employed in this study was the usage of facial landmarks using OpenCV and Dlib to improve face detection accuracy. Face detection has been effectively carried out based on facial landmark points, according to the findings of testing the entire system. Face landmark-based face detection is more accurate using the OpenCV Dlib, which can be seen during processing in the OpenCV Dlib, which can precision photos based on facial movements.

Facial landmark points detection using knowledge distillation-based neural networks

Facial landmark detection is a vital step for numerous facial image analysis applications. Although some deep learning-based methods have achieved good performances in this task, they are often not suitable for running on mobile devices. Such methods rely on networks with many parameters, which makes the training and inference time-consuming. Training lightweight neural networks such as MobileNets are often challenging, and the models might have low accuracy. Inspired by knowledge distillation (KD), this paper presents a novel loss function to train a lightweight Student network (e.g., MobileNetV2) for facial landmark detection. We use two Teacher networks, a Tolerant-Teacher and a Tough-Teacher in conjunction with the Student network. The Tolerant-Teacher is trained using Soft-landmarks created by active shape models, while the Tough-Teacher is trained using the ground truth (aka Hard-landmarks) landmark points. To utilize the facial landmark points predicted by the Teacher networks, we define an Assistive Loss (ALoss) for each Teacher network. Moreover, we define a loss function called KD-Loss that utilizes the facial landmark points predicted by the two pre-trained Teacher networks (EfficientNet-b3) to guide the lightweight Student network towards predicting the Hard-landmarks. Our experimental results on three challenging facial datasets show that the proposed architecture will result in a better-trained Student network that can extract facial landmark points with high accuracy.

3D face reconstruction and dense alignment with a new generated dataset

Face alignment and reconstruction are classical problems in the computer vision field, one of the greatest difficulties of which is the limited number of facial images with landmark points. The 300 W-LP dataset is the most commonly used for the existing methods of single-view 3D Morphable Model (3DMM)-based reconstruction; however, the model performance is limited by the small variety of facial images in this dataset. In this work, a 3D facial image dataset with landmark points generated by the rotate-and-render method is proposed. The key innovation of the proposed method is that the back-and-forth rotation of faces in 3D space and then re-rendering them to the 2D plane can provide strong self-supervision. The recent advances in 3D face modeling and high-resolution generative adversarial networks (GANs) are leveraged to constitute the blocks. To obtain more precise facial landmark points, the 3D dense face alignment (3DDFA) model is used to label the generated images and filter the landmark points. Finally, the 3DDFA model is retrained using the proposed dataset, and an improved result is achieved.

Machine Learning Generated Synthetic Faces for Use in Facial Aesthetic Research.

Importance: A centralized repository of clinically applicable facial images with unrestricted use would facilitate facial aesthetic research. Objective: Using a machine learning neural network, we aim to (1) create a repository of synthetic faces that can be used for facial aesthetic research and (2) analyze synthetic faces according to contemporary aesthetic principles. Design, Setting, and Participants: Synthetic facial images were generated using an open source generative adversarial network. Images were refined and then analyzed using computer vision technology. Interventions: Not applicable. Main Outcomes and Measures: Synthetic facial images were created for use as a facial aesthetic research data set. Results: One thousand synthetic images were generated, and 60 images underwent analysis. Image attributes, including age, gender, image principle axis, facial emotion, and facial landmark points, were attained. Images demonstrated accordance with contemporary aesthetic principles of horizontal thirds and vertical fifths. Images demonstrated excellent correspondence when compared with real human facial photographs. Conclusions and Relevance: We have generated realistic synthetic facial images that have potential as a valuable research tool and demonstrate similarity to real human photographs while adhering to contemporary aesthetic principles.

Low-compute facial expression recognition using fiducial feature-sets

Facial Expression Recognition is an exciting area of affective computing. As mobile and embedded devices become increasingly ubiquitous, exploration of low-compute approaches for facial expression recognition is essential. Facial landmark points are fiducial features that are used to localize and represent salient regions of the face, such as eyes, nose and lips. Any facial expression can be expressed as an activation of facial muscles in specific parts of the face, thereby affecting the locations of the facial landmark points in those parts. This relationship can be captured concretely by deriving appropriate feature-sets from these points. In this paper, an approach for deriving three types of feature-sets from a set of facial landmark points detected on a face is discussed. Such feature-sets are derived for three standard facial expression recognition datasets with labelled expression classes. The derived features-sets are used as inputs for training computationally light machine-learning classifiers, yielding encouraging classification accuracy. These results are presented and discussed with both quantitative and qualitative observations.

An Efficient Velocity Estimation Approach for Face Liveness Detection using Sparse Optical Flow Technique

Objectives: To propose a new liveness detection algorithm using optical flow to ensure the presence of actual live face into a photograph or 2D masks in face recognition biometric security systems. Methods: This work proposes an anti-spoofing model namely Sparse Optical Flow Technique with Velocity Estimation Approach (SOFT_VEA). Optical flow is an effective method for tracking objects in motion. It is adapted in this work to capture facial movements and decide the liveness state. The proposed algorithm considers real faces and two kinds of photo imposters. Findings: From the input video, the motion information of specific facial landmark points is captured by an optical flow algorithm. Then, the velocity of those landmark points is estimated via Euclidean distance. Based on this calculated velocity, the fake face is discriminated from the real face using a threshold value. The Empirical study shows that the proposed face liveness detection model is effective with an accuracy of 88% and Half Total Error Rate (HTER) of 2.45. Novelty: The proposed work is based on real face and photo imposters. The liveness detection algorithm is developed with a novel velocity estimation approach. It is very helpful for biometric security systems. Keywords Biometric security system, Liveness detection, Anti­spoofing, Facial landmarks, Optical flow, Euclidean distance

A spatio-temporal integrated model based on local and global features for video expression recognition

Facial expressions can be represented largely by the dynamic variations of important facial expression parts, i.e., eyebrows, eyes, nose, and mouth. The features of these parts are regarded as local features. However, facial global information is also useful for recognition because it is a necessary complement to local features. In this paper, a spatio-temporal integrated model that jointly learns local and global features is proposed for video expression recognition. Firstly, to capture the action of facial key units, a spatio-temporal attention part-gradient-based hierarchical bidirectional recurrent neural network (spatio-temporal attention PGHRNN) is constructed. It can capture the dynamic variations of gradients around facial landmark points. In addition, a new kind of spatial attention mechanism is introduced to recalibrate the features of facial various parts adaptively. Secondly, to complement the local features extracted by the spatio-temporal attention PGHRNN, a squeeze-and-excitation residual network of 50 layers with long short-term memory network (SE-ResNet-50-LSTM) is used as a global feature extractor and classifier. Finally, to integrate the local and global features and improve the performance of facial expression recognition, a joint adaptive fine-tuning method (JAFTM) is proposed to combine the two networks, which can adaptively adjust the network weights. Extensive experiments demonstrate that our proposed model can achieve a superior recognition accuracy of 98.95% on CK + for 7-class facial expressions and 85.40% on MMI database, which outperforms other state-of-the-art methods.

Facial Expression Recognition Using Hybrid Features of Pixel and Geometry

Facial Expression Recognition (FER) has long been a challenging task in the field of computer vision. Most of the existing FER methods extract facial features on the basis of face pixels, ignoring the relative geometric position dependencies of facial landmark points. This article presents a hybrid feature extraction network to enhance the discriminative power of emotional features. The proposed network consists of a Spatial Attention Convolutional Neural Network (SACNN) and a series of Long Short-term Memory networks with Attention mechanism (ALSTMs). The SACNN is employed to extract the expressional features from static face images and the ALSTMs is designed to explore the potentials of facial landmarks for expression recognition. A deep geometric feature descriptor is proposed to characterize the relative geometric position correlation of facial landmarks. The landmarks are divided into seven groups to extract deep geometric features, and the attention module in ALSTMs can adaptively estimate the importance of different landmark regions. By jointly combining SACNN and ALSTMs, the hybrid features are obtained for expression recognition. Experiments conducted on three public databases, FER2013, CK+, and JAFFE, demonstrate that the proposed method outperforms the previous methods, with the accuracies of 74.31%, 95.15%, and 98.57%, respectively. The preliminary results of Emotion Understanding Robot System (EURS) indicate that the proposed method has the potential to improve the performance of human-robot interaction.

ACN: Occlusion-tolerant face alignment by attentional combination of heterogeneous regression networks

This paper presents the Attentional Combination Network (ACN), which is a highly accurate face alignment method that is tolerant of occlusion. The method combines a coordinate regression network and a heatmap regression network with a spatial attention. The coordinate regression generates the coordinates of facial landmark points directly such that they are fitted to the input face on the whole. The heatmap regression generates the heatmap of facial landmark points such that each channel provides good localization of the detail of its facial landmark point. These independent regressions compensate for each other complementarily such that the overall fitting tendency of the coordinate regression compensates for the inaccurate alignment of the heatmap regression due to missing local information, and the detailed localization of the heatmap regression compensates for the relatively inaccurate alignment of the coordinate regression. The proposed ACN uses coordinate-to-heatmap and the heatmap-to-coordinate conversion networks to combine two heterogeneous regressions, and to generate the final coordinates of the facial landmark points. The ACN use the spatial attention mechanism to effectively reject impeditive local features that are caused by the occlusion. In experiments on several benchmarks, the proposed ACN achieved state-of-the-art accuracy

Comparison of the registration methods for the three-dimensional facial scans applied to the design of full-arch implant supported restoration

OBJECTIVE To compare the registration accuracy of three-dimensional (3D) facial scans for the design of full-arch implant supported restoration by five methods and to explore the suitable registration method. METHODS According to the criteria, ten patients with maxillary edentulous jaw or end-stage dentition requiring implant supported restorations were enrolled in this study. A special rim with individual feature marks reflected appropriate occlusal relationship and esthetic characteristics was made for each patient. Both 3D facial scan data of natural laughter and with opener traction to expose the teeth or occlusal rim of each patient were acquired by facial scan and input to the digital analysis software Geomagic Qualify 2012. The dataset was superimposed by five different methods: seven facial anatomical landmark points alignment, facial immobile area alignment (forehead and nasal area), facial anatomical landmark points and immobile area combining alignment, facial feature points alignment, facial and intraoral feature points alignment with the same local coordinate system. The three-dimensional deviation of the same selected area was calculated, the smaller the deviation, the higher the registration accuracy. The 3D deviation was compared among the three registration methods of facial anatomical landmark points, facial immobile area alignment and the combination of the above two methods. Friedman test was performed to analyze the difference among the three methods (α=0.05). The effect of the aid of the facial and intraoral feature points were evaluated. Paired t test were performed to analyze the difference (P<0.05). RESULTS The average three-dimensional deviation of the selected area after alignment with the facial anatomical landmarks was (1.501 2±0.406 1) mm, significantly larger than that of the facial immobile area best-fit alignment [(0.629 1±0.150 6) mm] and the combination of the two methods[(0.629 1±0.150 6) mm] (P < 0.001). The aid of the facial feature points could significantly reduce the deviation (t=1.001 3, P < 0.001). There was no significant statistical difference in the remaining groups. CONCLUSION The forehead area of the 3D facial scan can be exposed as much as possible. The establishment of facial characteristic landmark points and the use of the invariant area alignment can improve the accuracy of registration. It should be clinically feasible to apply three-dimensional facial scan to the design of full-arch implant supported restoration with the registration of the immobile area on the face especially the forehead area.

FACS-Based Graph Features for Real-Time Micro-Expression Recognition.

Several studies on micro-expression recognition have contributed mainly to accuracy improvement. However, the computational complexity receives lesser attention comparatively and therefore increases the cost of micro-expression recognition for real-time application. In addition, majority of the existing approaches required at least two frames (i.e., onset and apex frames) to compute features of every sample. This paper puts forward new facial graph features based on 68-point landmarks using Facial Action Coding System (FACS). The proposed feature extraction technique (FACS-based graph features) utilizes facial landmark points to compute graph for different Action Units (AUs), where the measured distance and gradient of every segment within an AU graph is presented as feature. Moreover, the proposed technique processes ME recognition based on single input frame sample. Results indicate that the proposed FACS-baed graph features achieve up to 87.33% of recognition accuracy with F1-score of 0.87 using leave one subject out cross-validation on SAMM datasets. Besides, the proposed technique computes features at the speed of 2 ms per sample on Xeon Processor E5-2650 machine.