Facial Landmark Detection Research Articles

Transcultural language, native Chilean peoples and a new AI-based artistic-cultural expression

This work aims to rescue, transcribe and create new artistic and cultural expressions through the use of native peoples’ historical visual recordings, integrating intelligent technologies. For this purpose, a Chilean native peoples’ digital repository is collected, in order to apply a Digital Humanities-based methodology. From the chosen material, portraits are selected, recoloured through a AI-based model; the facial mesh is constructed using a facial landmark detector; the points of the mesh are reconstructed by a Delaunay triangulation; to finally apply an additive manufacturing process. Thus, these physical pieces allow to compare the native people’s own physiognomies, creating new cultural expressions.

A Facial Landmark Detection Method Based on Deep Knowledge Transfer.

Facial landmark detection is a crucial preprocessing step in many applications that process facial images. Deep-learning-based methods have become mainstream and achieved outstanding performance in facial landmark detection. However, accurate models typically have a large number of parameters, which results in high computational complexity and execution time. A simple but effective facial landmark detection model that achieves a balance between accuracy and speed is crucial. To achieve this, a lightweight, efficient, and effective model is proposed called the efficient face alignment network (EfficientFAN) in this article. EfficientFAN adopts the encoder-decoder structure, with a simple backbone EfficientNet-B0 as the encoder and three upsampling layers and convolutional layers as the decoder. Moreover, deep dark knowledge is extracted through feature-aligned distillation and patch similarity distillation on the teacher network, which contains pixel distribution information in the feature space and multiscale structural information in the affinity space of feature maps. The accuracy of EfficientFAN is further improved after it absorbs dark knowledge. Extensive experimental results on public datasets, including 300 Faces in the Wild (300W), Wider Facial Landmarks in the Wild (WFLW), and Caltech Occluded Faces in the Wild (COFW), demonstrate the superiority of EfficientFAN over state-of-the-art methods.

EYE ASPECT RATIO ADJUSTMENT DETECTION FOR STRONG BLINKING SLEEPINESS BASED ON FACIAL LANDMARKS WITH EYE-BLINK DATASET

<span lang="id">Blink detection is an important technique in a variety of settings, including facial motion analysis and signal processing. However, automatic blink detection is challenging due to its blink rate. This paper proposes a real-time method for detecting eye blinks in a video series. The method is based on automatic facial landmark detection trained on real-world datasets and demonstrates robustness against various environmental factors, including lighting conditions, facial emotions, and head position. The proposed algorithm calculates the position of facial landmarks, extracts scalar values using the Eye Aspect Ratio (EAR), and characterises eye proximity in each frame. For each video frame, the proposed method calculates the location of the facial landmark and extracts the vertical distance between the eyelids using the position of the facial landmark. Blinks are detected by using the EAR threshold value and recognising the pattern of EAR values in a short temporal window. According to the results from a common data set, it is shown that the proposed approach is more efficient than state-of-the-art techniques.</span>

SD-HRNet: Slimming and Distilling High-Resolution Network for Efficient Face Alignment.

Face alignment is widely used in high-level face analysis applications, such as human activity recognition and human-computer interaction. However, most existing models involve a large number of parameters and are computationally inefficient in practical applications. In this paper, we aim to build a lightweight facial landmark detector by proposing a network-level architecture-slimming method. Concretely, we introduce a selective feature fusion mechanism to quantify and prune redundant transformation and aggregation operations in a high-resolution supernetwork. Moreover, we develop a triple knowledge distillation scheme to further refine a slimmed network, where two peer student networks could learn the implicit landmark distributions from each other while absorbing the knowledge from a teacher network. Extensive experiments on challenging benchmarks, including 300W, COFW, and WFLW, demonstrate that our approach achieves competitive performance with a better trade-off between the number of parameters (0.98 M-1.32 M) and the number of floating-point operations (0.59 G-0.6 G) when compared to recent state-of-the-art methods.

Masked Face Emotion Recognition Based on Facial Landmarks and Deep Learning Approaches for Visually Impaired People.

Current artificial intelligence systems for determining a person's emotions rely heavily on lip and mouth movement and other facial features such as eyebrows, eyes, and the forehead. Furthermore, low-light images are typically classified incorrectly because of the dark region around the eyes and eyebrows. In this work, we propose a facial emotion recognition method for masked facial images using low-light image enhancement and feature analysis of the upper features of the face with a convolutional neural network. The proposed approach employs the AffectNet image dataset, which includes eight types of facial expressions and 420,299 images. Initially, the facial input image's lower parts are covered behind a synthetic mask. Boundary and regional representation methods are used to indicate the head and upper features of the face. Secondly, we effectively adopt a facial landmark detection method-based feature extraction strategy using the partially covered masked face's features. Finally, the features, the coordinates of the landmarks that have been identified, and the histograms of the oriented gradients are then incorporated into the classification procedure using a convolutional neural network. An experimental evaluation shows that the proposed method surpasses others by achieving an accuracy of 69.3% on the AffectNet dataset.

Automatic part segmentation of facial anatomies using geometric deep learning toward a computer-aided facial rehabilitation

Detection, identification, and segmentation of facial landmarks and anatomies play an essential role in the automatic reconstruction of patient specific model of the human head for facial diagnosis, monitoring, and rehabilitation. The objective of the present study was to apply geometric deep learning to perform part segmentation on the human face to automatically segment facial anatomies from a 3D point set.A database of Computed Tomography images of 333 subjects was reconstructed. Labels of facial anatomies (eyes, nose, and mouth) were manually performed. Two state-of-the-art geometric deep learning models (PointNet++ and PointCNN) were implemented and evaluated. Then, the best model was applied to perform part segmentation on new Kinect-driven face data of healthy subjects and facial palsy patients. Accuracy and Intersection over Union (IoU) were used as evaluation metrics.An accuracy level of 99.19% and an IoU of 89.09% are obtained for the CT database using the PointNet++ model. Regarding the use of the PointCNN model, an accuracy level of 98.43 and an IoU of 78.33 were obtained. An accuracy range of [81.45%–92.09%] and [81.05%–84.08%] was obtained by using PointNet++ model on Kinect data for healthy subjects and facial palsy patients respectively.This study suggested that geometric deep learning can be used for automatic segmentation of facial anatomies from a 3D data set. The obtained outcomes confirmed the accuracy of PointNet++ and PointCNN architectures. As perspectives, the proposed method will be implemented into an available computer vision system for facial monitoring and rehabilitation.

The Temporal Alignment of Speech-Accompanying Eyebrow Movement and Voice Pitch: A Study Based on Late Night Show Interviews.

Previous research has shown that eyebrow movement during speech exhibits a systematic relationship with intonation: brow raises tend to be aligned with pitch accents, typically preceding them. The present study approaches the question of temporal alignment between brow movement and intonation from a new angle. The study makes use of footage from the Late Night Show with David Letterman, processed with 3D facial landmark detection. Pitch is modeled as a sinusoidal function whose parameters are correlated with the maximum height of the eyebrows in a brow raise. The results confirm some previous findings on audiovisual prosody but lead to new insights as well. First, the shape of the pitch signal in a region of approx. 630 ms before the brow raise is not random and tends to display a specific shape. Second, while being less informative than the post-peak pitch, the pitch signal in the pre-peak region also exhibits correlations with the magnitude of the associated brow raises. Both of these results point to early preparatory action in the speech signal, calling into question the visual-precedes-acoustic assumption. The results are interpreted as supporting a unified view of gesture/speech co-production that regards both signals as manifestations of a single communicative act.

Detecting Facial Landmarks on 3D Models Based on Geometric Properties—A Review of Algorithms, Enhancements, Additions and Open-Source Implementations

Facial landmark detection, a crucial aspect of face recognition, is widely used in various fields, such as facial surgeries, biometrics, and surveillance systems. With the advancement of affordable and capable 3D scanning technologies, research on automatically detecting facial landmarks on 3D models is gaining momentum. Utilizing the geometric properties of 3D facial models, researchers have developed algorithms for various landmarks with varying levels of accuracy. In this study, we reviewed existing literature and developed algorithms for thirty-eight landmarks using geometric properties and statistical information about facial measurements. The algorithms for thirty landmarks are original contributions to the literature. We provide the implementation of all the algorithms as open-source Python code, along with the pseudocode for both our algorithms and those found in the literature. To the best of our knowledge, this study covers the largest number of facial landmark detection algorithms based on the geometric properties of 3D models. This is the first study that provides the implementation of the algorithms along with detailed pseudocode. The results of the algorithms are presented by calculating the mean, median, standard deviation, minimum, and maximum of the errors and depicting the histogram for each landmark over a hundred 3D facial scans. The results show that geometric properties and statistics can be utilized to achieve more robust solutions for facial landmark detection.

Precise Facial Landmark Detection by Reference Heatmap Transformer.

Most facial landmark detection methods predict landmarks by mapping the input facial appearance features to landmark heatmaps and have achieved promising results. However, when the face image is suffering from large poses, heavy occlusions and complicated illuminations, they cannot learn discriminative feature representations and effective facial shape constraints, nor can they accurately predict the value of each element in the landmark heatmap, limiting their detection accuracy. To address this problem, we propose a novel Reference Heatmap Transformer (RHT) by introducing reference heatmap information for more precise facial landmark detection. The proposed RHT consists of a Soft Transformation Module (STM) and a Hard Transformation Module (HTM), which can cooperate with each other to encourage the accurate transformation of the reference heatmap information and facial shape constraints. Then, a Multi-Scale Feature Fusion Module (MSFFM) is proposed to fuse the transformed heatmap features and the semantic features learned from the original face images to enhance feature representations for producing more accurate target heatmaps. To the best of our knowledge, this is the first study to explore how to enhance facial landmark detection by transforming the reference heatmap information. The experimental results from challenging benchmark datasets demonstrate that our proposed method outperforms the state-of-the-art methods in the literature.

HybridMatch: Semi-Supervised Facial Landmark Detection via Hybrid Heatmap Representations

Facial landmark detection is an essential task in face-processing techniques. Traditional methods however require expensive pixel-level labels. Semi-supervised facial landmark detection has been explored as an alternative but previous approaches only focus on training-oriented issues (e.g., noisy pseudo-labels in the semi-supervised learning), neglecting task-oriented issues (i.e., the quantization error in the landmark detection). We argue that semi-supervised landmark detectors should resolve the two technical issues simultaneously. Through a simple experiment, we found that task- and training-oriented solutions may negatively influence each other, thus eliminating their negative interactions is important. To this end, we devise a new heatmap regression framework via hybrid representation, namely HybridMatch.We utilize both 1-D and 2-D heatmap representations. Here, the 1-D and 2-D heatmap help alleviate the task-oriented and the training-oriented issues, respectively. To exploit the advantages of our hybrid representation, we introduce curriculum learning; relying more on the 2-D heatmap at the early training stage and gradually increasing the effects of the 1-D heatmap. By resolving the two issues simultaneously, we can capture more precise landmark points than existing methods with only a few annotated data. Extensive experiments show that HybridMatch achieves state-of-the-art performance on three benchmark datasets, especially showing 26.3% NME improvement over the existing method in the 300-W <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">full</i> set at 5% data ratio. Surprisingly, our method records a comparable performance, 5.04 ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">challenging</i> set in the 300-W) to the fully-supervised facial landmark detector 5.03. The remarkable performance of HybridMatch shows its potential as a practical alternative to the fully-supervised model.

Cascade Learning for Driver Facial Monitoring

As a non-invasive method, vision-based driver monitoring aims to identify risky maneuvers for intelligent vehicles and it has gained an increasing interest over recent years. However, most existing methods tend to design models for specific tasks, such as head pose or gaze estimation, which results in redundant models hampering real time applications. Besides, most driver facial monitoring methods ignore the correlation of different tasks. In this work, we propose a unified framework based on cascade learning for simultaneous facial landmark detection and head pose estimation, as well as simultaneous eye center detection and gaze estimation. In particular, built upon the key idea that facial landmark locations and 3D face model parameters are implicitly correlated, we introduce a cascade regression framework to achieve these two tasks simultaneously. After coarsely extracting the driver’s eye region from the detected facial landmarks, we perform cascade regression for simultaneous eye center detection and gaze estimation. Leveraging the power of cascade learning allows our method to alternatively optimize facial landmark detection, head pose estimation, eye center localization, and gaze prediction. The comparison experiments conducted on benchmark datasets of 300-W, GI4E, BU, MPIIGaze, and driving dataset of SHRP2 demonstrate that our proposed method can achieve state-of-the-art performance with robust effectiveness on the real driver monitoring applications.

Drowsiness Detection System in Real Time Based on Behavioral Characteristics of Driver using Machine Learning Approach

The process of determining if a person, generally a driver, is becoming sleepy or drowsy while performing a task such as driving is known as drowsiness detection. It is a necessary system for detecting and alerting drivers to their tiredness, which might impair their driving ability and lead to accidents. The project aims to create a reliable and efficient system capable of real-time detection of drowsiness using OpenCV, Dlib, and facial landmark detection technologies. The project's results show that the sleepiness detection method can accurately and precisely identify tiredness in real time. The technology is less intrusive and more economical than conventional sleepiness detection techniques. The system is based on a 68 facial landmark detector, which is a highly trained and effective detector capable of recognizing human face points. The detector aids in assessing whether the driver's eyes are closed or open. The system analyses the data collected by the detector using machine learning methods to discover patterns associated with drowsiness. When drowsiness is detected, the system incorporates a warning mechanism, such as an alarm or a vibration in the steering wheel, to notify the driver. A variety of studies with different drivers and driving conditions were used to evaluate the performance of the real-time driver drowsiness detection system. The results show that the technology can detect tiredness properly and deliver timely warnings to the driver. This method can assist in preventing drowsy driving incidents, enhancing road safety, and saving lives. The results indicated that the algorithm had an average accuracy rate of 94% for identifying tiredness in drivers.

Convolutional Neural Network Model based Students’ Engagement Detection in Imbalanced DAiSEE Dataset

The COVID-19 pandemic has significantly changed learning processes. Learning, which had generally been carried out face-to-face, has now turned online. This learning strategy has both advantages and challenges. On the bright side, online learning is unbound by space and time, allowing it to take place anywhere and anytime. On the other side, it faces a common challenge in the lack of direct interaction between educators and students, making it difficult to assess students’ engagement during an online learning process. Therefore, it is necessary to conduct research with the aim of automatically detecting students’ engagement during online learning. The data used in this research were derived from the DAiSEE dataset (Dataset for Affective States in E-Environments), which comprises ten-second video recordings of students. This dataset classifies engagement levels into four categories: low, very low, high, and very high. However, the issue of imbalanced data found in the DAiSEE dataset has yet to be addressed in previous research. This data imbalance can cause errors in the classification model, resulting in overfitting and underfitting of the model. In this study, Convolutional Neural Network, a deep learning model, was utilized for feature extraction on the DAiSEE dataset. The OpenFace library was used to perform facial landmark detection, head pose estimation, facial expression unit recognition, and eye gaze estimation. The pre-processing stages included data selection, dimensional reduction, and normalization. The PCA and SVD techniques were used for dimensional reduction. The data were later oversampled using the SMOTE algorithm. The training and testing data were distributed at an 80:20 ratio. The results obtained from this experiment exceeded the benchmark evaluation values on the DAiSEE dataset, achieving the best accuracy of 77.97% using the SVD dimensional reduction technique.

Development of Algorithm for Person Re-Identification Using Extended Openface Method

Deep learning has risen in popularity as a face recognition technology in recent years. Facenet, a deep convolutional neural network (DCNN) developed by Google, recognizes faces with 128 bytes per face. It also claims to have achieved 99.96% on the reputed Labelled Faces in the Wild (LFW) dataset. However, the accuracy and validation rate of Facenet drops down eventually, there is a gradual decrease in the resolution of the images. This research paper aims at developing a new facial recognition system that can produce a higher accuracy rate and validation rate on low-resolution face images. The proposed system Extended Openface performs facial recognition by using three different features i) facial landmark ii) head pose iii) eye gaze. It extracts facial landmark detection using Scattered Gated Expert Network Constrained Local Model (SGEN-CLM). It also detects the head pose and eye gaze using Enhanced Constrained Local Neural field (ECLNF). Extended openface employs a simple Support Vector Machine (SVM) for training and testing the face images. The system’s performance is assessed on low-resolution datasets like LFW, Indian Movie Face Database (IMFDB). The results demonstrated that Extended Openface has a better accuracy rate (12%) and validation rate (22%) than Facenet on low-resolution images.

MACHINE LEARNING DETEKSI KANTUK PENGEMUDI MENGGUNAKAN FACIAL NERVE GRADING SYSTEM 2.0 (FNGS2.0) PADA FLATFORM THINKSPEAK

Traffic accidents are one of the negative impacts of technological advances in the field of transportation. The increase in traffic accidents is caused by several factors where the highest factor causing traffic accidents is human/HR. Driver fatigue is a significant factor in a large number of traffic accidents, therefore, to minimize it, an alarm system is needed to detect sleepiness in real time.&#x0D; In this study, a sleepy eye detection system will be made based on Facial Landmarks Detection which is the development of the Facial Nerve Grading System 2.0 (FNGS2.0) using the Eye Aspect Ratio (EAR) and Mouth viewpoint proportion (MAR) methods with the Thingspeak platform which is implemented into Orange Pi PC. The input of this system is video recorded from the webcam in real time. The output of this system uses the speaker as an alarm to give a warning that the driver is detected as sleepy.&#x0D; The system made in this study shows that it can detect sleepy eyes well. The test results show that the system can detect 93.3% of the eyes, 96.7% of eye blinks, and 95% of the mouth.

Fatigue Driving Recognition Method Based on Multi-Scale Facial Landmark Detector

Fatigue driving behavior recognition in all-weather real driving environments is a challenging task. Accurate recognition of fatigue driving behavior is helpful to improve traffic safety. The facial landmark detector is crucial to fatigue driving recognition. However, existing facial landmark detectors are mainly aimed at stable front face color images instead of side face gray images, which is difficult to adapt to the fatigue driving behavior recognition in real dynamic scenes. To maximize the driver’s facial feature information and temporal characteristics, a fatigue driving behavior recognition method based on a multi-scale facial landmark detector (MSFLD) is proposed. First, a spatial pyramid pooling and multi-scale feature output (SPP-MSFO) detection model is built to obtain a face region image. The MSFLD is a lightweight facial landmark detector, which is composed of convolution layers, inverted bottleneck blocks, and multi-scale full connection layers to achieve accurate detection of 23 key points on the face. Second, the aspect ratios of the left eye, right eye and mouth are calculated in accordance with the coordinates of the key points to form a fatigue parameter matrix. Finally, the combination of adaptive threshold and statistical threshold is used to avoid misjudgment of fatigue driving recognition. The adaptive threshold is dynamic, which solves the problem of the difference in the aspect ratio of the eyes and mouths of different drivers. The statistical threshold is a supplement to solve the problem of driver’s low eye threshold and high mouth threshold. The proposed methods are evaluated on the Hunan University Fatigue Detection (HNUFDD) dataset. The proposed MSFLD achieves a normalized mean error value of 5.4518%, and the accuracy of the fatigue driving recognition method based on MSFLD achieves 99.1329%, which outperforms that of state-of-the-art methods.

Two-Stream Spatial-Temporal Graph Convolutional Networks for Driver Drowsiness Detection.

Convolutional neural networks (CNNs) have achieved remarkable performance in driver drowsiness detection based on the extraction of deep features of drivers' faces. However, the performance of driver drowsiness detection methods decreases sharply when complications, such as illumination changes in the cab, occlusions and shadows on the driver's face, and variations in the driver's head pose, occur. In addition, current driver drowsiness detection methods are not capable of distinguishing between driver states, such as talking versus yawning or blinking versus closing eyes. Therefore, technical challenges remain in driver drowsiness detection. In this article, we propose a novel and robust two-stream spatial-temporal graph convolutional network (2s-STGCN) for driver drowsiness detection to solve the above-mentioned challenges. To take advantage of the spatial and temporal features of the input data, we use a facial landmark detection method to extract the driver's facial landmarks from real-time videos and then obtain the driver drowsiness detection result by 2s-STGCN. Unlike existing methods, our proposed method uses videos rather than consecutive video frames as processing units. This is the first effort to exploit these processing units in the field of driver drowsiness detection. Moreover, the two-stream framework not only models both the spatial and temporal features but also models both the first-order and second-order information simultaneously, thereby notably improving driver drowsiness detection. Extensive experiments have been performed on the yawn detection dataset (YawDD) and the National TsingHua University drowsy driver detection (NTHU-DDD) dataset. The experimental results validate the feasibility of the proposed method. This method achieves an average accuracy of 93.4% on the YawDD dataset and an average accuracy of 92.7% on the evaluation set of the NTHU-DDD dataset.

Low-Cost Human–Machine Interface for Computer Control with Facial Landmark Detection and Voice Commands

Nowadays, daily life involves the extensive use of computers, since human beings are immersed in a technological society. Therefore, it is mandatory to interact with computers, which represents a true disadvantage for people with upper limb disabilities. In this context, this work aims to develop an interface for emulating mouse and keyboard functions (EMKEY) by applying concepts of artificial vision and voice recognition to replace the use of hands. Pointer control is achieved by head movement, whereas voice recognition is used to perform interface functionalities, including speech-to-text transcription. To evaluate the interface's usability and usefulness, two studies were carried out. The first study was performed with 30 participants without physical disabilities. Throughout this study, there were significant correlations found between the emulator's usability and aspects such as adaptability, execution time, and the participant's age. In the second study, the use of the emulator was analyzed by four participants with motor disabilities. It was found that the interface was best used by the participant with cerebral palsy, followed by the participants with upper limb paralysis, spina bifida, and muscular dystrophy. In general, the results show that the proposed interface is easy to use, practical, fairly accurate, and works on a wide range of computers.

Stacked attention hourglass network based robust facial landmark detection

Deep learning based facial landmark detection (FLD) has made rapid progress. However, the accuracy and robustness of FLD algorithms are degraded heavily when the face is subject to diverse expressions, posture deflection, partial occlusion and other uncertain circumstances. To learn more discriminative representations and reduce the negative effect caused by outliers, a stacked attention hourglass network (SAHN) is proposed for FLD, where new attention mechanism is introduced. Basically, in the design of SAHN, a spatial attention residual (SAR) unit is constructed such that relevant areas of facial landmarks are specially emphasized and essential features of different scales can be well extracted, and a channel attention branch (CAB) is introduced to better guide the next-level hourglass network for feature extraction. Due to the introduction of SAR and CAB, only two hourglass networks are stacked as the proposed SAHN with fewer parameters, which is different from traditional SHNs stacked by four hourglass networks. Furthermore, a variable robustness (VR) loss function is introduced for the training of SAHN. The robustness of the proposed model for FLD is guaranteed with the help of the VR loss by adaptively adjusting a continuous parameter. Extensive experimental results on three public datasets including 300W, WFLW and COFW confirm that our method is superior to some previous ones.

A Two-Stage Deep Generative Model for Masked Face Synthesis.

Research on face recognition with masked faces has been increasingly important due to the prolonged COVID-19 pandemic. To make face recognition practical and robust, a large amount of face image data should be acquired for training purposes. However, it is difficult to obtain masked face images for each human subject. To cope with this difficulty, this paper proposes a simple yet practical method to synthesize a realistic masked face for an unseen face image. For this, a cascade of two convolutional auto-encoders (CAEs) has been designed. The former CAE generates a pose-alike face wearing a mask pattern, which is expected to fit the input face in terms of pose view. The output of the former CAE is readily fed into the secondary CAE for extracting a segmentation map that localizes the mask region on the face. Using the segmentation map, the mask pattern can be successfully fused with the input face by means of simple image processing techniques. The proposed method relies on face appearance reconstruction without any facial landmark detection or localization techniques. Extensive experiments with the GTAV Face database and Labeled Faces in the Wild (LFW) database show that the two complementary generators could rapidly and accurately produce synthetic faces even for challenging input faces (e.g., low-resolution face of 25 × 25 pixels with out-of-plane rotations).