Facial Landmark Detection Research Articles

2S-SGCN: A two-stage stratified graph convolutional network model for facial landmark detection on 3D data

Facial Landmark Detection (FLD) algorithms play a crucial role in numerous computer vision applications, particularly in tasks such as face recognition, head pose estimation, and facial expression analysis. While FLD on images has long been the focus, the emergence of 3D data has led to a surge of interest in FLD on it due to its potential applications in various fields, including medical research. However, automating FLD in this context presents significant challenges, such as selecting suitable network architectures, refining outputs for precise landmark localization and optimizing computational efficiency. In response, this paper presents a novel approach, the 2-Stage Stratified Graph Convolutional Network (2S-SGCN), which addresses these challenges comprehensively. The first stage aims to detect landmark regions using heatmap regression, which leverages both local and long-range dependencies through a stratified approach. In the second stage, 3D landmarks are precisely determined using a new post-processing technique, namely MSE-over-mesh. 2S-SGCN ensures both efficiency and suitability for resource-constrained devices. Experimental results on 3D scans from the public Facescape and Headspace datasets, as well as on point clouds derived from FLAME meshes collected in the DAD-3DHeads dataset, demonstrate that the proposed method achieves state-of-the-art performance across various conditions. Source code is accessible at https://github.com/gfacchi-dev/CVIU-2S-SGCN.

RP-Net: A Robust Polar Transformation Network for rotation-invariant face detection

Face detection is challenging in unconstrained environments, where it encounters various challenges such as orientation, pose, and occlusion. Deep convolutional neural networks, particularly cascaded ones, have greatly improved detection performance but still struggle with rotating objects due to limitations in the Cartesian coordinate system. Although data augmentation can mitigate this issue, it also increases computational demands. This paper introduces the Robust Polar Transformation Network (RP-Net) for rotation-invariant face detection. RP-Net converts the complex rotational problem into a simpler translational one to enhance feature extraction and computational efficiency. Additionally, the Advanced Spatial-Channel Restoration (ASCR) module optimizes facial landmark detection within polar domains and restores critical details lost during transformation. Experimental results on benchmark datasets show that RP-Net significantly improves rotation invariance over traditional CNNs and surpasses several state-of-the-art rotation-invariant face detection methods.

A real-time eye movement-based computer interface for people with disabilities

It is costly to develop systems that enable individuals exposed to Amyotrophic Lateral Sclerosis and similar diseases that directly affect the neuromotor ability to communicate with the outside world. In this study, a budget friendly, high-accuracy, software-based, gaze-controlled, real-time virtual keyboard approach that can enable these people to communicate effectively is proposed. The proposed application requires only a computer and a webcam and has a user-friendly interface that meets the basic daily needs of individuals with disabilities. Since the proposed system does not require an extra action such as blinking, it makes it possible to use computers in advanced stage patients who cannot blink their eyes. The application which uses a deep learning-based facial landmark detector, determines the letters the user focuses on the screen and converts thoughts into text. The part of the screen that the user focuses on is determined with a new selection approach inspired by the K-Nearest Neighbors algorithm. This approach, which does not require blinking, offers high speed and accuracy. In the tests, a typing speed of 23.33 characters per minute is achieved with an accuracy rate of 95.12%. It is anticipated that the study will increase computer accessibility for disabled individuals with limited mobility and contribute to the development of real-time eye tracking systems.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

Eye Based Interactive Communication System

Eye-based communication systems, like the Blink-To-Speak language, are essential tools for individuals with motor neuron disorders, enabling them to articulate their needs and emotions. These systems leverage eye movements as a means of interaction, providing a vital communication channel for those with severe speech impairments. By interpreting specific eye gestures, users can effectively convey their thoughts, fostering a sense of independence and improving their quality of life. Many of the existing eye-tracking technologies tend to be complex and costly, limiting their accessibility in low-income regions. Using a mobile phone camera, the system captures real-time video frames to detect and track the user’s eyes, leveraging advanced facial landmark detection. Four primary eye movements—Left, Right, Up, and Blink—form the basis of the Blink-To-Live communication model, allowing users to communicate a range of more than 60 daily commands. By translating sequences of three eye gestures, the system generates readable sentences that are displayed on a screen and can be vocalized through synthesized speech. Unlike other sensor-dependent systems, Blink-To-Live is designed for affordability, flexibility, and ease of use, making it accessible without the need for specialized software or hardware. A prototype has been tested on a range of participants, showing positive results in terms of ease of use, flexibility, and cost-effectiveness

Impression management in the boardroom: How CEO facial trustworthiness influences turnover risk

This paper examines the effect of perceived trustworthiness on the turnover of Chief Executive Officers (CEOs) and its moderating role in the relationship between firm performance and CEO turnover. We employ a unique dataset comprising headshots of male CEOs in US-listed firms from 2010 to 2019 and a machine learning-based facial landmark detector to construct a composite facial trustworthiness index. Our results show that CEOs exhibiting high facial trustworthiness experience a lower turnover risk. We also find that facial trustworthiness strongly influences the link between firm performance and CEO turnover. Well-performing executives have a lower risk of being dismissed from their positions, and the dismissal risk is even lower among CEOs perceived as having a high facial trustworthiness. Conversely, underperforming CEOs are more likely to be dismissed, and the dismissal probability is greater among CEOs who have been perceived as being highly trustworthy. Our findings are consistent with the predictions of the expectancy violation theory, which posits that trustworthy-looking executives are more likely to be punished if they do not live up to the expectations held by the board of directors.

AnyFace++: Deep Multi-Task, Multi-Domain Learning for Efficient Face AI.

Accurate face detection and subsequent localization of facial landmarks are mandatory steps in many computer vision applications, such as emotion recognition, age estimation, and gender identification. Thanks to advancements in deep learning, numerous facial applications have been developed for human faces. However, most have to employ multiple models to accomplish several tasks simultaneously. As a result, they require more memory usage and increased inference time. Also, less attention is paid to other domains, such as animals and cartoon characters. To address these challenges, we propose an input-agnostic face model, AnyFace++, to perform multiple face-related tasks concurrently. The tasks are face detection and prediction of facial landmarks for human, animal, and cartoon faces, including age estimation, gender classification, and emotion recognition for human faces. We trained the model using deep multi-task, multi-domain learning with a heterogeneous cost function. The experimental results demonstrate that AnyFace++ generates outcomes comparable to cutting-edge models designed for specific domains.

DualFaceNet: augmentation consistency for optimal facial landmark detection and face mask classification

<p>In an era where face masks are commonplace, facial recognition faces new challenges and opportunities. This study introduces DualFaceNet (DFN), a cutting-edge neural network that efficiently combines facial landmark detection with mask classification. Benefiting from multi-task learning (MTL) and enhanced with a unique consistency loss, DFN outperforms traditional single-task models. Tests using the reputable 300W dataset and a face mask dataset showcase DFN’s strengths: a significant reduction in landmark error to 5.42 and an increase in mask classification accuracy to 92.59%. These results highlight the potential of integrating MTL and custom loss functions in facial recognition. As face masks continue to be globally essential, DFN’s integrated approach offers a fresh perspective in facial recognition studies. Furthermore, DFN paves the way for adaptive facial recognition systems, emphasizing the adaptability needed in our current era.</p>

Exploring Attentive User Interface Input via Raspberry Pi, Based on Face Landmark Detection, Eye Open-Closed Detection and Head Movements Detection

Exploring Attentive User Interface Input via Raspberry Pi, Based on Face Landmark Detection, Eye Open-Closed Detection and Head Movements Detection

A Survey Study of the 3D Facial Landmark Detection Techniques Used as a Screening Tool for Diagnosis of the Obstructive Sleep Apnea Syndrome.

Obstructive Sleep Apnea (OSA) is a common disorder affecting both adults and children. It is characterized by repeated episodes of apnea (stopped breathing) and hypopnea (reduced breathing), which result in intermittent hypoxia. We recognize pediatric and adult OSA, and this paper focuses on pediatric OSA. While adults often suffer from daytime sleepiness, children are more likely to develop behavioral abnormalities. Early diagnosis and treatment are important to prevent negative effects on children's development. Without the treatment, children may be at increased risk of developing high blood pressure or other heart problems. The gold standard for OSA diagnosis is the polysomnography (sleep study) PSG performed at a sleep center. Not only is it an expensive procedure, but it can also be very stressful, especially for children. Patients have to stay at the sleep center during the night. Therefore, screening tools are very important. Multiple studies have shown that OSA screening tools can be based on facial anatomical landmarks. Anatomical landmarks are landmarks located at specific anatomical locations. For the purpose of the screening tool, a specific list of anatomical locations needs to be identified. We are presenting a survey study of the automatic identification of these landmarks on 3D scans of the patient's head. We are considering and comparing both knowledge-based and AI-based identification techniques, with a focus on the development of the automatic OSA screening tool.

A Study on Facial Landmark Detection and Image Processing

A Study on Facial Landmark Detection and Image Processing

Precise facial landmark detection by Dynamic Semantic Aggregation Transformer

At present, deep neural network methods have played a dominant role in face alignment field. However, they generally use predefined network structures to predict landmarks, which tends to learn general features and leads to mediocre performance, e.g., they perform well on neutral samples but struggle with faces exhibiting large poses or occlusions. Moreover, they cannot effectively deal with semantic gaps and ambiguities among features at different scales, which may hinder them from learning efficient features. To address the above issues, in this paper, we propose a Dynamic Semantic-Aggregation Transformer (DSAT) for more discriminative and representative feature (i.e., specialized feature) learning. Specifically, a Dynamic Semantic-Aware (DSA) model is first proposed to partition samples into subsets and activate the specific pathways for them by estimating the semantic correlations of feature channels, making it possible to learn specialized features from each subset. Then, a novel Dynamic Semantic Specialization (DSS) model is designed to mine the homogeneous information from features at different scales for eliminating the semantic gap and ambiguities and enhancing the representation ability. Finally, by integrating the DSA model and DSS model into our proposed DSAT in both dynamic architecture and dynamic parameter manners, more specialized features can be learned for achieving more precise face alignment. It is interesting to show that harder samples can be handled by activating more feature channels. Extensive experiments on popular face alignment datasets demonstrate that our proposed DSAT outperforms state-of-the-art models in the literature. Our code is available at https://github.com/GERMINO-LiuHe/DSAT.

Micro-expression recognition based on a novel GCN-transformer cooperation model for IoT-eHealth

If a person is truly healthy, his/her well-being encompasses both physical and psychological health. However, the existing IoT-eHealth system typically focus only on monitoring the user’s physical data through various sensors, neglecting their mental state. To enhance the intelligence level of IoT-eHealth system and enable it to have the psychological monitoring ability, a novel collaborative model based on Graph Convolutional Network (GCN) and Transformer is designed for Micro-Expression (ME) recognition in this paper. Firstly, facial information within each frame is transformed into a Spatial Topological Relationship Graph (STRG) by using facial landmarks detection and psychological relationship of local patches. Then, in order to automatically aggregate the key information on facial patches that contribute to ME recognition from the structured graph data, a Hierarchical Adaptive Graph Pooling (HAGP) module is designed for obtaining discriminative frame-level feature based on GCN utilizing graph structure and vertex global dependencies. Finally, in order to model the long-term dependencies among frames and capture the key frame-level features that are beneficial for ME recognition, a Temporal Sensitive Self-Attention (TSSA) mechanism is designed, and a novel Temporal Sensitive Transformer (TST) encoder is constructed based on TSSA to explore the evolution law of facial patterns and obtain discriminative video-level features that are helpful for ME recognition. In the comparative experiments of standard dataset verification and practical dataset testing, designed collaborative model is superior to other methods and can achieve the highest recognition accuracy, which almost can meet the application requirements of IoT-eHealth system.

A smartphone application for personalized facial aesthetic monitoring.

Methods available at home for capturing facial images to track changes in skin quality and evaluate skincare treatments are limited. In this study, we developed a smartphone camera application (app) for personalized facial aesthetic monitoring. A face alignment indicators (FAIN) system utilizing facial landmark detection, an artificial intelligence technique, to estimate key facial parts, was implemented into the app to maintain a consistent facial appearance during image capture. The FAIN system is composed of a fixed target indicator and an alignment indicator that dynamically changes its shape according to the user's face position, size, and orientation. Users align their faces to match the alignment indicator with the fixed target indicator, and the image is automatically captured when alignment is achieved. We investigated the app's effectiveness in ensuring a consistent facial appearance by analyzing both geometric and colorimetric data. Geometric information from captured faces and colorimetric data from stickers applied to the faces were utilized. The coefficients of variation (CVs) for the L*, a*, and b* values of the stickers were higher compared to those measured by a colorimeter, with CVs of 14.9 times, 8.14 times, and 4.41 times for L*, a*, and b*, respectively. To assess the feasibility of the app for facial aesthetic monitoring, we tracked changes in pseudo-skin color on the cheek of a participant using skin-colored stickers. As a result, we observed the smallest color difference ∆Eab of 1.901, which can be considered as the experimentally validated detection limit using images acquired by the app. While the current monitoring method is a relative quantification approach, it contributes to evidence-based evaluations of skincare treatments.

Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection

AbstractIn this paper, we examine three important issues in the practical use of state‐of‐the‐art facial landmark detectors and show how a combination of specific architectural modifications can directly improve their accuracy and temporal stability. First, many facial landmark detectors require a face normalization step as a pre‐process, often accomplished by a separately trained neural network that crops and resizes the face in the input image. There is no guarantee that this pre‐trained network performs optimal face normalization for the task of landmark detection. Thus, we instead analyse the use of a spatial transformer network that is trained alongside the landmark detector in an unsupervised manner, jointly learning an optimal face normalization and landmark detection by a single neural network. Second, we show that modifying the output head of the landmark predictor to infer landmarks in a canonical 3D space rather than directly in 2D can further improve accuracy. To convert the predicted 3D landmarks into screen‐space, we additionally predict the camera intrinsics and head pose from the input image. As a side benefit, this allows to predict the 3D face shape from a given image only using 2D landmarks as supervision, which is useful in determining landmark visibility among other things. Third, when training a landmark detector on multiple datasets at the same time, annotation inconsistencies across datasets forces the network to produce a sub‐optimal average. We propose to add a semantic correction network to address this issue. This additional lightweight neural network is trained alongside the landmark detector, without requiring any additional supervision. While the insights of this paper can be applied to most common landmark detectors, we specifically target a recently proposed continuous 2D landmark detector to demonstrate how each of our additions leads to meaningful improvements over the state‐of‐the‐art on standard benchmarks.

Modified DSFD and TCDCN Based Facial Landmark Detection for Gender and Age Classification

Modified DSFD and TCDCN Based Facial Landmark Detection for Gender and Age Classification

Alertness Assured: Sleep Detection and Video Control Technology

Abstract: The system utilizes advanced computer vision algorithms to analyze real-time video streams, employing techniques such as facial landmark detection and eye movement tracking. By continuously monitoring these visual cues, the system can accurately detect signs of drowsiness, such as drooping eyelids or prolonged eye closures, prompting timely interventions to prevent accidents or errors caused by fatigue. In addition to detecting drowsiness, the system also evaluates the user's attention levels by analysing head orientation and facial expressions. By tracking head movements and assessing changes in facial expressions indicative of engagement or distraction, the system provides valuable insights into the user's cognitive state. This attention assessment component enables the system to adapt its feedback and intervention strategies based on the user's level of alertness and focus. To enhance user awareness and responsiveness, the system employs graphical visualization techniques to display real-time feedback on drowsiness and attention status. Visual indicators, such as color-coded alerts or dynamic graphs depicting attention trends, provide users with intuitive insights into their cognitive performance, empowering them to make informed decisions about their work habits or driving patterns. Moreover, the system utilizes auditory alerts to supplement visual feedback, ensuring that users receive timely notifications even in noisy or visually demanding environments. Whether it's a gentle reminder to take a short break or a more urgent warning about escalating drowsiness levels, auditory cues serve as an effective means of alerting users to potential risks and encouraging proactive intervention

Facial Action Unit detection based on multi-task learning strategy for unlabeled facial images in the wild

Facial Action Unit (AU) detection often relies on highly-cost accurate labeling or inaccurate pseudo labeling techniques in recent years. How to introduce large amounts of unlabeled facial images in the wild into supervised AU detection frameworks has become a challenging problem. Additionally, nearly every type of AUs has the problem of unbalanced positive and negative samples. Inspired by other multi-task learning frameworks, we first propose a multi-task learning strategy boosting AU detection in the wild through jointing facial landmark detection and AU domain separation and reconstruction. Our introduced dual domains facial landmark detection framework can solve the lack of accurate facial landmark coordinates during the AU domain separation and reconstruction training process, while the parameters of homostructural facial extraction modules from these two similar facial tasks are shared. Moreover, we propose a pixel-level feature alignment scheme to maintain the consistency of features obtained from two separation and reconstruction processes. Furthermore, a weighted asymmetric loss is proposed to change the contribution of positive and negative samples of each type of AUs to model parameters updating. Experimental results on three widely used benchmarks demonstrate our superiority to most state-of-the-art methods for AU detection.

Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation

Most facial analysis methods perform well in standardized testing but not in real-world testing. The main reason is that training models cannot easily learn various human features and background noise, especially for facial landmark detection and head pose estimation tasks with limited and noisy training datasets. To alleviate the gap between standardized and real-world testing, we propose a pseudo-labeling technique using a face recognition dataset consisting of various people and background noise. The use of our pseudo-labeled training dataset can help to overcome the lack of diversity among the people in the dataset. Our integrated framework is constructed using complementary multitask learning methods to extract robust features for each task. Furthermore, introducing pseudo-labeling and multitask learning improves the face recognition performance by enabling the learning of pose-invariant features. Our method achieves state-of-the-art (SOTA) or near-SOTA performance on the AFLW2000-3D and BIWI datasets for facial landmark detection and head pose estimation, with competitive face verification performance on the IJB-C test dataset for face recognition. We demonstrate this through a novel testing methodology that categorizes cases as soft, medium, and hard based on the pose values of IJB-C. The proposed method achieves stable performance even when the dataset lacks diverse face identifications.

DRIVER DROWSINESS DETECTION SYSTEM

The proposed system utilizes facial landmark detection and eye aspect ratio calculation to detect drowsiness in real-time from a webcam feed. It employs the dlib library for face detection and facial landmark prediction. The eye aspect ratio (EAR) is computed from the distances between specific points on the eyes. When the EAR falls below a predefined threshold for a certain number of consecutive frames, it triggers an alert indicating potential drowsiness. The alert includes visual cues on the video feed, an auditory alert using the playsound library, and even sends an instant WhatsApp message using pywhatkit to notify a designated contact about the drowsiness. This system serves as a useful tool for monitoring driver fatigue or alertness in various contexts, promoting safety and awareness. Its capacity to swiftly identify and alert users to the onset of drowsiness represents a significant leap forward in safety technology. By mitigating the risk of accidents stemming from fatigue-induced impairments, this project underscores the pivotal role of computer vision in real-time human behavior monitoring, offering a versatile and invaluable tool for fostering safety and awareness across various domains. Keywords: Drowsiness Detection; Python; Whatsapp messaging; Webcam feed; Computer Vision; Alert System; Real-time monitoring; Eye-Aspect Ratio; Frame Processing; Facial Recognition.