Facial Landmark Detection Research Articles

DRIVER DROWSINESS DETECTION AND CREATING SAFETY MEASURES USING OPENCV,PYTHON

Accidents due to driver drowsy driving is significantly increasing in high ratio now days. So we proposed an AI based solution to reduce the driver drossy driving accident by alerting the driver when they fall asleep. This is an brief abstract that explains system working and principles. This paper present a novel, Real-Time Driver Drowsiness Detection System(DDDS) that utilizes a multi-model approach for fatigue assessment. The system incorporates computer vision and machine learning techniques to analyze facial features, eye movements, and physical signal(if applicable) captured through a webcam processing techniques enhance data processing techniques enhanced data quality for better analysis.Facial landmarks are identified to locate the eyes, and features like Eye Aspect Ratio(EAR) are used to detect signs of fatigue. The system employs a multimodal approach, utilizing computer vision techniques to analyze facial features and eye movements captured through a webcam. Specifically, the DDDS leverages the Haar cascade algorithm for efficient facial landmark detection and the Histogram of Oriented Gradients (HOG) algorithm for feature extraction from facial regions of interest. This combination allows for accurate localization and analysis of key features like eyes. Additionally, the system explores the potential of physiological data (e.g., heart rate variability) for a more comprehensive analysis (if applicable). Machine learning techniques (if applicable) can be employed on the combined features, including those extracted using Haar and HOG, for enhanced fatigue classification. Upon detecting drowsiness, the DDDS triggers customizable alerts (visual, audio, or haptic) to warn the driver. The system additionally explores the potential of physiological data (e.g., heart rate variability) for fatigue detection, offering a more comprehensive approach (if applicable).A machine learning model can be employed on the combined features for robust fatigue classification. Upon detection of drowsiness,the DDDS triggers customizable alerts ( visual, audio, or haptic) to warn the driver. This system addresses limitations of existing solution by: 1.Real-Time processing:Ensuring immediate detection and response to fatigue signs. 2.Multi feature analysis: Combining facial features, eye movements,and physiological data for robust detection. 3.Customizable alerts: Providing drivers control over alert type for a personalized experience. The evaluation methodology employs a dataset of [data type e.g.., image, videos] to access the system’s performance.Metrics like accuracy,precision,recall,and F1-score will be used to evaluate the effectiveness of the DDDS in detecting drowsy driving. This research aims to contribute to safer roads by developing a comprehensive and adaptable Driver Drowsiness Detection System. II.KEYWORDS: Drowsy Driving, Driver Monitoring,Computer Vision, Machine Learning,Eye Tracking, Fatigue Detection,Physiological Signs(if applicable)

Fatigue Driving Detection Based on Driver Facial Temporal Sequences

Addressing the issues of low real-time performance and high false positive rates in driver fatigue detection methods based on deep learning, this paper proposes a temporal sequence Transformer-based fatigue detection method grounded in the localization of facial landmarks in drivers. Initially, the facial positions are obtained using the single-stage face detection algorithm RetinaFace. Subsequently, a lightweight GM module is designed as the principal feature extraction module for constructing a multi-scale fusion facial landmark detection network, and facial fatigue feature parameters based on temporal sequences are calculated according to the facial landmarks. Finally, a fatigue driving classification method based on temporal sequences Transformers is developed for classifying the sequences of fatigue feature parameters. Experimental results demonstrate that the inference time required for facial detection and landmark detection is merely 16.8 milliseconds, with the per-frame inference time for facial landmark detection being only 2.5 milliseconds, thus fulfilling the real-time requirements of fatigue driving detection during the feature extraction phase. A temporal sequence of fatigue feature parameters built on the NTHU-DDD dataset and the trained temporal sequence Transformer model resulted in an accuracy rate of 91.4% for the proposed method.

Advancements in Drowsiness Detection: A Review

This review study looks at current advancements in sleepiness detection technologies to address the urgent problem of driving while sleepy leading to accidents. In order to prevent future accidents, it analyzes several approaches and technology and emphasizes how crucial it is to correctly recognize drowsy conditions in drivers. Important results highlight the effectiveness of methods for real-time drowsiness detection such as convolutional neural networks (CNNs), eye aspect ratio (EAR) analysis, and deep learning integration. Additionally, improvements in computer vision algorithms make it possible to precisely analyze facial landmarks and eye blinking patterns, which improves the accuracy of detection systems. In order to protect driver comfort and system reliability, the evaluation emphasizes the importance of non-intrusive fatigue detection techniques while also emphasizing the necessity to reduce false warnings. Remarkably, certain models detect drowsy-eye with an accuracy of above 95%. All things considered, the review emphasizes how important cutting-edge technology is in reducing the risks related to drunk driving, especially deep learning and computer vision algorithms. It promotes the deployment of reliable sleepiness detection technologies in order to increase road safety and reduce accidents brought on by fatigued drivers. Key Words: Drowsiness detection, Eye blink duration, Facial landmark detection, Deep learning algorithms, Computer vision, Eye aspect ratio, Road safety.

Automated Detection of Cat Facial Landmarks

The field of animal affective computing is rapidly emerging, and analysis of facial expressions is a crucial aspect. One of the most significant challenges that researchers in the field currently face is the scarcity of high-quality, comprehensive datasets that allow the development of models for facial expressions analysis. One of the possible approaches is the utilisation of facial landmarks, which has been shown for humans and animals. In this paper we present a novel dataset of cat facial images annotated with bounding boxes and 48 facial landmarks grounded in cat facial anatomy. We also introduce a landmark detection convolution neural network-based model which uses a magnifying ensemble method. Our model shows excellent performance on cat faces and is generalizable to human and other animals facial landmark detection.

Quantifying nasal deformities using a novel mathematical method to complement preoperative assessment in rhinoplasty patients

BackgroundRhinoplasty enhances facial symmetry and functionality. However, the accurate and reliable quantification of nasal defects pre-surgery remains an ongoing challenge. AimThis study introduces a novel approach for defect quantification using 2D images and artificial intelligence, providing a tool for better preoperative planning and improved surgical outcomes. Materials and MethodsA pre-trained AI model for facial landmark detection was utilised on a dataset of 250 images of male patients aged 18 to 24 who underwent rhinoplasty for cosmetic nasal deformity correction. The analysis concentrated on 36 different distances between the facial landmarks. These distances were normalised using min-max scaling to counter image size and quality variations. Post-normalisation, statistical parameters, including mean, median, and standard deviation, were calculated to identify and quantify nasal defects. ResultsThe methodology was tested and validated using images from different ethnicities and regions, showing promising potential as a beneficial surgical aid. The normalised data produced reliable quantifications of nasal defects (average 76.2%), aiding in preoperative planning and improving surgical outcomes and patient satisfaction. ApplicationsThe developed method can be extended to other facial plastic surgeries. Furthermore, it can be used to create app-based software, assist medical education, and improve patient-doctor communication. ConclusionThis novel method for defect quantification in rhinoplasty using AI and image processing holds significant potential in improving surgical planning, outcomes, and patient satisfaction, marking an essential step in the fusion of AI and plastic surgery.

One level deep convolutional neural network for facial key points detection

Facial landmark detection has a lot of applications in face recognition, face alignment, facial expression recognition, video surveillance and security systems. In the existing literature, there are multiple methods utilizing convolutional neural networks (CNNs) that address this problem in various ways. In many cases, the models use a tree-like structure of CNNs to achieve better results. This paper proposes a combination of three parallel deep convolutional neural networks (DCNNs) to estimate the accurate localization of each keypoint. The first one focuses on the whole face to outperform five points, including the eyes, nose, and mouth corners. The second one focuses on the eyes-nose parts to outperform three points, specifically the eyes and nose. The last one focuses on the nose-mouth parts to outperform three points, namely the nose and mouth corners. Further, we combine all outputs of the three DCNNs and take the average value of each detected key point as the final output. In the first step, we improvthe the parameter efficiency and accuracy of each DCNNs through a set of experiments using the labeled face parts in-the-wild database (LFPW) and the helen facial feature dataset (Helen). Then, we demonstrate that our approach yields more accurate estimations of facial key points than two state-of-the-art methods and commercial software in terms of accuracy.

Automated anatomical landmark detection on 3D facial images using U-NET-based deep learning algorithm.

Facial anthropometry based on 3-dimensional (3D) imaging technology, or 3D photogrammetry, has gained increasing popularity among surgeons. It outperforms direct measurement and 2-dimensional (2D) photogrammetry because of many advantages. However, a main limitation of 3D photogrammetry is the time-consuming process of manual landmark localization. To address this problem, this study developed a U-NET-based deep learning algorithm to enable automated and accurate anatomical landmark detection on 3D facial models. The main structure of the algorithm stacked 2 U-NETs. In each U-NET block, we used 3×3 convolution kernel and rectified linear unit (ReLU) as activation function. A total of 200 3D images of healthy cases, acromegaly patients, and localized scleroderma patients were captured by Vectra H1 handheld 3D camera and input for algorithm training. The algorithm was tested to detect 20 landmarks on 3D images. Percentage of correct key points (PCK) and normalized mean error (NME) were used to evaluate facial landmark detection accuracy. Among healthy cases, the average NME was 1.4 mm. The PCK reached 90% when the threshold was set to the clinically acceptable limit of 2 mm. The average NME was 2.8 and 2.2 mm among acromegaly patients and localized scleroderma patients, respectively. This study developed a deep learning algorithm for automated facial landmark detection on 3D images. The algorithm was innovatively validated in 3 different groups of participants. It achieved accurate landmark detection and improved the efficiency of 3D image analysis.

You good?: Examining the role of intonation and eyebrow movements in sentence type distinction

By analyzing the pitch accents, edge tones, and eyebrow movements of African American English (AAE) speakers, this study examines the coordination of prosody and co-speech gestures. Twelve self-identified AAE speakers engaged in scripted dialogues with an AAE-speaking confederate via Zoom. Half of the dialogues contained the polysemous AAE idiomatic phrase “you good” which has declarative and interrogative forms hypothesized to be distinguished by various eyebrow and intonation patterns. The other dialogues included non-idiomatic phrases semantically related to their “you good” equivalents. These dialogues were designed to explore whether the coordination patterns for the idioms occur with non-idiomatic utterances of the same sentence type. Acoustic data is prosodically annotated using the MAE-ToBI transcription system and visual eyebrow data is tracked using OpenFace 2.0, a facial landmark detection toolkit. The derived eyebrow movement trajectories are time-aligned with the pitch trajectories for the semi-automatic labeling of multimodal signals. The temporal relationship between eyebrow movement landmarks, pitch accents, and edge tones is examined to investigate how different meanings and sentence types are created through their coordination. Preliminary examination indicates intonationally similar contours for the idiomatic and non-idiomatic interrogative phrases, but shared and consistent intonational contours are not evident in the declarative data.

Faces in Event Streams (FES): An Annotated Face Dataset for Event Cameras.

The use of event-based cameras in computer vision is a growing research direction. However, despite the existing research on face detection using the event camera, a substantial gap persists in the availability of a large dataset featuring annotations for faces and facial landmarks on event streams, thus hampering the development of applications in this direction. In this work, we address this issue by publishing the first large and varied dataset (Faces in Event Streams) with a duration of 689 min for face and facial landmark detection in direct event-based camera outputs. In addition, this article presents 12 models trained on our dataset to predict bounding box and facial landmark coordinates with an mAP50 score of more than 90%. We also performed a demonstration of real-time detection with an event-based camera using our models.

FPIRST: Fatigue Driving Recognition Method Based on Feature Parameter Images and a Residual Swin Transformer.

Fatigue driving is a serious threat to road safety, which is why accurately identifying fatigue driving behavior and warning drivers in time are of great significance in improving traffic safety. However, accurately recognizing fatigue driving is still challenging due to large intra-class variations in facial expression, continuity of behaviors, and illumination conditions. A fatigue driving recognition method based on feature parameter images and a residual Swin Transformer is proposed in this paper. First, the face region is detected through spatial pyramid pooling and a multi-scale feature output module. Then, a multi-scale facial landmark detector is used to locate 23 key points on the face. The aspect ratios of the eyes and mouth are calculated based on the coordinates of these key points, and a feature parameter matrix for fatigue driving recognition is obtained. Finally, the feature parameter matrix is converted into an image, and the residual Swin Transformer network is presented to recognize fatigue driving. Experimental results on the HNUFD dataset show that the proposed method achieves an accuracy of 96.512%, thus outperforming state-of-the-art methods.

Lightweight facial landmark detection network based on improved MobileViT

Lightweight facial landmark detection network based on improved MobileViT

SleepyWheels: An ensemble model for drowsiness detection leading to accident prevention

<p>Approximately 30% of traffic accidents in the world are attributed to driver drowsiness. Various approaches have been suggested by different research teams to detect drowsiness, but their methods have drawbacks. Some involve invasive techniques that cause driver discomfort, some involve too many false positives that cause distractions while driving while others rely on complex models that are too resource-intensive. In this paper, we present SleepyWheels, a novel drowsiness detection system. Our key insight is the combined use of two lightweight neural networks: a binary classifier and a facial landmark detector. This innovative approach minimizes false positives and proves resilient across diverse testing scenarios, such as different camera positions, variations in skin tone and when there is obscurement of facial features by objects like eyewear. Research outcomes include a working prototype of the system, a custom dataset for training the classifier and a trained model, which attains an impressive 97% accuracy rate. Deployment and testing were performed on Windows 10 but the system can be deployed on edge devices like Raspberry Pi. The lightweight nature of the models unlocks possibilities of deployment on mobile and embedded devices for use in vehicles.</p>

Multi-Angle Facial Recognition: Enhancing Biometric Security with a Broadly Positioned Stereo-Camera System

This study addresses the vulnerabilities of traditional monocular camera-based face recognition systems, emphasizing the need for improved security and reliability in biometric authentication under varying environmental conditions, lighting, and human poses. To counteract the risk of spoofing attacks using masks or static images, we introduce a multi-angle stereo camera system. This system is strategically designed to capture facial imagery from multiple perspectives, thereby enhancing depth perception and spatial accuracy, crucial for high-security authentication. Employing a novel image processing approach, the study integrates a Convolutional Neural Networks (CNN) with a simple Boolean operation to differentiate the landmarks detected on each camera. This method exploits CNN’s robust feature extraction capabilities and the effective usage of stereo camera, enabling precise detection and analysis of 3D facial landmarks. Such an approach significantly bolsters the system’s ability to differentiate between genuine faces and deceptive representations like masks or static images. Empirical results demonstrate that the stereo camera configuration substantially improves recognition accuracy, reducing both false positives and negatives, especially in controlled spoofing scenarios. The advanced 3D facial landmark detection further reinforces the system’s security. With its enhanced robustness and security, the developed system shows great potential for applications in areas requiring stringent identity verification, such as banking, public facilities, and smart home technologies.

Image-based investigation of lip aging features in a large number of Korean women.

The lips play a significant role in shaping facial aesthetics. Due to the distinct attributes of lips in contrast to other facial skin, a unique approach is imperative for managing lip aging. We analyzed lip characteristics (morphology, wrinkles, and color) to investigate visual changes and distinctive attributes of aging lips. By utilizing image data processing methods, including facial landmark detection, pattern recognition, and color quantification, we extracted 11 lip characteristic indices (four morphological indices, four wrinkle indices, and three color indices) from high-resolution images of 1000 Korean women aged 20-69. Correlation tests were conducted to assess the relationship between lip characteristic indices and age, and also between lip morphological and wrinkle indices. Lip height significantly decreased, while lip width and lip ratio (lip width divided by the sum of the upper and lower lip height) significantly increased with aging. Lip wrinkles significantly increased with aging, whereas lip colors (redness and yellowness) decreased. The lip wrinkle indices, which are segmented for the first time in this study, exhibited significant correlations with lip width, and three of them additionally were correlated with lip ratio (p<0.05). The results imply such morphological changes can be associated with wrinkle formation of human lips. The indices suggested in this study can be used for assessing lip aging characteristics, and the study results can contribute to deeper understanding of lip aging.

Facial Landmark Detection Using Machine Learning

Facial landmark detection is a pivotal task in the field of computer vision, holding significant implications across various applications, including facial analysis, augmented reality, facial recognition, and emotion detection. In visual communication, the face serves as a primary means of conveying information, and humans possess an innate ability to discern a wealth of information about another person's intentions, identity, and emotions merely by observing their facial expressions. The process of facial landmark detection using machine learning involves training a model on a dataset comprising real-time facial images, where each image is meticulously annotated with the coordinates of crucial facial landmarks. These landmarks typically include points representing the eyes, nose, mouth, and other facial features. Once trained, the model becomes proficient in recognizing and predicting the precise locations of these landmarks in new, unseen images and videos. The primary objective of this work is to predict facial landmarks in real-time using a live webcam input. This task is achieved through a two-step process. First, a Convolutional Neural Network (CNN) is trained on a comprehensive dataset, such as the 300W-LP dataset available on platforms like Kaggle. This dataset consists of facial images meticulously annotated with 68 distinct facial landmarks. The CNN learns to extract intricate facial features and landmark relationships from this training data, enabling it to recognize these landmarks with high accuracy. With the trained CNN in place, the next step is to extend its capabilities to perform facial landmark detection on live video streams. This involves integrating face detection techniques to identify and isolate faces within the live video feed. Once a face is detected, the CNN can then be applied to predict the locations of the 68 facial landmarks in real-time, enhancing our ability to understand and interpret facial expressions and movements in dynamic scenarios. In summary, this work underscores the significance of facial landmark detection in various computer vision applications. By training a CNN on a dataset of annotated facial images and integrating it with live video feeds, we aim to empower machines with the ability to understand and interpret human facial expressions and gestures in real-time, opening up new possibilities for applications in fields like human-computer interaction, emotion analysis, and beyond.

PicassoNet: Searching Adaptive Architecture for Efficient Facial Landmark Localization.

Since recent facial landmark localization methods achieve satisfying accuracy, few of them enable fast inference speed, which, however, is critical in many real-world facial applications. Existing methods typically employ complicated network structure and predict all the key points through uniform computation, which is inefficient since individual facial part might take different computation to obtain the best performance. Taking both accuracy and efficiency into consideration, we propose the PicassoNet, a lightweight cascaded facial landmark detector with adaptive computation for individual facial part. Different from the conventional cascaded methods, PicassoNet integrates refinement submodules into a single network with group convolution, where each convolution group predicts landmarks from an individual facial part. Note that the groups' structures are flexible in the training process. Then, a novel grouping search algorithm is proposed to optimize the group division. With formulating the optimization as a network architecture search (NAS) problem, the grouping search adaptively allocates computation to each group and obtains an efficient structure. In addition, we propose a boundary-aware loss to optimize along tangent and normal of facial boundaries, instead of optimizing along horizontal and vertical as the conventional loss (L2, SmoothL1, WingLoss, and so on) do. The novel loss improves the joint locations of predicted keypoints. Experiments on three benchmark datasets AFLW, 300W, and WFLW show that the proposed method runs over 6× times faster than the state of the arts and meanwhile achieves comparable accuracy.

Local eye-net: An attention based deep learning architecture for localization of eyes

Development of human machine interface has become a necessity for modern day machines to catalyze more autonomy and more efficiency. Gaze driven human intervention is an effective and convenient option for creating an interface to alleviate human errors. Facial landmark detection is very crucial for designing a robust gaze detection system. Regression based methods capacitate good spatial localization of the landmarks corresponding to different parts of the faces. But there are still scope of improvements which have been addressed by incorporating attention. In this paper, we have proposed a deep coarse-to-fine architecture called LocalEyenet for localization of only the eye regions that can be trained end-to-end. The model architecture, build on stacked hourglass backbone, learns the self-attention in feature maps which aids in preserving global as well as local spatial dependencies in face image. We have incorporated deep layer aggregation in each hourglass to minimize the loss of attention over the depth of architecture. Our model shows good generalization ability in cross-dataset evaluation and in real-time localization of eyes.

Anti-motion imaging photoplethysmography via self-adaptive multi-ROI tracking and selection

Objective. The imaging photoplethysmography (IPPG) technique allows people to measure heart rate (HR) from face videos. However, motion artifacts caused by rigid head movements and nonrigid facial muscular movements are one of the key challenges. Approach. This paper proposes a self-adaptive region of interest (ROI) pre-tracking and signal selection method to resist motion artifacts. Based on robust facial landmark detection, we split the whole facial skin (including the forehead, cheeks, and chin) symmetrically into small circular regions. And two symmetric sub-regions constitute a complete ROI. These ROIs are tracked and the motion state is simultaneously assessed to automatically determine the visibility of these ROIs. The obscured or invisible sub-regions will be discarded while the corresponding symmetric sub-regions will be retained as available ROIs to ensure the continuity of the IPPG signal. In addition, based on the frequency spectrum features of IPPG signals extracted from different ROIs, a self-adaptive selection module is constructed to select the optimum IPPG signal for HR calculation. All these operations are updated per frame dynamically for the real-time monitor. Results. Experimental results on the four public databases show that the IPPG signal derived by our proposed method exhibits higher quality for more accurate HR estimation. Compared with the previous method, metrics of the evaluated HR value on our approach demonstrates superior or comparable performance on PURE, VIPL-HR, UBFC-RPPG and MAHNOB-HCI datasets. For instance, the RMSEs on PURE, VIPL-HR, and UBFC-RPPG datasets decrease from 4.29, 7.62, and 3.80 to 4.15, 3.87, and 3.35, respectively. Significance. Our proposed method can help enhance the robustness of IPPG in real applications, especially given motion disturbances.

Open-Source 3D Morphing Software for Facial Plastic Surgery and Facial Landmark Detection Research and Open Access Face Data Set Based on Deep Learning (Artificial Intelligence) Generated Synthetic 3D Models.

Background: The scarcity of 3D facial models presents a significant hurdle for researchers and educators. Gathering such data demands substantial resources. Objective: To introduce an open-source 3D morphing software to generate 3D facial data sets for research and to provide a large sample data set that is based on synthetically generated 3D models. Methods: Software is developed to morph 3D facial models in bulk by altering landmark locations. Twenty synthetic 3D facial models are generated utilizing deep learning tools and 28 landmarks located on each. The measurements of synthetic models are confirmed to be realistic by comparing them with facial statistics. Several facial deformities and types are simulated at various magnitudes on 3D models to generate a large data set. Results: An open-source software and an open-access data set of 980 3D facial models, each with 28 landmark locations, are provided. Since the data set is based on synthetically generated 3D models, no institutional review board approval is required. Conclusion: The 3D morphing software and the large 3D data set are expected to benefit researchers and educators in the field of facial surgery and facial landmark detection.

SHELF: Combination of Shape Fitting and Heatmap Regression for Landmark Detection in Human Face

Today, facial emotion recognition is widely adopted in many intelligent applications including the driver monitoring system, the smart customer care as well as the e-learning system. In fact, the human emotions can be well represented by facial landmarks which are hard to be detected from images, due to the high number of discrete landmarks, the variation of shapes and poses of the human face in real world. Over decades, many methods have been proposed for facial landmark detection including the shape fitting, the coordinate regression such as ASMNet and AnchorFace. However, their performance is still limited for real-time applications in terms of both accuracy and efficiency. In this paper, we propose a novel method called SHELF which is the first to combine the shape fitting and heatmap regression approaches for landmark detection in human face. The heatmap model aims to generate the landmarks that fit to the common shapes. The method has been evaluated on three datasets 300W-Challenging, WFLW, 300VW-E with 31557 images and achieved a normalized mean error (NME) of 6.67% , 7.34%, 12.55% correspondingly, which overcomes most existing methods. For the first two datasets, the method is also comparable to the state of the art AnchorFace with a NME of 6.19%, 4.62%, respectively.