Facial Keypoints Research Articles

Face Keypoint Detection Method Based on Blaze_ghost Network

The accuracy and speed of facial keypoint detection are crucial factors for effectively extracting fatigue features, such as eye blinking and yawning. This paper focuses on the improvement and optimization of facial keypoint detection algorithms, presenting a facial keypoint detection method based on the Blaze_ghost network and providing more reliable support for facial fatigue analysis. Firstly, the Blaze_ghost network is designed as the backbone network with a deeper structure and more parameters to better capture facial detail features, improving the accuracy of keypoint localization. Secondly, HuberWingloss is designed as the loss function to further reduce the training difficulty of the model and enhance its generalization ability. Compared to traditional loss functions, HuberWingloss can reduce the interference of outliers (such as noise and occlusion) in model training, improve the model’s robustness to complex situations, and further enhance the accuracy of keypoint detection. Experimental results show that the proposed method achieves significant improvements in both the NME (Normal Mean Error) and FR (Failure Rate) evaluation metrics. Compared to traditional methods, the proposed model demonstrates a considerable improvement in keypoint localization accuracy while still maintaining high detection efficiency.

Leveraging Computer Vision to Establish a Correlation Between Eye-Gaze Estimation and Saccades

The purpose of this project was to develop a model that can detect pupils on a subject’s face and draw the graph of both pupils to determine if a subject has saccades. Current methods cannot accurately detect contours when the face is presented at an angle. To resolve this issue, subjects recorded a video while performing the Head Impulse Test (HIT). Next, a computer vision library, Opencv, extracted frames from the recorded video and detected the facial key point for the nose. The Multi-task Cascaded Convolutional Neural Network (MTCNN) extracted the face from the frame and generated contours for the eyes using a segmentation library. The largest contours on the mask were divided into two parts and their extreme bounding box points were identified using dilation, erosion, and blur. Our model rendered a video of both contours applied to the patient performing the HIT test. Also, the model generated two graphs, comparing each eye’s gaze with the pose estimation. There were two main outputs: the graph may resemble a y = -x line if a patient does not have saccades or the graph may be more distorted if the patient has saccades due to a sudden shift in eye-gaze movement. This study demonstrates a precise method of eye-gaze estimation and the detection of saccades. Further experiments will help to validate the notion that saccades are connected to neurological disorders. Future experiments may include more data during the training process and to quantitatively determine the accuracy of our model.

Efficient and Robust Driver Fatigue Detection Framework Based on the Visual Analysis of Eye States

Fatigue detection based on vision is widely employed in vehicles due to its real-time and reliable detection results. With the coronavirus disease (COVID-19) outbreak, many proposed detection systems based on facial characteristics would be unreliable due to the face covering with the mask. In this paper, we propose a robust visual-based fatigue detection system for monitoring drivers, which is robust regarding the coverings of masks, changing illumination and head movement of drivers. Our system has three main modules: face key point alignment, fatigue feature extraction and fatigue measurement based on fused features. The innovative core techniques are described as follows: (1) a robust key point alignment algorithm by fusing global face information and regional eye information, (2) dynamic threshold methods to extract fatigue characteristics and (3) a stable fatigue measurement based on fusing percentage of eyelid closure (PERCLOS) and proportion of long closure duration blink (PLCDB). The excellent performance of our proposed algorithm and methods are verified in experiments. The experimental results show that our key point alignment algorithm is robust to different scenes, and the performance of our proposed fatigue measurement is more reliable due to the fusion of PERCLOS and PLCDB.

Measurement Method of Interpupillary Distance and Pupil Height Based on Ensemble of Regression Trees and the BlendMask Algorithm

Measuring interpupilary distance and pupil height is a crucial step in the process of optometry. However, existing methods suffer from low accuracy, high cost, a lack of portability, and limited research on studying both parameters simultaneously. To overcome these challenges, we propose a method that combines ensemble regression trees (ERT) with the BlendMask algorithm to accurately measure both interpupillary distance and pupil height. First, we train an ERT-based face keypoint model to locate the pupils and calculate their center coordinates. Then, we develop an eyeglass dataset and train a BlendMask model to obtain the coordinates of the lowest point of the lenses. Finally, we calculate the numerical values of interpupillary distance and pupil height based on their respective definitions. The experimental results demonstrate that the proposed method can accurately measure interpupillary distance (IPD) and pupil height, and the calculated IPD and pupil height values are in good agreement with the measurements obtained by an auto-refractometer. By combining the advantages of the two models, our method overcomes the limitations of traditional methods with high measurement accuracy, low cost, and strong portability. Moreover, this method enables fast and automatic measurement, minimizing operation time, and reducing human errors. Therefore, it possesses broad prospects for application, particularly in the fields of eyeglass customization and vision inspection.

Facial keypoints detection using capsule neural networks

Facial keypoints detection using capsule neural networks

Driver Attention Detection Based on Improved YOLOv5

In response to negative impacts such as personal and property safety hazards caused by drivers being distracted while driving on the road, this article proposes a driver’s attention state-detection method based on the improved You Only Look Once version five (YOLOv5). Both fatigue and distracted behavior can cause a driver’s attention to be diverted during the driving process. Firstly, key facial points of the driver are located, and the aspect ratio of the eyes and mouth is calculated. Through the examination of relevant information and repeated experimental verification, threshold values for the aspect ratio of the eyes and mouth under fatigue conditions, corresponding to closed eyes and yawning, are established. By calculating the aspect ratio of the driver’s eyes and mouth, it is possible to accurately detect whether the driver is in a state of fatigue. Secondly, distracted abnormal behavior is detected using an improved YOLOv5 model. The backbone network feature extraction element is modified by adding specific modules to obtain different receptive fields through multiple convolution operations on the input feature map, thereby enhancing the feature extraction ability of the network. The introduction of Swin Transformer modules in the feature fusion network replaces the Bottleneck modules in the C3 module, reducing the computational complexity of the model while increasing its receptive field. Additionally, the network connection in the feature fusion element has been modified to enhance its ability to fuse information from feature maps of different sizes. Three datasets were created of distracting behaviors commonly observed during driving: smoking, drinking water, and using a mobile phone. These datasets were used to train and test the model. After testing, the mAP (mean average precision) has improved by 2.4% compared to the model before improvement. Finally, through comparison and ablation experiments, the feasibility of this method has been verified, which can effectively detect fatigue and distracted abnormal behavior.

Analysis and Study of Fatigue Driving Detection System based on Multimodal Fusion

Fatigue driving always endangers road traffic safety and is one of the main causes of traffic accidents. How to effectively monitor and determine the fatigue driving state, so as to realize the fatigue early warning, has become a hot spot of scientific research. The traditional detection methods are mostly based on a certain human physiological parameters, through the monitoring of the physiological parameters changes, to complete the determination of the fatigue state. Compared with the traditional detection methods, the fatigue detection method based on computer vision has the advantages of accurate and reliable detection results, detection not relying on wearable devices, and good human-computer interaction experience. However, in the specific algorithm application scenarios, there are often interference from environmental factors such as partial occlusion of the inspected image, low image resolution, complex illumination of the detection environment, and unstable acquisition equipment. In the face of many challenges, how to effectively improve the detection accuracy of fatigue state, and the algorithm has the practical application space, still needs to be explored. The traditional fatigue detection method based on facial information often detects and analyzes a certain subordination. Due to the detection of individual differences and the fuzzy definition of fatigue state, the algorithm has the problems of high recognition error rate and poor robustness. Moreover, the single attribute model separates the potential integration and exclusion relationship between facial attributes. At the same time, the huge model size and complex model structure also limit the actual implementation of the algorithm. In view of these urgent problems, this paper first proposes the fusion model of face detection and head posture analysis based on the multi-task convolutional neural network (MTCNN). The model uses the architecture strategy of multi-task cascade to complete the complete facial posture analysis task while completing the facial detection task. Secondly, based on the lightweight convolutional neural network (SqueezeNet) of the network cascade, the facial key point detection network is designed to obtain the 72-point key point coordinates of the face. For the obtained facial attribute information, extract fatigue features and propose a fatigue determination method of multimodal fusion. Finally, combined with the research results, the construction of the online fatigue detection and early warning system platform was completed.

Research on Railway Dispatcher Fatigue Detection Method Based on Deep Learning with Multi-Feature Fusion

Traffic command and scheduling are the core monitoring aspects of railway transportation. Detecting the fatigued state of dispatchers is, therefore, of great significance to ensure the safety of railway operations. In this paper, we present a multi-feature fatigue detection method based on key points of the human face and body posture. Considering unfavorable factors such as facial occlusion and angle changes that have limited single-feature fatigue state detection methods, we developed our model based on the fusion of body postures and facial features for better accuracy. Using facial key points and eye features, we calculate the percentage of eye closure that accounts for more than 80% of the time duration, as well as blinking and yawning frequency, and we analyze fatigue behaviors, such as yawning, a bowed head (that could indicate sleep state), and lying down on a table, using a behavior recognition algorithm. We fuse five facial features and behavioral postures to comprehensively determine the fatigue state of dispatchers. The results show that on the 300 W dataset, as well as a hand-crafted dataset, the inference time of the improved facial key point detection algorithm based on the retina–face model was 100 ms and that the normalized average error (NME) was 3.58. On our own dataset, the classification accuracy based the an Bi-LSTM-SVM adaptive enhancement algorithm model reached 97%. Video data of volunteers who carried out scheduling operations in the simulation laboratory were used for our experiments, and our multi-feature fusion fatigue detection algorithm showed an accuracy rate of 96.30% and a recall rate of 96.30% in fatigue classification, both of which were higher than those of existing single-feature detection methods. Our multi-feature fatigue detection method offers a potential solution for fatigue level classification in vital areas of the industry, such as in railway transportation.

Local Corner and Motion Key Point Trajectory Extraction for Facial Forgery Identification

At present, the development of deep forgery technology has brought new challenges to media content forensics, and the use of deep forgery identification methods to identify forged audio and video has become a significant focus of research and difficulty. Deep forgery technology and forensic technology play a mutual game and promote each other’s development. This paper proposes a spatiotemporal local feature abstraction (STLFA) framework for facial forgery identification to solve the media industry challenges of deep forgery technology. To adequately utilize local facial features, we combine facial key points, key point movement, and facial corner points to detect forgery content. This paper establishes a spatiotemporal relation, which realizes face forgery detection by identifying abnormalities of facial keypoints and corner points for interframe judgments. Meanwhile, we utilize RNNs to predict the sequences from facial key point movement abnormalities and corner points for interframe. Experimental results show that our method achieves better performance than some existing methods and good anticompression forgery face detection performance on FF++.

MHealth technology identifies visual impairment in young children.

mHealth technology identifies visual impairment in young children.

Mental health trends among medical students

Student mental health concerns can manifest in several forms. Medical students juggling a multitude of trials (i.e., intense academic rigor, financial debt, sleep deprivation, lack of control, continual exposure to sickness and death, and training mistreatment) can help explain the higher prevalence of psychological disorders within this population. Furthermore, these mental health difficulties are not static; certain challenges move into the forefront as students face key transition points in schooling. Primary examples include the entry year of medical school, the shift from preclinical curriculum to clinical training, and the final moments prior to beginning residency. Given the existing mental health trends among medical students at baseline, it can be concluded that the COVID-19 pandemic has exacerbated the stress, anxiety, and depression associated with medical education. Solutions do indeed exist to address the moral injury medical students face, from expanded crisis management training and implementation of peer support networks to destigmatization of and improved access to professional mental health resources. It is up to the curators of the medical education system to make these solutions the new status quo.

Regenerating vital facial keypoints for impostor identification from disguised images using CNN

Technical advancements in the digital era have eased up the time required to complete complex tasks by a decisive rate. These advancements introduce several threats. Hence, there is a compelling need to provide security for systems against these threats. Identity and data thefts are the major threats which require immediate, effective and responsive solutions. Face recognition systems have evolved over the last few years, providing a suitable solution for authenticating an individual. However, the system running on facial recognition can be interfered by disguise. This paper discusses the approaches used in detecting a disguise and analysing the captured face among the other non-disguised faces in the dataset. The parameters used for detection and analysis are the keypoints of the face. These keypoints may be blocked, hidden or disoriented due to the presence of props on the face. The proposed approach aims to avoid this interference by creating an estimate of the face based on fewer available keypoints. The estimated face so created, is then matched with the available set of faces to determine the probabilistic chance of the matched person’s presence even though a disguise had been detected. Thus the solution provides a clear distinction if the image obtained proves to be of an impostor or an intruder with obfuscation with a said probability. The Disguised Faces in the Wild (DFW) dataset has been used.

A Real-Time Face Detection Method Based on Blink Detection

Aiming at the photo fraud that often occurs in identity verification and the accuracy and robustness of real-time video face recognition, this paper proposes a real-time face detection method based on blink detection. This method first extracts the image texture features through the LBP algorithm, which eliminates the problem of illumination changes to a certain extent. Then the extracted features are input into the ResNet network, and the facial feature extraction is enhanced by adding an attention mechanism is added to enhance the face feature extraction. Meanwhile, the BiLSTM method is used to extract the temporal characteristics of images from different angles or at different times to obtain more facial details. In addition, the fusion of local and global features is realized by SPP pooling, which enriches the expression ability of feature maps and improves detection accuracy. Finally, the eye EAR value is calculated by the face key point detection technology to achieve face anti-spoofing, and then the real-time face recognition against fraud is realized. The experimental results show that the algorithm proposed in this paper has good accuracy on NUAA, CASIA-SURF and CASIA-FASD datasets, which can reach 99.48%, 98.65% and 99.17%, respectively.

Toward Fine-Grained Talking Face Generation.

Talking face generation is the process of synthesizing a lip-synchronized video when given a reference portrait and an audio clip. However, generating a fine-grained talking video is nontrivial due to several challenges: 1) capturing vivid facial expressions, such as muscle movements; 2) ensuring smooth transitions between consecutive frames; and 3) preserving the details of the reference portrait. Existing efforts have only focused on modeling rigid lip movements, resulting in low-fidelity videos with jerky facial muscle deformations. To address these challenges, we propose a novel Fine-gRained mOtioN moDel (FROND), consisting of three components. In the first component, we adopt a two-stream encoder to capture local facial movement keypoints and embed their overall motion context as the global code. In the second component, we design a motion estimation module to predict audio-driven movements. This enables the learning of local key point motion in the continuous trajectory space to achieve smooth temporal facial movements. Additionally, the local and global motions are fused to estimate a continuous dense motion field, resulting in spatially smooth movements. In the third component, we devise a novel implicit image decoder based on an implicit neural network. This decoder recovers high-frequency information from the input image, resulting in a high-fidelity talking face. In summary, the FROND refines the motion trajectories of facial keypoints into a continuous dense motion field, which is followed by a decoder that fully exploits the inherent smoothness of the motion. We conduct quantitative and qualitative model evaluations on benchmark datasets. The experimental results show that our proposed FROND significantly outperforms several state-of-the-art baselines.

FSA-Net: A Cost-efficient Face Swapping Attention Network with Occlusion-Aware Normalization

The main challenges in face swapping are the preservation and adaptive superimposition of attributes of two images. In this study, the Face Swapping Attention Network (FSA-Net) is proposed to generate photorealistic face swapping. The existing face-swapping methods ignore the blending attributes or mismatch the facial keypoint (cheek, mouth, eye, nose, etc.), which causes artifacts and makes the generated face silhouette non-realistic. To address this problem, a novel reinforced multi-aware attention module, referred to as RMAA, is proposed for handling facial fusion and expression occlusion flaws. The framework includes two stages. In the first stage, a novel attribute encoder is proposed to extract multiple levels of target face attributes and integrate identities and attributes when synthesizing swapped faces. In the second stage, a novel Stochastic Error Refinement (SRE) module is designed to solve the problem of facial occlusion, which is used to repair occlusion regions in a semi-supervised way without any post-processing. The proposed method is then compared with the current state-of-the-art methods. The obtained results demonstrate the qualitative and quantitative outperformance of the proposed method. More details are provided at the footnote link and at .

Uncertainty-aware Gaze Tracking for Assisted Living Environments.

Effective assisted living environments must be able to infer how their occupants interact in a variety of scenarios. Gaze direction provides strong indications of how a person engages with the environment and its occupants. In this paper, we investigate the problem of gaze tracking in multi-camera assisted living environments. We propose a gaze tracking method based on predictions generated by a neural network regressor that relies only on the relative positions of facial keypoints to estimate gaze. For each gaze prediction, our regressor also provides an estimate of its own uncertainty, which is used to weigh the contribution of previously estimated gazes within a tracking framework based on an angular Kalman filter. Our gaze estimation neural network uses confidence gated units to alleviate keypoint prediction uncertainties in scenarios involving partial occlusions or unfavorable views of the subjects. We evaluate our method using videos from the MoDiPro dataset, which we acquired in a real assisted living facility, and on the publicly available GazeFollow and Gaze360 datasets. Experimental results show that our gaze estimation network outperforms sophisticated state-of-the-art methods, while additionally providing uncertainty predictions that are highly correlated with the actual angular error of the corresponding estimates. Finally, an analysis of the temporal integration performance of our method demonstrates that it generates accurate and temporally stable gaze predictions.

The Development and Challenges of Face Alignment: A survey

Finding the face key-points location or which is also called Face alignment has gotten a lot of interest in the last twenty years because of its wide range of applications in the major of automatic face analysis. Nonetheless, this task has proven an extremely challenging task in an unconstrained environment because of multiple confusing factors, such as occlusions, position, illumination, and expression. However, this task is still an open problem, even though researchers still developing a great many techniques to face the challenges of this problem, in this survey, a critical review of the current literature in the face alignment field had already been presented focusing on those methods tackling the total of this topic's challenges and difficulties under uncontrollable circumstances. Especially, the existing face alignment techniques have been categorized existing referring to more than the type of categorizing. Then represent a survey of the recent related works in face alignment and lastly, we discuss the general steps of the face alignment system.

Pedestrian Trajectory Prediction in Heterogeneous Traffic using Facial Keypoints-based Convolutional Encoder-decoder Network

Future pedestrian trajectory prediction offers great prospects for many practical applications such as unmanned vehicles, building evacuation design and robotic path planning. Most existing methods focus on social interaction among pedestrians but ignore the fact that heterogeneous traffic objects (cars, dogs, bicycles, motorcycles, etc.) have significant influence on the future trajectory of a subject pedestrian. Also, the walking direction intention of a pedestrian may be referred by his/her facial keypoints. Considering this, this work proposes to predict a pedestrian's future trajectory by jointly using neighboring heterogeneous traffic information and his/her facial keypoints. To fulfill this, an end-to-end facial keypoints-based convolutional encoder-decoder network (FK-CEN) is designed, in which the heterogeneous traffic and facial keypoints are input. After training, FK-CEN is evaluated on 5 crowded video sequences collected from the public datasets MOT-16 and MOT-17. Experimental results demonstrate that it outperforms state-of-the-art approaches, in terms of prediction errors.

Development of Real-Time Landmark-Based Emotion Recognition CNN for Masked Faces.

Owing to the availability of a wide range of emotion recognition applications in our lives, such as for mental status calculation, the demand for high-performance emotion recognition approaches remains uncertain. Nevertheless, the wearing of facial masks has been indispensable during the COVID-19 pandemic. In this study, we propose a graph-based emotion recognition method that adopts landmarks on the upper part of the face. Based on the proposed approach, several pre-processing steps were applied. After pre-processing, facial expression features need to be extracted from facial key points. The main steps of emotion recognition on masked faces include face detection by using Haar-Cascade, landmark implementation through a media-pipe face mesh model, and model training on seven emotional classes. The FER-2013 dataset was used for model training. An emotion detection model was developed for non-masked faces. Thereafter, landmarks were applied to the upper part of the face. After the detection of faces and landmark locations were extracted, we captured coordinates of emotional class landmarks and exported to a comma-separated values (csv) file. After that, model weights were transferred to the emotional classes. Finally, a landmark-based emotion recognition model for the upper facial parts was tested both on images and in real time using a web camera application. The results showed that the proposed model achieved an overall accuracy of 91.2% for seven emotional classes in the case of an image application. Image based emotion detection of the proposed model accuracy showed relatively higher results than the real-time emotion detection.

DeepFocus: A visual focus of attention detection framework using deep learning in multi-object scenarios

In recent years, recognizing the visual focus of attention (VFoA) has attracted much attention among computer vision experts due to its various Human–Computer Interaction (HCI) or Human-Robot Interaction (HRI) applications. Although eye gaze is a potential cue to determine someone’s focus of attention (FOA), it is challenging to determine FOA alone when the interacting partners are far away or the camera cannot capture high-resolution images from long-distance. Therefore, the head pose can be used as an approximation to recognize the focus of someone’s attention. This paper proposes a vision-based framework to detect the FOA of humans using nine head poses consisting of four main modules: face detection and facial key-point selection (FDKPSM), head pose classification (HPCM), object localization and classification (OLCM), and focus of attention estimation (FoAEM). The FDKPSM uses the Multi-task Cascaded Neural Network (MTCNN) framework to detect head poses, and the HPCM classifies them into nine classes using the ResNet18. To estimate the FoA, the FoAEM uses a mapping Algorithm (EFoA) which integrates head poses on the focused object. Experimental results show that the proposed model outperformed other deep learning models by achieving the highest accuracy on three datasets: BIWI-M (96.97%), Pointing’04-M (96.04%) and HPoD 9 (98.99%). The visual focus of the attention model gained an accuracy of 94.12% in the multi-object scenario.