Facial Landmarks Research Articles

Automated landmark-based cat facial analysis and its applications

Facial landmarks, widely studied in human affective computing, are beginning to gain interest in the animal domain. Specifically, landmark-based geometric morphometric methods have been used to objectively assess facial expressions in cats, focusing on pain recognition and the impact of breed-specific morphology on facial signaling. These methods employed a 48-landmark scheme grounded in cat facial anatomy. Manually annotating these landmarks, however, is a labor-intensive process, deeming it impractical for generating sufficiently large amounts of data for machine learning purposes and for use in applied real-time contexts with cats. Our previous work introduced an AI pipeline for automated landmark detection, which showed good performance in standard machine learning metrics. Nonetheless, the effectiveness of fully automated, end-to-end landmark-based systems for practical cat facial analysis tasks remained underexplored. In this paper we develop AI pipelines for three benchmark tasks using two previously collected datasets of cat faces. The tasks include automated cat breed recognition, cephalic type recognition and pain recognition. Our fully automated end-to-end pipelines reached accuracy of 75% and 66% in cephalic type and pain recognition respectively, suggesting that landmark-based approaches hold promise for automated pain assessment and morphological explorations.

Cross-dataset micro-expression identification based on facial ROIs contribution quantification

Abstract In this work, we propose a novel cross-dataset micro-expression identification method based on facial regions of interest contribution quantification, where the training samples are from the source dataset and the test samples are from the target dataset. Specifically, a micro-expression video clip (i.e., a sample) is first sampled at intervals to capture an image sequence consisting of multiple consecutive video frames. Then, locate and crop the facial area in the image frame, and perform facial landmark detection on the cropped area. Next calculate the optical-flow field of the image sequence, and extract the improved Main Directional Mean Optical-flow feature. Finally, the learned group sparse model is used as the classifier to predict the label information of the unmarked target samples, so as to complete the category recognition of micro-expressions. Extensive cross-dataset comparative experiments on four spontaneous micro-expression datasets, CASME, CASME II, SMIC-HS, and SAMM, show that the recognition strategy proposed in this work is effective. Compared with several other advanced recognition methods, our method has better cross-dataset recognition performance, not only the classification accuracy is higher, but also the classification stability is better when faced with different target datasets and different micro-expression categories. It can be applied in many fields such as clinical diagnosis, interrogation, negotiation, and national security.

A Research on Two-Stage Facial Occlusion Recognition Algorithm based on CNN

In recent years, pattern recognition has garnered widespread attention, especially in the domain of face recognition. Traditional face recognition methods have certain limitations in unconstrained environments due to factors such as lighting, facial expressions, and poses. Deep learning can be used to address these challenges. This paper proposes a comprehensive approach to face occlusion recognition based on a two-stage Convolutional Neural Network (CNN). Face verification aims at verifying whether two face images belong to the same individual, and it is a more fundamental task compared to face recognition. The process of face recognition essentially involves multiple instances of face verification, sequentially validating different individuals to ultimately determine the corresponding individual for each face. The primary steps in this research include facial detection, image preprocessing, facial landmark localization, facial landmark extraction, feature matching recognition, and 2D image-assisted 3D face reconstruction. A novel two-stage CNN was designed for facial detection and alignment. The first stage of the network is dedicated to the search for facial windows and regressing vector boundaries. The second stage utilizes 2D images to assist in 3D face reconstruction and perform secondary recognition for cases not identified in the first stage. This method demonstrated excellent performance in handling facial occlusions, achieving high accuracy on datasets such as AFW and FDDB. On the test dataset, face recognition accuracy reached 97.3%, surpassing the original network accuracy of 89.1%. This method outperforms traditional algorithms and general CNN approaches. This study achieved efficient face validation and further handling of unrecognized situations, contributing to the enhancement of face recognition system performance.

Feature Weighted Cycle Generative Adversarial Network with Facial Landmark Recognition and Perceptual Color Distance for Enhanced Face Animation Generation

We propose an anime style transfer model to generate anime faces from human face images. We improve the model by modifying the normalization function to obtain more feature information. To make the face feature position of the anime face similar to the human face, we propose facial landmark loss to calculate the error between the generated image and the real human face image. To avoid obvious color deviation in the generated images, we introduced perceptual color loss into the loss function. In addition, due to the lack of reasonable metrics to evaluate the quality of the animated images, we propose the use of Fréchet anime inception distance to calculate the distance between the distribution of the generated animated images and the real animated images in high-dimensional space, so as to understand the quality of the generated animated images. In the user survey, up to 74.46% of users think that the image produced by the proposed method is the best compared with other models. Also, the proposed method reaches a score of 126.05 for Fréchet anime inception distance. Our model performs the best in both user studies and FAID, showing that we have achieved better performance in human visual perception and model distribution. According to the experimental results and user feedback, our proposed method can generate results with better quality compared to existing methods.

Facial Feature Localization and Analysis Using Machine Learning

The facial feature localization system with machine learning (ML) is an innovative solution designed to accurately detect and locate key facial landmarks. By integrating advanced ML algorithms, the system processes images or video streams in real time to identify the positions of facial features such as the eyes, nose, mouth, and jawline. The ML component analyses the data to precisely locate these features, enabling applications in areas like facial expression analysis and biometric verification. This approach enhances accuracy by adapting to various lighting conditions, face orientations, and individual facial variations. The system’s scalability and flexibility further extend its utility, supporting diverse applications across industries like healthcare, security, and human-computer interaction. Overall, this project represents a significant advancement in facial feature localization, using IoT and ML to enable more effective and accurate facial analysis. By adapting to various lighting conditions, face orientations, and individual facial variations" to "by adapting to diverse lighting conditions, different face orientations, and individual facial variations, ensuring reliable performance in various environments.

Advanced AI-Driven Emotion Recognition Systems: Applications and Technical Challenges

Focusing on face and landmark analysis, the technical analysis investigates AI-based emotion identification systems' design, application, and difficulties. This thorough article explores the use of these systems in the manufacturing, scientific research, and healthcare industries, emphasizing both creative solutions and technological challenges. The article explores fundamental elements such as real-time emotion analysis frameworks, facial landmark identification systems, convolutional neural networks, and deep learning techniques. It shows how these systems are transforming human-computer interaction while addressing important issues in data protection and ethical implementation through a thorough examination of technical aspects, domain-specific applications, difficulties, and privacy considerations. To shed light on how emotion recognition technology is developing, the study also looks at potential future development paths, such as improvements in scalability, cross-platform integration, and sensor fusion technologies.

Patch-based convolutional neural networks for automatic landmark detection of 3D facial images in clinical settings.

The facial landmark annotation of 3D facial images is crucial in clinical orthodontics and orthognathic surgeries for accurate diagnosis and treatment planning. While manual landmarking has traditionally been the gold standard, it is labour-intensive and prone to variability. This study presents a framework for automated landmark detection in 3D facial images within a clinical context, using convolutional neural networks (CNNs), and it assesses its accuracy in comparison to that of ground-truth data. Initially, an in-house dataset of 408 3D facial images, each annotated with 37 landmarks by an expert, was constructed. Subsequently, a 2.5D patch-based CNN architecture was trained using this dataset to detect the same set of landmarks automatically. The developed CNN model demonstrated high accuracy, with an overall mean localization error of 0.83 ± 0.49mm. The majority of the landmarks had low localization errors, with 95% exhibiting a mean error of less than 1mm across all axes. Moreover, the method achieved a high success detection rate, with 88% of detections having an error below 1.5mm and 94% below 2mm. The automated method used in this study demonstrated accuracy comparable to that achieved with manual annotations within clinical settings. In addition, the proposed framework for automatic landmark localization exhibited improved accuracy over existing models in the literature. Despite these advancements, it is important to acknowledge the limitations of this research, such as that it was based on a single-centre study and a single annotator. Future work should address computational time challenges to achieve further enhancements. This approach has significant potential to improve the efficiency and accuracy of orthodontic and orthognathic procedures.

Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave Radar

Facial Landmark Detection Based on High Precision Spatial Sampling via Millimeter-wave Radar

Internet of things assisted deep learning enabled driver drowsiness monitoring and alert system using CNN-LSTM framework

Abstract Driver fatigue has been generally viewed as a critical road safety factor and has been cited for contributing to a good percentage of traffic accidents. Developing systems to monitor and alert drowsy drivers is essential in reducing incidents. This research proposes a robust framework for driver drowsiness detection using a CNN-LSTM architecture developed by fusing facial landmark analysis with multiple aspect ratios. It uses, as key metrics in detecting drowsiness, the Eye Aspect Ratio (EAR), Pupil Circularity (PUC), the Mouth Aspect Ratio (MAR), and the Mouth over Eye Aspect Ratio (MOE). CNN-LSTM had been trained on YawDD, NITYMD, FL3D, and custom datasets. Data augmentation techniques such as flipping, scaling, shearing, rotation, brightness and contrast adjustment are used to generalize under different illumination conditions and driver postures. The system is implemented on NVIDIA’s 128-core Jetson Nano GPU platform and does real-time processing of video frames captured by a CSI camera. It detects eye closure and yawning as symptoms of driver fatigue and immediately raises an alert through vibrations in the seatbelt and pre-recorded voice messages. The Internet connectivity allows remote monitoring via mobile applications, making it safer by ensuring that alerts reach the driver and passengers. This CNN-LSTM model has been carefully tested for various scenarios, including day and night conditions, proving its effectiveness. The proposed framework indicated excellent performance concerning accuracy (98%), precision (95%), recall (93%), F1 score (94%), and AUC (99%), thereby revealing its efficiency in practical scenarios. By incorporating EAR, MAR, PUC, and MOE for the early detection of drowsiness in this CNN-LSTM architecture, the system will be able to alert the driver ahead of time so that necessary precautions can be taken to avoid an accident. The proposed approach enhances the driver's safety and is a scalable solution that adapts to different environments and populations.

Automated video-based pain recognition in cats using facial landmarks

Affective states are reflected in the facial expressions of all mammals. Facial behaviors linked to pain have attracted most of the attention so far in non-human animals, leading to the development of numerous instruments for evaluating pain through facial expressions for various animal species. Nevertheless, manual facial expression analysis is susceptible to subjectivity and bias, is labor-intensive and often necessitates specialized expertise and training. This challenge has spurred a growing body of research into automated pain recognition, which has been explored for multiple species, including cats. In our previous studies, we have presented and studied artificial intelligence (AI) pipelines for automated pain recognition in cats using 48 facial landmarks grounded in cats’ facial musculature, as well as an automated detector of these landmarks. However, so far automated recognition of pain in cats used solely static information obtained from hand-picked single images of good quality. This study takes a significant step forward in fully automated pain detection applications by presenting an end-to-end AI pipeline that requires no manual efforts in the selection of suitable images or their landmark annotation. By working with video rather than still images, this new pipeline approach also optimises the temporal dimension of visual information capture in a way that is not practical to preform manually. The presented pipeline reaches over 70% and 66% accuracy respectively in two different cat pain datasets, outperforming previous automated landmark-based approaches using single frames under similar conditions, indicating that dynamics matter in cat pain recognition. We further define metrics for measuring different dimensions of deficiencies in datasets with animal pain faces, and investigate their impact on the performance of the presented pain recognition AI pipeline.

Computational investigation of the social function of domestic cat facial signals

There is growing interest in the facial signals of domestic cats. Domestication may have shifted feline social dynamics towards a greater emphasis on facial signals that promote affiliative bonding. Most studies have focused on cat facial signals during human interactions or in response to pain. Research on intraspecific facial communication in cats has predominantly examined non-affiliative social interactions. A recent study by Scott and Florkiewicz1 demonstrated significant differences between cats’ facial signals during affiliative and non-affiliative intraspecific interactions. This follow-up study applies computational approaches to make two main contributions. First, we develop a machine learning classifier for affiliative/non-affiliative interactions based on manual CatFACS codings and automatically detected facial landmarks, reaching above 77% in CatFACS codings and 68% in landmarks by integrating a temporal dimension. Secondly, we introduce novel measures for rapid facial mimicry based on CatFACS coding. Our analysis suggests that domestic cats exhibit more rapid facial mimicry in affiliative contexts than non-affiliative ones, which is consistent with the proposed function of mimicry. Moreover, we found that ear movements (such as EAD103 and EAD104) are highly prone to rapid facial mimicry. Our research introduces new possibilities for analyzing cat facial signals and exploring shared moods with innovative AI-based approaches.

Using facial reaction analysis and machine learning to objectively assess the taste of medicines in children.

For orally administered drugs, palatability is key in ensuring patient acceptability and treatment compliance. Therefore, understanding children's taste sensitivity and preferences can support formulators in making paediatric medicines more acceptable. Presently, we explore if the application of computer-vision techniques to videos of children's reaction to gustatory taste strips can provide an objective assessment of palatability. Children aged 4 to 11 years old tasted four different flavoured strips: no taste, bitter, sweet, and sour. Data was collected at home, under the supervision of a guardian, with responses recorded using the Aparito Atom app and smartphone camera. Participants scored each strip on a 5-point hedonic scale. Facial landmarks were identified in the videos, and quantitative measures, such as changes around the eyes, nose, and mouth, were extracted to train models to classify strip taste and score. We received 197 videos and 256 self-reported scores from 64 participants. The hedonic scale elicited expected results: children like sweetness, dislike bitterness and have varying opinions for sourness. The findings revealed the complexity and variability of facial reactions and highlighted specific measures, such as eyebrow and mouth corner elevations, as significant indicators of palatability. This study capturing children's objective reactions to taste sensations holds promise in identifying palatable drug formulations and assessing patient acceptability of paediatric medicines. Moreover, collecting data in the home setting allows for natural behaviour, with minimal burden for participants.

Eye Mapping: Innovative Technique for Precise Iris Positioning in Prosthetic Eye.

The aim of this technique is to institute "Augmented Reality Tool" in the field of maxillofacial prosthesis for an accurate and precise iris positioning in a prosthetic eye. A congenital defect, irreversible damage, a painful blind eye, sympathetic ophthalmia, or the requirement for histological confirmation of a suspected illness can all result in the absence or loss of an eye. In such circumstances, meticulous preoperative, surgical, and prosthetic planning using a multidisciplinary approach is imperative for successful rehabilitation. Augmented reality filter was used to provide precise mapping of facial landmarks and to aid in iris positioning. Smartphone device (S22) was used and artificial intelligence (AI)-generated Instagram application was unfurled and augmented reality (AR) filter was instituted. The filter available in the application made it possible to see and confirm the iris shell's location in three dimensions from various angles. The dimensions thus provided, aided in the correct positioning of the iris in the prosthetic eye. Iris positioning stands as one of the crucial steps in fabricating a patient-specific ocular prosthesis. In the present case report, iris positioning was done using AI has provided an excellent esthetics results and patient compliance was met with satisfaction. Accurate positioning of iris can be done using AR tool which is popular, easily accessible, less technique sensitive, and can be performed with slightest efforts in small clinical set-ups. Relating to patient, it is comfortable, economic, and trouble-free procedure. How to cite this article: Zilpilwar NR, Nimonkar SV, Chitlange P, et al. Eye Mapping: Innovative Technique for Precise Iris Positioning in Prosthetic Eye. J Contemp Dent Pract 2024;25(7):711-714.

Mapping the localization of the facial artery and vein at the inferior border of the mandible for clinical applications.

To spatially map the position of the facial artery and vein along the inferior border of the mandible based on specific facial reference points. This cross-sectional study included 40 sides of 21 formalin-fixed cadaver heads (11 female, 10 male) with a mean age of 72.92 ± 11.72years. Superficial dissection was performed to visualize the facial artery and vein at the inferior border of the mandible (FA and FV). Measurements were taken from the FA and FV to various facial landmarks using a digital caliper and medical protractor. Statistical analysis was conducted using Shapiro-Wilk, independent sample t-tests, and paired sample t-tests. The mean distance between FA and FV was 6.20 ± 2.97mm. Distances from FA and FV to gnathion, oral commissure, nasal wing, lateral canthus, angle of mandible, intertragic notch, and Manson's point were determined, with no significant differences found between sides and genders (p > 0.05). The closest distances from FA and FV to the line between the intertragic notch and oral commissure were 38.63 ± 4.86mm and 37.78 ± 5.28mm, respectively. The angular measurements of FA and FV with inferior border of mandible were 61.45 ± 13.71 and 76.56 ± 10.17 degrees, respectively. The angle between intertragic notch, FA and oral commisure was 102.40 ± 11.12 degrees. The detailed measurements and analysis provided in this study aim to enhance the precision of surgical interventions involving the facial artery and vein. The practical localization methods proposed can assist in minimizing the risk of vascular injuries, improving outcomes in reconstructive and aesthetic procedures.

Quantitative analysis of facial shape in children to support respirator design

The COVID-19 pandemic demonstrated the need for respiratory protection against airborne pathogens. Respirator options for children are limited, and existing designs do not consider differences in facial shape or size. We created a dataset of children's facial images from three cohorts, then used geometric morphometric analyses of dense and sparse facial landmark representations to quantify age, sex and ancestry-related variation in shape. We found facial shape and size in children vary significantly with age from ages 2 to 18, particularly in dimensions relevant to respirator design. Sex differences are small throughout most of the age range of our sample. Ancestry is associated with significant facial shape variation in dimensions that may affect respirator fit. We offer guidance on how to our results can be used for the appropriate design of devices such as respirators for pediatric populations. We also highlight the need to consider ancestry-related variation in facial morphology to promote equitable, inclusive products.

A Two-Stream Method for Human Action Recognition Using Facial Action Cues.

Human action recognition (HAR) is a critical area in computer vision with wide-ranging applications, including video surveillance, healthcare monitoring, and abnormal behavior detection. Current HAR methods predominantly rely on full-body data, which can limit their effectiveness in real-world scenarios where occlusion is common. In such situations, the face often remains visible, providing valuable cues for action recognition. This paper introduces Face in Action (FIA), a novel two-stream method that leverages facial action cues for robust action recognition under conditions of significant occlusion. FIA consists of an RGB stream and a landmark stream. The RGB stream processes facial image sequences using a fine-spatio-multitemporal (FSM) 3D convolution module, which employs smaller spatial receptive fields to capture detailed local facial movements and larger temporal receptive fields to model broader temporal dynamics. The landmark stream processes facial landmark sequences using a normalized temporal attention (NTA) module within an NTA-GCN block, enhancing the detection of key facial frames and improving overall recognition accuracy. We validate the effectiveness of FIA using the NTU RGB+D and NTU RGB+D 120 datasets, focusing on action categories related to medical conditions. Our experiments demonstrate that FIA significantly outperforms existing methods in scenarios with extensive occlusion, highlighting its potential for practical applications in surveillance and healthcare settings.

A comprehensive design for biomimetic behavior of robotic head

In robotics and human-robot interaction, the ability to express emotions and respond accurately to human emotions is essential. An important aspect of this ability involves controlling the robot’s facial actuators in a way that resonates with human emotions. This study systematically presented the construction of a robot head based on the theoretical foundation of human sustainability. Facial landmarks are extracted for various purposes, such as to determine anthropometric metrics for robot head design, a comprehensive design for the robotic head is proposed in this study including anthropometric measurements used as a theoretical basis for the design, the calculations, and design, control system, electricity, and robot skin are presented. The robotic head with 24 degrees of freedom (DoF) aims to express all human emotions, signals are fed back from strain gauge sensors and cameras to calibrate the response of the robot’s prosthetic muscles. Strain gauge sensors are set up at locations with large deformations to double-check the functioning of prosthetic muscles. The two experiments are set up to evaluate the completeness of emotions, the results showed that both experiments achieved 93.88% and 91.85% respectively.

A lightweight network-based sign language robot with facial mirroring and speech system

Human–robot interaction is an essential capability for humanoid robots to enter the physical world and become companions in people’s lives, learning, and work. While the majority of current research focuses on the voice-based interactions of robots, yet over 60% of communication occurs through nonverbal behaviors, such as facial expressions and hand gestures. Endowing robots with the ability to communicate through nonverbal behavior not only enhances the interactive experience with robots but also provides a potential communication tool for individuals with hearing or speech impairments. Here, we develop a humanoid robot capable of adjusting facial movements by driving servos, and design a novel framework for the robot to integrate sign language recognition and facial landmark detection algorithms. This framework facilitates the robot recognize sign language and translate it into spoken language, while also imitating the facial expressions of the signers. To achieve this, we also propose a lightweight deep learning network called RealTimeSignNet for real-time sign language recognition. Leveraging lightweight 3D convolution modules and time-dependent constraints, this model adapts to various time scales, ensuring efficient processing of sign language recognition tasks. Experimental results demonstrate the outstanding performance of the RealTimeSignNet model on mainstream sign language datasets, achieving an accuracy of 88.1% on the large continuous sign language dataset (continuous SLR), 98.2% on the isolated sign language dataset (SLR 500), and 91.50% on the English sign language dataset (WLAS). The overall assessment demonstrates that our humanoid robot is capable of recognizing sign language and translating it into spoken language, while imitating the facial emotions, providing a comprehensive solution to the communication challenges faced by individuals with hearing and speech impairments.

FTM: The Face Truth Machine—Hand-crafted features from micro-expressions to support lie detection

This work deals with the delicate task of lie detection from facial dynamics. The proposed Face Truth Machine (FTM) is an intelligent system able to support a human operator without any special equipment. It can be embedded in the present infrastructures for forensic investigation or whenever it is required to assess the trustworthiness of responses during an interview. Due to its flexibility and its non-invasiveness, it can overcome some limitations of present solutions. Of course, privacy issues may arise from the use of such systems, as often underlined nowadays. However, it is up to the utilizer to take these into account and make fair use of tools of this kind. The paper will discuss particular aspects of the dynamic analysis of face landmarks to detect lies. In particular, it will delve into the behavior of the features used for detection and how these influence the system’s final decision. The novel detection system underlying the Face Truth Machine is able to analyze the subject’s expressions in a wide range of poses. The results of the experiments presented testify to the potential of the proposed approach and also highlight the very good results obtained in cross-dataset testing, which usually represents a challenge for other approaches.

Investigating Digital Illiterate Classification Techniques Based on DeepFace Technology

This paper presents an algorithm to identify digital illiterates by analyzing age and emotions from facial recognition. In this paper, digital illiterates refer to people who struggle in using digital devices. The study assumed that older individuals who display surprised or angry facial expressions while using digital devices are more likely to be identified as digitally illiterate. For age detection, the study used MTCNN (Multi-task Cascaded Convolutional Networks), and for emotion detection, it employed the VGG-Face model. MTCNN detects facial features and landmarks to preprocess images and distinguish facial characteristics. The VGG-Face model uses convolution operations to analyze facial images and classify emotional states. The dataset used in the study consisted of 3,000 facial images collected from the internet. The research team categorized the images into faces of individuals aged over 50, angry expressions, and surprised expressions. The dataset included 411 individuals (13.7%) aged over 50, 163 individuals (5.4%) with angry expressions, and 145 individuals (4.8%) with surprised expressions. Accuracy was estimated by comparing the results from the DeepFace algorithm with those from the research team’s classifications. The DeepFace algorithm achieved 95.77% accuracy in detecting individuals aged over 50, 83.45% accuracy for surprise, and 76.07% for anger. The results demonstrate that it is possible to identify digital illiterates based on their age and emotional expressions and could enable the development of personalized services to directly or indirectly support digital illiterates, potentially improving digital accessibility.