Facial Landmarks Research Articles

Determining emotions from students' facial expressions using CNN

Recognizing and understanding human emotions, particularly in educational settings, is of great importance. This research focuses on utilizing Convolutional Neural Networks (CNNs) to accurately identify students' emotions based on their facial expressions. By leveraging facial cues, an automated system can be developed to effectively recognize and interpret emotions in educational contexts. A diverse dataset of facial images featuring students expressing various emotions is carefully curated for this study. Facial landmarks and action units are extracted to capture essential information from different facial regions. These images are meticulously annotated with ground truth labels, ensuring precise training and evaluation of the CNN model. CNNs are chosen as the core technology for feature extraction and emotion classification due to their ability to learn intricate spatial patterns and hierarchical representations. Extensive training, including techniques like data augmentation and transfer learning, enables the model to generalize and adapt to a wide range of emotional expressions. The performance of the CNN model is evaluated using metrics such as accuracy, precision, recall, and F1 score. Thorough experiments compare the proposed CNN approach with existing methods for facial emotion recognition, demonstrating the superior performance of the CNN model in accurately identifying students' emotions from facial expressions.

한글 모음 및 표정 변화에 따른 안면 분석 시스템

Face deformation is caused by various causes such as facial paralysis or asymmetry. Therefore, accurate cause analysis and diagnosis are required, but CT, MRI, and Nottingham Grading System are subjective in nature and are affected by clinician proficiency and fatigue. In order to reduce the time and cost burden of patients who have to continuously visit specialized institutions for treatment, this study aims to develop a facial analysis system according to changes in Korean vowels and facial expressions. First, the coordinate value of the face landmark was extracted based on the webcam image, and then the distance value between the left and right face landmarks was calculated. In addition, quantification was performed based on the actual length of the eyes and lips and the results of the system derivation, and the final system analysis results could be checked with figures and charts.

Early Identify and Notify Driver’s Drowsiness by Machine Learning

Abstract: Drowsiness and intoxication are major contributors to car accidents, posing significant risks to road safety. The implementation of effective drowsiness detection technologies could help prevent numerous fatal accidents by alerting fatigued drivers in advance. Various techniques can be adopted to monitor driver attentiveness while driving and provide timely notifications. In the context of self-driving cars, sensors play a crucial role in identifying signs of sleepiness, anger, or extreme emotional changes in drivers. These sensors continuously monitor facial expressions and detect facial landmarks to assess the driver's state and ensure safe driving. Once such changes are detected, the system promptly assumes control of the vehicle, reducing its speed, and alerts the driver through alarms to draw attention to the situation. To enhance accuracy, the proposed system integrates with the vehicle's electronics, tracking its statistics and providing precise results. In this research, we have implemented real-time image segmentation and drowsiness detection using machine learning methodologies. Specifically, an emotion detection method based on Support Vector Machines (SVM) has been employed, utilizing facial expressions. The algorithm underwent testing under varying luminance conditions and exhibited superior performance compared to existing research, achieving an 83.25% accuracy rate in detecting facial expression changes.

An optimized deep convolutional neural network for adaptive learning using feature fusion in multimodal data

The outbreak of COVID-19 has caused an unprecedented increase in the usage of e-Learning platforms. The closure of educational institutions globally has significantly impacted the traditional education system. Without physical interaction, engaging students in an e-Learning environment has become a major challenge for teachers and e-Learning platform providers. Unlike a face-to-face classroom, where teachers can easily monitor students’ behaviour and adapt the learning content according to their needs, this is impossible in an e-Learning environment. This study introduces an optimized deep learning-based approach for detecting student engagement levels to ensure students remain connected to the learning process. The proposed optimized model uses facial emotion recognition and head movement detection to track real-time engagement levels for big data as real-time-based face and head datasets are analysed. The system leverages facial landmark detection to monitor head movements and deep learning models such as VGG19 and ResNet50 for facial emotion recognition. The system combines the output of both approaches to accurately predict the student engagement state as either ‘engaged’ or ‘disengaged’ with an accuracy rate of 91.67%.

FreeEnricher: Enriching Face Landmarks without Additional Cost

Recent years have witnessed significant growth of face alignment. Though dense facial landmark is highly demanded in various scenarios, e.g., cosmetic medicine and facial beautification, most works only consider sparse face alignment. To address this problem, we present a framework that can enrich landmark density by existing sparse landmark datasets, e.g., 300W with 68 points and WFLW with 98 points. Firstly, we observe that the local patches along each semantic contour are highly similar in appearance. Then, we propose a weakly-supervised idea of learning the refinement ability on original sparse landmarks and adapting this ability to enriched dense landmarks. Meanwhile, several operators are devised and organized together to implement the idea. Finally, the trained model is applied as a plug-and-play module to the existing face alignment networks. To evaluate our method, we manually label the dense landmarks on 300W testset. Our method yields state-of-the-art accuracy not only in newly-constructed dense 300W testset but also in the original sparse 300W and WFLW testsets without additional cost.

JR2Net: Joint Monocular 3D Face Reconstruction and Reenactment

Face reenactment and reconstruction benefit various applications in self-media, VR, etc. Recent face reenactment methods use 2D facial landmarks to implicitly retarget facial expressions and poses from driving videos to source images, while they suffer from pose and expression preservation issues for cross-identity scenarios, i.e., when the source and the driving subjects are different. Current self-supervised face reconstruction methods also demonstrate impressive results. However, these methods do not handle large expressions well, since their training data lacks samples of large expressions, and 2D facial attributes are inaccurate on such samples. To mitigate the above problems, we propose to explore the inner connection between the two tasks, i.e., using face reconstruction to provide sufficient 3D information for reenactment, and synthesizing videos paired with captured face model parameters through face reenactment to enhance the expression module of face reconstruction. In particular, we propose a novel cascade framework named JR2Net for Joint Face Reconstruction and Reenactment, which begins with the training of a coarse reconstruction network, followed by a 3D-aware face reenactment network based on the coarse reconstruction results. In the end, we train an expression tracking network based on our synthesized videos composed by image-face model parameter pairs. Such an expression tracking network can further enhance the coarse face reconstruction. Extensive experiments show that our JR2Net outperforms the state-of-the-art methods on several face reconstruction and reenactment benchmarks.

Designing Effective Visual Feedback for Facial Rehabilitation Exercises: Investigating the Role of Shape, Transparency, and Age on User Experience.

Facial expression recognition technology has been utilized both for entertainment purposes and as a valuable aid in rehabilitation and facial exercise assistance. This technology leverages artificial intelligence models to predict facial landmark points and provide visual feedback, thereby facilitating users' facial movements. However, feedback designs that disregard user preferences may cause discomfort and diminish the benefits of exercise. This study aimed to develop a feedback design guide for facial rehabilitation exercises by investigating user responses to various feedback design methods. We created a facial recognition mobile application and designed six feedback variations based on shape and transparency. To evaluate user experience, we conducted a usability test involving 48 participants (24 subjects in their 20s and 24 over 60 years of age), assessing factors such as feedback, assistance, disturbance, aesthetics, cognitive ease, and appropriateness. The experimental results revealed significant differences in transparency, age, and the interaction between transparency and age. Consequently, it is essential to consider both transparency and user age when designing facial recognition feedback. The findings of this study could potentially inform the design of more effective and personalized visual feedback for facial motion, ultimately benefiting users in rehabilitation and exercise contexts.

Risky Driver Identification Using Beta Regression Based on Naturalistic Driving Data

Naturalistic driving data are widely used to investigate factors related to road safety. Crashes and near-crashes can be regarded as the critical events on the road. The existing studies typically modeled crash and near-crash events at the trip level. However, individual drivers may have different risk levels, and other factors such as distraction can also play a role. This study uses variables automatically derived from naturalistic driving data. Driver distraction is detected from videos using facial landmarks. Based on the collected variables, a beta regression model is developed to identify the significant variables affecting drivers’ risk levels. It is found that the average acceleration rate, number of hard accelerations, driver distraction, and age are significant variables. The findings from this study can be used to identify risky drivers and improve the design of automated vehicles by eliminating human errors and risky driving patterns. Moreover, advanced driver assistance systems (ADAS) can be promoted to alert drivers to risky driving behaviors. The proposed model is also easy to implement in real driving conditions as most of the variables can be extracted automatically. Relevant agencies can also use the model to identify risky drivers and provide proactive customized education programs.

IoT-based Eyes Closure Alert System for Drowsy Driver

Internet of Things (IoTs) is a technology that links objects and mechanisms to the Internet. IoT devices are deployed in a number of areas within the transportation sector, notably in security and surveillance. The ability to track driving behaviour on the road using IoT can improve driver safety and reduce traffic accidents. In this paper, a prototype system to detect a driver’s drowsiness is proposed. In particular, a facial landmark is used to extract the eye region, the eye aspect ratio, and closure patterns. This system will audibly alert the driver and invoke third party notification if drowsiness of the driver is detected. Through this system, traffic accidents caused by drowsy drivers can be avoided and driving behaviour can be evaluated by relevant personnel or employers.

Facial emotion recognition using geometrical features based deep learning techniques

In recent years, intelligent emotion recognition is active research in computer vision to understand the dynamic communication between machines and humans. As a result, automatic emotion recognition allows the machine to assess and acquire the human emotional state to predict the intents based on the facial expression. Researchers mainly focus on speech features and body motions; identifying affect from facial expressions remains a less explored topic. Hence, this paper proposes a novel approach for intelligent facial emotion recognition using optimal geometrical features from facial landmarks using VGG-19s (FCNN). Here, we utilize Haarcascade to detect the subject face and determine the distance and angle measurements. The entire process is to classify the facial expressions based on extracting relevant features with the normalized angle and distance measures. The experimental analysis shows high accuracy on the MUG dataset of 94.22% and 86.45% on GEMEP datasets, respectively.

The relationship between biological parameters and facial soft tissue thickness measured by ultrasound and its forensic implications.

Facial soft tissue thickness (FSTT) data are currently widely used in forensic and medical science. In the forensic sciences, they form the basis for craniofacial reconstruction and identification methods. Since there are few FSTT data in the Slovak population, this study aims to enrich the data in well-defined age categories, taking into account differences between sexes and body mass index (BMI). The sample consisted of 127 participants aged 17 to 86 years from Slovakia. In addition to biological sex and age information, stature and body weight were recorded to calculate BMI. Subsequently, 17 facial anthropometric landmarks were used to measure FSTT using a noninvasive General Electric LOGIQe R7 ultrasound device. The mean values of FSTT were greater in the mouth region in males and in the zygomatic and eye regions in females. Differences between males and females, regardless of sex and BMI, were significant only at two landmarks. When BMI and age were taken into account, there were differences in 12 of 17 landmarks. Linear regression results showed the strongest correlation of most landmarks with BMI, followed by age and sex. When the FSTT was estimated in association with sex/age/BMI, landmarks in the zygomatic, mandibular, and frontal regions were the best regressors. The results of the present study demonstrate that B-mode ultrasound measurements of FSTT can be used in facial reconstruction as a function of BMI, age, and sex of the subject. Furthermore, the present regression equations can help practitioners in the forensic/medical field to calculate individual tissue thickness.

The Detection of Students' Abnormal Behavior in Online Exams Using Facial Landmarks in Conjunction with the YOLOv5 Models

The popularity of massive open online courses (MOOCs) and other forms of distance learning has increased recently. Schools and institutions are going online to serve their students better. Exam integrity depends on the effectiveness of proctoring remote online exams. Proctoring services powered by computer vision and artificial intelligence have also gained popularity. Such systems should employ methods to guarantee an impartial examination. This research demonstrates how to create a multi-model computer vision system to identify and prevent abnormal student behaviour during exams. The system uses You only look once (YOLO) models and Dlib facial landmarks to recognize faces, objects, eye, hand, and mouth opening movement, gaze sideways, and use a mobile phone. Our approach offered a model that analyzes student behaviour using a deep neural network model learned from our newly produced dataset" StudentBehavioralDS." On the generated dataset, the "Behavioral Detection Model" had a mean Average Precision (mAP) of 0.87, while the "Mouth Opening Detection Model" and "Person and Objects Detection Model" had accuracies of 0.95 and 0.96, respectively. This work demonstrates good detection accuracy. We conclude that using computer vision and deep learning models trained on a private dataset, our idea provides a range of techniques to spot odd student behaviour during online tests.

AI-Powered Road Safety: Detecting Driver Fatigue through Visual Cues

Driving when fatigued is among the main causes of road deaths. Consequently, one ongoing research area is how to recognize driver fatigue and how to determine whether it is present. A large percentage of conventional methods are either based on machines, the behavior of people, or physiological processes. Some solutions need expensive sensors and data processing, while others are infiltrating and uncomfortable to the driver. As a consequence, this study creates an accurate, real-time method for identifying driver fatigue. The footage is captured by a camera, and image processing techniques are employed to recognize the driver's face in each frame. When facial landmarks on the detected face are pointed, the eye aspect ratio and mouth opening ratio are computed based on their values, and drowsiness is recognized utilizing generated adaptive thresholding. The following stage involves determining whether or not a discovered item is a face using SVM. It also checks the driver's eye aspect ratio (EAR) and mouth opening ratio (MOR) up to a predetermined number of times to look for signs of sleepiness and yawning. If sleepiness is identified, a warning email is sent to the registered email address.

Deep Learning for the Assessment of Facial Nerve Palsy: Opportunities and Challenges.

Automated evaluation of facial palsy using machine learning offers a promising solution to the limitations of current assessment methods, which can be time-consuming, labor-intensive, and subject to clinician bias. Deep learning-driven systems have the potential to rapidly triage patients with varying levels of palsy severity and accurately track recovery over time. However, developing a clinically usable tool faces several challenges, such as data quality, inherent biases in machine learning algorithms, and explainability of decision-making processes. The development of the eFACE scale and its associated software has improved clinician scoring of facial palsy. Additionally, Emotrics is a semiautomated tool that provides quantitative data of facial landmarks on patient photographs. The ideal artificial intelligence (AI)-enabled system would analyze patient videos in real time, extracting anatomic landmark data to quantify symmetry and movement, and estimate clinical eFACE scores. This would not replace clinician eFACE scoring but would offer a rapid automated estimate of both anatomic data, similar to Emotrics, and clinical severity, similar to the eFACE. This review explores the current state of facial palsy assessment, recent advancements in AI, and the opportunities and challenges in developing an AI-driven solution.

A Smartphone Application for Personalized Tooth Shade Determination.

Tooth shade determination methods for evaluating the effectiveness of whitening products at home are limited. In this study, an iPhone app for personalized tooth shade determination was developed. While capturing dental photographs in selfie mode before and after whitening, the app can maintain consistent illumination and tooth appearance conditions that affect tooth color measurement. An ambient light sensor was used to standardize the illumination conditions. To maintain consistent tooth appearance conditions determined by appropriately opening the mouth, facial landmark detection, an artificial intelligence technique that estimates key face parts and outlines, was used. The effectiveness of the app in ensuring uniform tooth appearance was investigated through color measurements of the upper incisors of seven participants via photographs captured in succession. The coefficients of variation for incisors L*, a*, and b* were less than 0.0256 (95% CI, 0.0173-0.0338), 0.2748 (0.1596-0.3899), and 0.1053 (0.0078-0.2028), respectively. To examine the feasibility of the app for tooth shade determination, gel whitening after pseudo-staining by coffee and grape juice was performed. Consequently, whitening results were evaluated by monitoring the ∆Eab color difference values (1.3 unit minimum). Although tooth shade determination remains a relative quantification method, the proposed method can support evidence-based selection of whitening products.

Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning models

Emotion estimation from face expression analysis is nowadays a widely-explored computer vision task. In turn, the classification of expressions relies on relevant facial features and their dynamics. Despite the promising accuracy results achieved in controlled and favorable conditions, the processing of faces acquired at a distance, entailing low-quality images, still suffers from a significant performance decrease. In particular, most approaches and related computational models become extremely unstable in the case of the very small amount of useful pixels that is typical in these conditions. Therefore, their behavior should be investigated more carefully. On the other hand, real-time emotion recognition at a distance may play a critical role in smart video surveillance, especially when controlling particular kinds of events, e.g., political meetings, to try to prevent adverse actions.This work compares facial expression recognition at a distance by: 1) a deep learning architecture based on state-of-the-art (SOTA) proposals, which exploits the whole images to autonomously learn the relevant embeddings; 2) a machine learning approach that relies on hand-crafted features, namely the facial landmarks preliminarily extracted using the popular Mediapipe framework. Instead of using either the complete sequence of frames or only the final still image of the expression, like current SOTA approaches, the two proposed methods are designed to use rich temporal information to identify three different stages of emotion. Expressions are time-split accordingly into four phases to better exploit their temporal-dependent dynamics. Experiments were conducted on the popular Extended Cohn-Kanade dataset (CK+). It was chosen for its wide use in related literature, and because it includes videos of facial expressions and not only still images. The results show that the approach relying on machine learning via hand-crafted features is more suitable for classifying the initial phases of the expression and does not decay in terms of accuracy when images are at a distance (only 0.08% of decay). On the contrary, deep learning not only has difficulties classifying the initial phases of the expressions but also suffers from relevant performance decay when considering images at a distance (52.68% accuracy decay).

Drowsy Driver Detection with Crash Alert Mechanism using Arduino and Image Processing

Driver drowsiness is a major cause of automobile accidents, resulting in many fatalities each year. The use of face detection techniques for identifying and warning fatigued drivers can solve this problem and improve transportation safety. This technique detects drowsiness using computer vision technologies based on facial landmarks. Image processing is used by the system to recognize the driver’s face, extract pictures of the eyes, and detect tiredness. The camera monitors the driver’s eyes in real-time, processing visual data to detect symptoms of tiredness. When sleepiness is detected, an alarm sounds to wake and notify the driver. Furthermore, the system is deployed with an Arduino UNO board with GSM, GPS, and MPU 6050 models to detect and alert emergency services in the case of a car accident, as well as to notify the driver’s family. Here the MPU- 6050 sensor detects angular position changes, while the NEO- 6m GPS module identifies the vehicle’s location. The SIM800L GSM module delivers an SMS including the location as well as a warning message. This system could be kept in automobiles to improve driver and passenger safety and security.

Semantic Learning for Facial Action Unit Detection

This article proposes semantic embedding for image transformers (SEiTs) to explore semantic features of facial morphology in the action unit (AU) detection task. The conventional approaches typically rely on external information (e.g., facial landmarks) to obtain the location of facial components, whereas the SEiT can learn morphological features intrinsically from the face image. The pre-training task, namely semantic masked facial image modeling (SMFIM), aims to actively obtain facial morphological information. The pixels of the input facial image are randomly erased with semantic masks (e.g., nose, eyes, eyebrows, mouth, and lip). The embedding model tries to predict the presence of facial components for the input image that can learn semantic representations of the face simultaneously. The learned semantic embeddings are fed to transformer blocks, which enable global interaction between semantic elements. The SEiT integrates facial morphological information and global interaction characters, appropriate for AU detection. The experiments are conducted on the Binghamton-Pittsburgh 4D (BP4D) dataset and Denver intensity of spontaneous facial action (DISFA) dataset, and the results demonstrate the effectiveness of the proposed SEiT.

EPA Consensus Project Paper: The Vertical Dimension of Occlusion. How to Determine and How to Alter? A Systematic Review.

The aim of this systematic review was to explore the dental literature to identify high quality clinical studies that introduced methods of determining the vertical dimension of occlusion (VDO), and additionally to find studies which assessed alterations in the VDO. An electronic search of the literature was conducted through PubMed , Embase, and Cochrane Library databases referring to the determination and alteration of the VDO by 12/2021. A total of 215 records were obtained from the initial search. After the first two screenings, 33 studies were selected for inclusion. Correlations in the morphometric group ranged between r=0.18-0.87, p⟨0.05-0.001, correlations in the cephalometric group ranged between r=0.28-0.92, p⟨0.05-0.001, and correlations in the mechanometric group ranged between r=0.21- 0.75, p⟨0.05-0.01. Regarding the alteration of VDO, in all studies the increase ranged between 1.8-8 mm and the patients adapted . No clear guidelines can be established yet, in relation to the determination and alteration of the VDO. There is no apparent benefit in using more invasive and complex methods compared to the use of the facial anatomical landmarks. Patient adaptation seems to be successful when the range of VDO increase was 1.8-6.0 mm.

FaceFormer: Aggregating Global and Local Representation for Face Hallucination

Recently, face hallucination methods either feed whole face image into convolutional neural networks (CNNs) or utilize extra facial priors (e.g., facial parsing maps and landmarks) to focus on global facial structure and constrain facial texture generation. However, the limited receptive fields of CNNs and inaccurate facial priors will reduce the naturalness and fidelity of restored face. In this paper, we propose a FaceFormer that aggregates global representation of Transformers and local representation of CNNs to maintain the consistency of facial structure while restoring local facial details. The reason for this design is that the Transformer can capture global facial information by exploiting the long-distance visual relation modeling, while the local modeling capability of CNNs can recover fine-grained facial details. Therefore, aggregating these two independent representations can help to maximize their merits and reconstruct high-quality and high-fidelity face images. Experimental results of face reconstruction and recognition verify that the proposed FaceFormer significantly outperforms current state-of-the-arts.