Facial Motion Analysis Research Articles

IoT-Assisted Automatic Driver Drowsiness Detection through Facial Movement Analysis Using Deep Learning and a U-Net-Based Architecture

The main purpose of a detection system is to ascertain the state of an individual’s eyes, whether they are open and alert or closed, and then alert them to their level of fatigue. As a result of this, they will refrain from approaching an accident site. In addition, it would be advantageous for people to be promptly alerted in real time before the occurrence of any calamitous events affecting multiple people. The implementation of Internet-of-Things (IoT) technology in driver action recognition has become imperative due to the ongoing advancements in Artificial Intelligence (AI) and deep learning (DL) within Advanced Driver Assistance Systems (ADAS), which are significantly transforming the driving encounter. This work presents a deep learning model that utilizes a CNN–Long Short-Term Memory network to detect driver sleepiness. We employ different algorithms on datasets such as EM-CNN, VGG-16, GoogLeNet, AlexNet, ResNet50, and CNN-LSTM. The aforementioned algorithms are used for classification, and it is evident that the CNN-LSTM algorithm exhibits superior accuracy compared to alternative deep learning algorithms. The model is provided with video clips of a certain period, and it distinguishes the clip by analyzing the sequence of motions exhibited by the driver in the video. The key objective of this work is to promote road safety by notifying drivers when they exhibit signs of drowsiness, minimizing the probability of accidents caused by fatigue-related disorders. It would help in developing an ADAS that is capable of detecting and addressing driver tiredness proactively. This work intends to limit the potential dangers associated with drowsy driving, hence promoting enhanced road safety and a decrease in accidents caused by fatigue-related variables. This work aims to achieve high efficacy while maintaining a non-intrusive nature. This work endeavors to offer a non-intrusive solution that may be seamlessly integrated into current automobiles, hence enhancing accessibility to a broader spectrum of drivers through the utilization of facial movement analysis employing CNN-LSTM and a U-Net-based architecture.

Guided Self-rehabilitation Contract vs conventional therapy in chronic peripheral facial paresis: VISAGE, a multicenter randomized controlled trial

BackgroundOne year after persistent peripheral facial paresis (PFP), prescriptions of conventional rehabilitation are often downgraded into maintenance rehabilitation or discontinued, the patient entering what is seen as a chronic stage. This therapeutic choice is not consistent with current knowledge about behavior-induced plasticity, which is available all life long and may allow intense sensorimotor rehabilitation to remain effective. This prospective, randomized, multicenter single-blind study in subjects with chronic unilateral PFP evaluates changes in facial motor function with a Guided Self-rehabilitation Contract (GSC) vs. conventional therapy alone, carried out for six months.MethodsEighty-two adult subjects with chronic unilateral PFP (> 1 year since facial nerve injury) will be included in four tertiary, maxillofacial surgery (2), otolaryngology (1) and rehabilitation (1) centers to be randomized into two rehabilitation groups. In the experimental group, the PM&R specialist will implement the GSC method, which for PFP involves intensive series of motor strengthening performed daily on three facial key muscle groups, i.e. Frontalis, Orbicularis oculi and Zygomatici. The GSC strategy involves: i) prescription of a daily self-rehabilitation program, ii) teaching of the techniques involved in the program, iii) encouragement and guidance of the patient over time, in particular by requesting a quantified diary of the work achieved to be returned by the patient at each visit. In the control group, participants will benefit from community-based conventional therapy only, according to their physician’s prescription. The primary outcome measure is the composite score of Sunnybrook Facial Grading System. Secondary outcome measures include clinical and biomechanical facial motor function quantifications (Créteil Scale and 3D facial motion analysis through the Cara system), quality of life (Facial Clinimetric Evaluation and Short-Form 12), aesthetic considerations (FACE-Q scale) and mood representations (Hospital Anxiety and Depression scale). Participants will be evaluated every three months by a blinded investigator, in addition to four phone calls (D30/D60/D120/D150) to monitor compliance and tolerance to treatment.DiscussionThis study will increase the level of knowledge on the effects of intense facial motor streng- Facial paralysisthening prescribed through a GSC in patients with chronic peripheral facial paresis.Trial registrationClinicalTrials.gov, NCT04074018. Registered 29 August 2019.Protocol versionVersion N°4.0—04/02/2021.

Real-time emotion detection by quantitative facial motion analysis.

Research into mood and emotion has often depended on slow and subjective self-report, highlighting a need for rapid, accurate, and objective assessment tools. To address this gap, we developed a method using digital image speckle correlation (DISC), which tracks subtle changes in facial expressions invisible to the naked eye, to assess emotions in real-time. We presented ten participants with visual stimuli triggering neutral, happy, and sad emotions and quantified their associated facial responses via detailed DISC analysis. We identified key alterations in facial expression (facial maps) that reliably signal changes in mood state across all individuals based on these data. Furthermore, principal component analysis of these facial maps identified regions associated with happy and sad emotions. Compared with commercial deep learning solutions that use individual images to detect facial expressions and classify emotions, such as Amazon Rekognition, our DISC-based classifiers utilize frame-to-frame changes. Our data show that DISC-based classifiers deliver substantially better predictions, and they are inherently free of racial or gender bias. Our sample size was limited, and participants were aware their faces were recorded on video. Despite this, our results remained consistent across individuals. We demonstrate that DISC-based facial analysis can be used to reliably identify an individual's emotion and may provide a robust and economic modality for real-time, noninvasive clinical monitoring in the future.

Combined Surface Electromyography and Motion Capture for Quantitative Analysis of Facial Movements

Combined Surface Electromyography and Motion Capture for Quantitative Analysis of Facial Movements

EYE ASPECT RATIO ADJUSTMENT DETECTION FOR STRONG BLINKING SLEEPINESS BASED ON FACIAL LANDMARKS WITH EYE-BLINK DATASET

<span lang="id">Blink detection is an important technique in a variety of settings, including facial motion analysis and signal processing. However, automatic blink detection is challenging due to its blink rate. This paper proposes a real-time method for detecting eye blinks in a video series. The method is based on automatic facial landmark detection trained on real-world datasets and demonstrates robustness against various environmental factors, including lighting conditions, facial emotions, and head position. The proposed algorithm calculates the position of facial landmarks, extracts scalar values using the Eye Aspect Ratio (EAR), and characterises eye proximity in each frame. For each video frame, the proposed method calculates the location of the facial landmark and extracts the vertical distance between the eyelids using the position of the facial landmark. Blinks are detected by using the EAR threshold value and recognising the pattern of EAR values in a short temporal window. According to the results from a common data set, it is shown that the proposed approach is more efficient than state-of-the-art techniques.</span>

Quantified analysis of facial movement: A reference for clinical applications.

Most techniques for evaluating unilateral impairments in facial movement yield subjective measurements. The objective of the present study was to define a reference dataset and develop a visualization tool for clinical assessments. In this prospective study, a motion capture system was used to quantify facial movements in 30 healthy adults and 2 patients. We analyzed the displacements of 105 reflective markers placed on the participant's face during five movements (M1-M5). For each marker, the primary endpoint was the maximum amplitude of displacement from the static position (M0) in an analysis of variance. The measurement precision was 0.1 mm. Significant displacements of markers were identified for M1-M5, and displacement patterns were defined. The patients and age-matched healthy participants were compared with regard to the amplitude of displacement. We created a new type of radar plot to visually represent the diagnosis and facilitate effective communication between medical professionals. In proof-of-concept experiments, we collected quantitative data on patients with facial palsy and created a patient-specific radar plot. Our new protocol for clinical facial motion capture ("quantified analysis of facial movement," QAFM) was accurate and should thus facilitate the long-term clinical follow-up of patients with facial palsy. To take account of the limitations affecting the comparison with the healthy side, we created a dataset of healthy facial movements; our method might therefore be applicable to other conditions in which movements on one or both sides of the face are impaired. The patient-specific radar plot enables clinicians to read and understand the results rapidly.

DeepSmile: Anomaly Detection Software for Facial Movement Assessment.

Facial movements are crucial for human interaction because they provide relevant information on verbal and non-verbal communication and social interactions. From a clinical point of view, the analysis of facial movements is important for diagnosis, follow-up, drug therapy, and surgical treatment. Current methods of assessing facial palsy are either (i) objective but inaccurate, (ii) subjective and, thus, depending on the clinician's level of experience, or (iii) based on static data. To address the aforementioned problems, we implemented a deep learning algorithm to assess facial movements during smiling. Such a model was trained on a dataset that contains healthy smiles only following an anomaly detection strategy. Generally speaking, the degree of anomaly is computed by comparing the model's suggested healthy smile with the person's actual smile. The experimentation showed that the model successfully computed a high degree of anomaly when assessing the patients' smiles. Furthermore, a graphical user interface was developed to test its practical usage in a clinical routine. In conclusion, we present a deep learning model, implemented on open-source software, designed to help clinicians to assess facial movements.

Video-Based Facial Movement Analysis in the Assessment of Bulbar Amyotrophic Lateral Sclerosis: Clinical Validation.

Facial movement analysis during facial gestures and speech provides clinically useful information for assessing bulbar amyotrophic lateral sclerosis (ALS). However, current kinematic methods have limited clinical application due to the equipment costs. Recent advancements in consumer-grade hardware and machine/deep learning made it possible to estimate facial movements from videos. This study aimed to establish the clinical validity of a video-based facial analysis for disease staging classification and estimation of clinical scores. Fifteen individuals with ALS and 11 controls participated in this study. Participants with ALS were stratified into early and late bulbar ALS groups based on their speaking rate. Participants were recorded with a three-dimensional (3D) camera (color + depth) while repeating a simple sentence 10 times. The lips and jaw movements were estimated, and features related to sentence duration and facial movements were used to train a machine learning model for multiclass classification and to predict the Amyotrophic Lateral Sclerosis Functional Rating Scale-Revised (ALSFRS-R) bulbar subscore and speaking rate. The classification model successfully separated healthy controls, the early ALS group, and the late ALS group with an overall accuracy of 96.1%. Video-based features demonstrated a high ability to estimate the speaking rate (adjusted R 2 = .82) and a moderate ability to predict the ALSFRS-R bulbar subscore (adjusted R 2 = .55). The proposed approach based on a 3D camera and machine learning algorithms represents an easy-to-use and inexpensive system that can be included as part of a clinical assessment of bulbar ALS to integrate facial movement analysis with other clinical data seamlessly.

Eye Aspect Ratio for Real-Time Drowsiness Detection to Improve Driver Safety

Drowsiness is a major risk factor for road safety, contributing to serious injury, death, and economic loss on the road. Driving performance decreases because of increased drowsiness. In several different applications, such as facial movement analysis and driver safety, blink detection is an essential requirement that is used. The extremely rapid blink rate, on the other hand, makes automatic blink detection an extremely challenging task. This research paper presents a technique for identifying eye blinks in a video series recorded by a car dashboard camera in real time. The suggested technique determines the facial landmark positions for each video frame and then extracts the vertical distance between the eyelids from the facial landmark positions. The algorithm that has been proposed estimates the facial landmark positions, extracts a single scalar quantity by making use of Eye Aspect Ratio (EAR), and identifies the eye closeness in each frame. In the end, blinks are recognized by employing the modified EAR threshold value in conjunction with a pattern of EAR values in a relatively short period of time. Experimental evidence indicates that the greater the EAR threshold, the worse the AUC’s accuracy and performance. Further, 0.18 was determined to be the optimum EAR threshold in our research.

Learning Facial Motion Representation with a Lightweight Encoder for Identity Verification

Deep learning became an important image classification and object detection technique more than a decade ago. It has since achieved human-like performance for many computer vision tasks. Some of them involve the analysis of human face for applications like facial recognition, expression recognition, and facial landmark detection. In recent years, researchers have generated and made publicly available many valuable datasets that allow for the development of more accurate and robust models for these important tasks. Exploiting the information contained inside these pretrained deep structures could open the door to many new applications and provide a quick path to their success. This research focuses on a unique application that analyzes short facial motion video for identity verification. Our proposed solution leverages the rich information in those deep structures to provide accurate face representation for facial motion analysis. We have developed two strategies to employ the information contained in the existing models for image-based face analysis to learn the facial motion representations for our application. Combining with those pretrained spatial feature extractors for face-related analyses, our customized sequence encoder is capable of generating accurate facial motion embedding for identity verification application. The experimental results show that the facial geometry information from those feature extractors is valuable and helps our model achieve an impressive average precision of 98.8% for identity verification using facial motion.

Facial Motion Analysis beyond Emotional Expressions.

Facial motion analysis is a research field with many practical applications, and has been strongly developed in the last years. However, most effort has been focused on the recognition of basic facial expressions of emotion and neglects the analysis of facial motions related to non-verbal communication signals. This paper focuses on the classification of facial expressions that are of the utmost importance in sign languages (Grammatical Facial Expressions) but also present in expressive spoken language. We have collected a dataset of Spanish Sign Language sentences and extracted the intervals for three types of Grammatical Facial Expressions: negation, closed queries and open queries. A study of several deep learning models using different input features on the collected dataset (LSE_GFE) and an external dataset (BUHMAP) shows that GFEs can be learned reliably with Graph Convolutional Networks simply fed with face landmarks.

Static and Motion Facial Analysis for Craniofacial Assessment and Diagnosing Diseases.

Deviation from a normal facial shape and symmetry can arise from numerous sources, including physical injury and congenital birth defects. Such abnormalities can have important aesthetic and functional consequences. Furthermore, in clinical genetics distinctive facial appearances are often associated with clinical or genetic diagnoses; the recognition of a characteristic facial appearance can substantially narrow the search space of potential diagnoses for the clinician. Unusual patterns of facial movement and expression can indicate disturbances to normal mechanical functioning or emotional affect. Computational analyses of static and moving 2D and 3D images can serve clinicians and researchers by detecting and describing facial structural, mechanical, and affective abnormalities objectively. In this review we survey traditional and emerging methods of facial analysis, including statistical shape modeling, syndrome classification, modeling clinical face phenotype spaces, and analysis of facial motion and affect.

Adjusting eye aspect ratio for strong eye blink detection based on facial landmarks.

Blink detection is an important technique in a variety of settings, including facial movement analysis and signal processing. However, automatic blink detection is very challenging because of the blink rate. This research work proposed a real-time method for detecting eye blinks in a video series. Automatic facial landmarks detectors are trained on a real-world dataset and demonstrate exceptional resilience to a wide range of environmental factors, including lighting conditions, face emotions, and head position. For each video frame, the proposed method calculates the facial landmark locations and extracts the vertical distance between the eyelids using the facial landmark positions. Our results show that the recognizable landmarks are sufficiently accurate to determine the degree of eye-opening and closing consistently. The proposed algorithm estimates the facial landmark positions, extracts a single scalar quantity by using Modified Eye Aspect Ratio (Modified EAR) and characterizing the eye closeness in each frame. Finally, blinks are detected by the Modified EAR threshold value and detecting eye blinks as a pattern of EAR values in a short temporal window. According to the results from a typical data set, it is seen that the suggested approach is more efficient than the state-of-the-art technique.

Analysis of Facial Movement in Repaired Unilateral Cleft Lip Using Three-Dimensional Motion Capture.

Unilateral cleft lip (UCL) is one of the most common craniofacial deformities. Surgical intervention reconstructs lip and nose anatomy; however, some degree of asymmetry persists after repair. This demonstrates a need for a model for studying and improving outcomes for patients with orofacial clefts. This study's main question was whether there is a significant difference in dynamic facial asymmetry between participants with repaired UCLs and control participants during smiling. Ten pediatric subjects with repaired left UCLs and 12 with no craniofacial diagnoses were recorded performing maximum smiles using a markerless 4D video stereophotogrammetrical system. A facial mesh template containing 884 landmarks was conformed to each initial frame and tracked throughout. Kinetic analysis of smiles was performed by calculating landmark 3D Euclidean distance between frames. Patients with left repaired UCL showed increasing facial asymmetry throughout smiling. Oral commissures, upper, and lower lips demonstrated significantly greater movement on the right side (P < 0.05). Control patients showed facial asymmetry during the first half of smiling, with greater movement on the left side. Displacement difference between right and left was significantly greater at oral commissures and upper lips in patients with repaired ULC compared to control patients. This study provides a highly detailed, quantitative analysis of postoperative UCLs, and help improve outcomes of future repair surgeries.

A New Dataset for Facial Motion Analysis in Individuals With Neurological Disorders.

We present the first public dataset with videos of oro-facial gestures performed by individuals with oro-facial impairment due to neurological disorders, such as amyotrophic lateral sclerosis (ALS) and stroke. Perceptual clinical scores from trained clinicians are provided as metadata. Manual annotation of facial landmarks is also provided for a subset of over 3300 frames. Through extensive experiments with multiple facial landmark detection algorithms, including state-of-the-art convolutional neural network (CNN) models, we demonstrated the presence of bias in the landmark localization accuracy of pre-trained face alignment approaches in our participant groups. The pre-trained models produced higher errors in the two clinical groups compared to age-matched healthy control subjects. We also investigated how this bias changes when the existing models are fine-tuned using data from the target population. The release of this dataset aims to propel the development of face alignment algorithms robust to the presence of oro-facial impairment, support the automatic analysis and recognition of oro-facial gestures, enhance the automatic identification of neurological diseases, as well as the estimation of disease severity from videos and images.

Masseteric-zygomatic nerve transfer for the management of eye closure-smile excursion synkinesis.

The purpose of this study is to illustrate the efficacy of masseteric-to-zygomatic nerve transfer to address eye closure-smile excursion synkinesis after facial nerve paralysis. Synkinesis after facial nerve paralysis represents a wide range of facial movement disability. One manifestation is involuntary smiling with eye closure and a concomitant reduction of oral commissure movement with attempted smile ("frozen smile") - arising as a result of aberrant fibers populating the zygomatic branch-muscle complex. This is a particularly difficult area to treat with conservative management. We propose a single-stage procedure to sever the dysfunctional zygomatic nerve and perform a masseteric-zygomatic nerve coaptation to recover a voluntary smile. We present a case series of eight patients with eye closure/smile excursion synkinesis who underwent single-stage masseteric-zygomatic nerve transfer by a single surgeon. The surgical technique and indications for surgery were reviewed. Patients underwent facial movement analysis using Emotrics. We analyzed the pre- and post- surgical photographic images of 8 patients with synkinesis (7 female, 1 male). Masseteric-facial nerve transfer was performed from 18months to 22years after the initial facial paralysis. Eyelid and brow positioning were more symmetric after surgery, with discrepancy between affected and unaffected side decreasing from 2.1 to 1.0mm (p<.05) and 1.74 to 1.29mm (p<.05), respectively. Symmetry of smile excursion postoperatively was also improved with commissure excursion discrepancy decreasing from 8.8 to 3.78mm (p<.05). Discrepancy in the smile angle when comparing affected to unaffected side improved postoperatively from 10.3 to 5.2 degrees (p<.05). Improvement in oral commissure height was noted, but not statistically significant. The masseteric-zygomatic nerve transfer is a useful technique for the treatment of eye closure/smile excursion synkinesis after failure of chemodenervation and/or physical therapy.

Software-based video analysis of functional outcomes of face transplantation.

Assessment of outcomes after face transplantation (FT) is necessary to provide sound evidence on the benefits of this life-giving surgery. Current methods for outcomes assessment, however, are imprecise or prone to subjectivity. Software-based video analysis may allow fast, objective and retrospective assessment of restoration of facial movements and functions after FT. We recorded videos of 7 subjects before as well as every 3-6 months after facial transplantation. Patients performed the same sequence of facial movements in every video: smile, open mouth, purse lips, wrinkle nose, frown, close eyes, and lift eyebrows. The videos were retrospectively analyzed using EMOTIENT software, which is capable of automatic tracking and detailed measurements of facial movements and expressions. These measurements were subsequently compared to the same patient at different time points, as well as to the normal population. Open mouth, wrinkle nose and smile functions significantly improved in all patients when compared to pre-transplant functions; this improvement was significant at 3, 6, and 12 months after transplant, respectively. Lip purse, eye closure and frown functions improved by 6, 9, and 18 months after transplantation, respectively; however, improvement in these particular functions was not significantly with respect to pre-transplant. Face transplantation did not improve any of the patients' ability to lift their eyebrows. Most remarkably, mouth opening and smiling functions both reached values comparable to the normal population at 3 and 12 months after transplantation, respectively. Software-based video analysis provides a valuable assessment tool capable of objective, precise and reproducible analysis of facial movements and functions after FT.

Kinematic Features of Jaw and Lips Distinguish Symptomatic From Presymptomatic Stages of Bulbar Decline in Amyotrophic Lateral Sclerosis.

The goals of this study were to (a) classify speech movements of patients with amyotrophic lateral sclerosis (ALS) in presymptomatic and symptomatic phases of bulbar function decline relying solely on kinematic features of lips and jaw and (b) identify the most important measures that detect the transition between early and late bulbar changes. One hundred ninety-two recordings obtained from 64 patients with ALS were considered for the analysis. Feature selection and classification algorithms were used to analyze lip and jaw movements recorded with Optotrak Certus (Northern Digital Inc.) during a sentence task. A feature set, which included 35 measures of movement range, velocity, acceleration, jerk, and area measures of lips and jaw, was used to classify sessions according to the speaking rate into presymptomatic (> 160 words per minute) and symptomatic (< 160 words per minute) groups. Presymptomatic and symptomatic phases of bulbar decline were distinguished with high accuracy (87%), relying only on lip and jaw movements. The best features that allowed detecting the differences between early and later bulbar stages included cumulative path of lower lip and jaw, peak values of velocity, acceleration, and jerk of lower lip and jaw. The results established a relationship between facial kinematics and bulbar function decline in ALS. Considering that facial movements can be recorded by means of novel inexpensive and easy-to-use, video-based methods, this work supports the development of an automatic system for facial movement analysis to help clinicians in tracking the disease progression in ALS.

Head motion invariance in three-dimensional video analysis system for the analysis of facial movement

Head motion invariance in three-dimensional video analysis system for the analysis of facial movement

Three-dimensional motion analysis of facial movement during verbal and nonverbal expressions in healthy subjects.

Functional impairments of facial expressions alter the quality of life and their quantitative analysis is key to describing and grading facial function and dysfunction. The aim of this investigation was to quantify 3D lip movements in a group of healthy young subjects. Lip movements during five repetitions of common and random sequences of vowels and nonverbal expressions were recorded using an optoelectronic 3D motion analyzer. The mean maximum displacement of facial landmarks was used to compute 3D unilateral mobility and the symmetry indices (SI) separately for each sex. Facial mobility was significantly greater during open-mouth than closed-mouth smiles. Facial movement among the subjects was largest during articulation of the vowel /a/ and smallest for /i/. Lips were pursed with a SI very close to 95%, while smiles were slightly more asymmetric. The vowel /a/ was articulated with significantly greater symmetry than /e/, /i/ and /u/. The outcomes suggest that the proposed method characterized normal facial animations satisfactorily, so it could be a useful tool for assessing patients with facial lesions and dentofacial deformities. Patient assessment would profit from this quantitative approach, which would reduce discrepancies among several clinical examinations. Clin. Anat. 29:991-997, 2016. © 2016 Wiley Periodicals, Inc.