Facial Landmark Detection Research Articles

A Study on Tiredness Assessment by Using Eye Blink Detection

In this paper, the loss of attention of automotive drivers is studied by using eye blink detection. Facial landmark detection for detecting eye is explored. Afterward, eye blink is detected using Eye Aspect Ratio. By comparing the time of eye closure to a particular period, the driver’s tiredness is decided. The total number of eye blinks in a minute is counted to detect drowsiness. Calculation of total eye blinks in a minute for the driver is done, then compared it with a known standard value. If any of the above conditions fulfills, the system decides the driver is unconscious. A total of 120 samples were taken by placing the light source front, back, and side. There were 40 samples for each position of the light source. The maximum error rate occurred when the light source was placed back with a 15% error rate. The best scenario was 7.5% error rate where the light source was placed front side. The eye blinking process gave an average error of 11.67% depending on the various position of the light source. Another 120 samples were taken at a different time of the day for calculating total eye blink in a minute. The maximum number of blinks was in the morning with an average blink rate of 5.78 per minute, and the lowest number of blink rate was in midnight with 3.33% blink rate. The system performed satisfactorily and achieved the eye blink pattern with 92.7% accuracy.

Cascade of encoder-decoder CNNs with learned coordinates regressor for robust facial landmarks detection

Convolutional Neural Nets (CNNs) have become the reference technology for many computer vision problems. Although CNNs for facial landmark detection are very robust, they still lack accuracy when processing images acquired in unrestricted conditions. In this paper we investigate the use of a cascade of Neural Net regressors to increase the accuracy of the estimated facial landmarks. To this end we append two encoder-decoder CNNs with the same architecture. The first net produces a set of heatmaps with a rough estimation of landmark locations. The second, trained with synthetically generated occlusions, refines the location of ambiguous and occluded landmarks. Finally, a densely connected layer with shared weights among all heatmaps, accurately regresses the landmark coordinates. The proposed approach achieves state-of-the-art results in 300W, COFW and WFLW that are widely considered the most challenging public data sets.

A Unified Tensor-based Active Appearance Model

Appearance variations result in many difficulties in face image analysis. To deal with this challenge, we present a Unified Tensor-based Active Appearance Model (UT-AAM) for jointly modelling the geometry and texture information of 2D faces. For each type of face information, namely shape and texture, we construct a unified tensor model capturing all relevant appearance variations. This contrasts with the variation-specific models of the classical tensor AAM. To achieve the unification across pose variations, a strategy for dealing with self-occluded faces is proposed to obtain consistent shape and texture representations of pose-varied faces. In addition, our UT-AAM is capable of constructing the model from an incomplete training dataset, using tensor completion methods. Last, we use an effective cascaded-regression-based method for UT-AAM fitting. With these advancements, the utility of UT-AAM in practice is considerably enhanced. As an example, we demonstrate the improvements in training facial landmark detectors through the use of UT-AAM to synthesise a large number of virtual samples. Experimental results obtained on a number of well-known face datasets demonstrate the merits of the proposed approach.

A Modular System for Detection, Tracking and Analysis of Human Faces in Thermal Infrared Recordings.

We present a system that utilizes a range of image processing algorithms to allow fully automated thermal face analysis under both laboratory and real-world conditions. We implement methods for face detection, facial landmark detection, face frontalization and analysis, combining all of these into a fully automated workflow. The system is fully modular and allows implementing own additional algorithms for improved performance or specialized tasks. Our suggested pipeline contains a histogtam of oriented gradients support vector machine (HOG-SVM) based face detector and different landmark detecion methods implemented using feature-based active appearance models, deep alignment networks and a deep shape regression network. Face frontalization is achieved by utilizing piecewise affine transformations. For the final analysis, we present an emotion recognition system that utilizes HOG features and a random forest classifier and a respiratory rate analysis module that computes average temperatures from an automatically detected region of interest. Results show that our combined system achieves a performance which is comparable to current stand-alone state-of-the-art methods for thermal face and landmark datection and a classification accuracy of 65.75% for four basic emotions.

A Standalone Vision Device to Recognize Facial Landmarks and Smile in Real Time Using Raspberry Pi and Sensor

In current scenario of technological advancement, human-machine association is becoming sought after and machine needs to comprehend human emotions and feelings. The productivity of an exercise can be improved to a considerable extent, if a machine can distinguish human feelings by understanding the human conduct. Feelings can comprehend by content, vocal, verbal and outward appearances. The major deciding factor in the identification of human emotions is Facial expression. Working with facial images and emotion is real time is a big task. It is also found that confined amount of work has been done in this field. In this paper, we propose a technique for facial landmark detection and feature extraction which is the most crucial prerequisite for emotion recognition system by capturing the facial images in real time. The proposed system is divided into three tightly coupled stages of face detection, landmark detection and feature extraction. This is done by HOG and Linear SVM-based face detector using dlib and OpenCV. The curiosity of our proposed strategy lies in the execution stage. Raspberry Pi III, B+ and a normal exactness of 99.9% is accomplished at ongoing. This paper can be proved as the basis of real time emotion recognition in majority of applications.

Pupil Detection Algorithm Based on Feature Extraction for Eye Gaze

Exact real-time pupil tracking is an essential step in a live eye gaze. Since pupil centre is a base point’s reference, eye centre localization is essential for many applications. In this research, we extract pupil eye features exactly within different intensity levels of eye images, mostly with localization of determined interest objects and where the human is looking for. Since it’s a digital world and digital transformation, everything is becoming virtual. Hence this concept has a huge scope in e-learning, class room training and analyzing human behaviour. This research covers eye tracking technology to track and analyze the learners' behavior and emotion on e-learning platform like level of attention and tiredness. Harr’s cascade classifier was used to first locate the eye’s area, and once found and support vector machine (SVM) for classification with the trained datasets. We also include the state of emotions, facial landmarks of the salient patches on face image using automated learning-free facial landmark detection technique.Experimental results help in developing learner eye gaze detection in system using Pycharm and hardware output using Raspberry Pi. In Raspberry Pi is given with the input image captured using external webcam and based on the engagement level of the learner content 1 or 2 would be displayed in the Raspbian OS environment

Automatic cephalometric landmarks detection on frontal faces: An approach based on supervised learning techniques

Facial landmarks are employed in many research areas, including facial recognition, craniofacial identification, age and sex estimation being the most important. In forensics, the focus is on the analysis of a particular set of facial landmarks, defined as cephalometric landmarks. Previous studies demonstrated that the descriptive adequacy of these anatomical references for indirect application (photo-anthropometric description) increased the marking precision of these points, contributing to greater reliability of these analyses. Nevertheless, most are performed manually and all are subject to bias on the part of expert examiners. Therefore, the purpose of this work was to develop and validate automatic techniques for detection of cephalometric landmarks from digital images of frontal facial images in forensics. The presented approach uses a combination of computer vision and image processing techniques within supervised learning procedures. The proposed methodology obtains similar precision to a group of human manual cephalometric reference markers and results that are more accurate than other state-of-the-art facial landmark detection frameworks. It achieves a normalized mean distance (in pixels) error of 0.014, similar to the mean inter-expert dispersion (0.009) and clearly better than other automatic approaches that were analyzed during the course of this study (0.026 and 0.101).

Soft Facial Landmark Detection by Label Distribution Learning

Most existing facial landmark detection algorithms regard the manually annotated landmarks as precise hard labels, therefore, the accurate annotated landmarks are essential to the training of these algorithms. However, in many cases, there exist deviations in manual annotations, and the landmarks marked for facial parts with occlusion and large poses are not always accurate, which means that the “ground truth” landmarks are usually not annotated precisely. In such case, it is more reasonable to use soft labels rather than explicit hard labels. Therefore, this paper proposes to associate a bivariate label distribution (BLD) to each landmark of an image. A BLD covers the neighboring pixels around the original manually annotated point, alleviating the problem of inaccurate landmarks. After generating a BLD for each landmark, the proposed method firstly learns the mappings from an image patch to the BLD of each landmark, and then the predicted BLDs are used in a deformable model fitting process to obtain the final facial shape for the image. Experimental results show that the proposed method performs better than the compared state-of-the-art facial landmark detection algorithms. Furthermore, the proposed method appears to be much more robust against the landmark noise in the training set than other compared baselines.

The face of art

Facial Landmark detection in natural images is a very active research domain. Impressive progress has been made in recent years, with the rise of neural-network based methods and large-scale datasets. However, it is still a challenging and largely unexplored problem in the artistic portraits domain. Compared to natural face images, artistic portraits are much more diverse. They contain a much wider style variation in both geometry and texture and are more complex to analyze. Moreover, datasets that are necessary to train neural networks are unavailable. We propose a method for artistic augmentation of natural face images that enables training deep neural networks for landmark detection in artistic portraits. We utilize conventional facial landmarks datasets, and transform their content from natural images into "artistic face" images. In addition, we use a feature-based landmark correction step, to reduce the dependency between the different facial features, which is necessary due to position and shape variations of facial landmarks in artworks. To evaluate our landmark detection framework, we created an "Artistic-Faces" dataset, containing 160 artworks of various art genres, artists and styles, with a large variation in both geometry and texture. Using our method, we can detect facial features in artistic portraits and analyze their geometric style. This allows the definition of signatures for artistic styles of artworks and artists, that encode both the geometry and the texture style. It also allows us to present a geometric-aware style transfer method for portraits.

First-person Video Analysis for Evaluating Skill Level in the Humanitude Tender-Care Technique

In this paper, we describe a wearable first-person video (FPV) analysis system for evaluating the skill levels of caregivers. This is a part of our project that aims to quantize and analyze the tender-care technique known as Humanitude by using wearable sensing and AI technology devices. Using our system, caregivers can evaluate and elevate their care levels by themselves. From the FPVs of care sessions taken by wearable cameras worn by caregivers, we obtained the 3D facial distance, pose and eye-contact states between caregivers and receivers by using facial landmark detection and deep neural network (DNN)-based eye contact detection. We applied statistical analysis to these features and developed algorithms that provide scores for tender-care skill. In experiments, we first evaluated the performance of our DNN-based eye contact detection by using eye contact datasets prepared from YouTube videos and FPVs that assume conversational scenes. We then performed skill evaluations by using Humanitude training scenes involving three novice caregivers, two Humanitude experts and seven middle-level students. The results showed that our eye contact detection outperformed existing methods and that our skill evaluations can estimate the care skill levels.

A Fast Facial Landmarks Detection and Posture Classification Algorithm

In the field of face recognition research, the efficiency of facial landmarks detection and posture classification algorithm is a serious problem. This paper proposes a lightweight network based on convolutional neural network to quickly detect the facial landmarks of human faces. Based on this, the posture is predicted using the relative position of the obtained feature points. The model was tested by the face images captured in various scenes in real life. The accuracy and recall rate of the proposed algorithm were 96.3% and 98.2%, respectively. The test time for a single picture is 0.9ms. The proposed algorithm has less running time and its accuracy and recall rate are similar or even better than other models.

Visual and Thermal Image Processing for Facial Specific Landmark Detection to Infer Emotions in a Child-Robot Interaction.

Child-Robot Interaction (CRI) has become increasingly addressed in research and applications. This work proposes a system for emotion recognition in children, recording facial images by both visual (RGB—red, green and blue) and Infrared Thermal Imaging (IRTI) cameras. For this purpose, the Viola-Jones algorithm is used on color images to detect facial regions of interest (ROIs), which are transferred to the thermal camera plane by multiplying a homography matrix obtained through the calibration process of the camera system. As a novelty, we propose to compute the error probability for each ROI located over thermal images, using a reference frame manually marked by a trained expert, in order to choose that ROI better placed according to the expert criteria. Then, this selected ROI is used to relocate the other ROIs, increasing the concordance with respect to the reference manual annotations. Afterwards, other methods for feature extraction, dimensionality reduction through Principal Component Analysis (PCA) and pattern classification by Linear Discriminant Analysis (LDA) are applied to infer emotions. The results show that our approach for ROI locations may track facial landmarks with significant low errors with respect to the traditional Viola-Jones algorithm. These ROIs have shown to be relevant for recognition of five emotions, specifically disgust, fear, happiness, sadness, and surprise, with our recognition system based on PCA and LDA achieving mean accuracy (ACC) and Kappa values of 85.75% and 81.84%, respectively. As a second stage, the proposed recognition system was trained with a dataset of thermal images, collected on 28 typically developing children, in order to infer one of five basic emotions (disgust, fear, happiness, sadness, and surprise) during a child-robot interaction. The results show that our system can be integrated to a social robot to infer child emotions during a child-robot interaction.

Branched convolutional neural networks incorporated with Jacobian deep regression for facial landmark detection

Facial landmark detection is to localize multiple facial key-points for a given facial image. While many methods have achieved remarkable performance in recent years, the accuracy remains unsatisfactory due to some uncontrolled conditions such as occlusion, head pose variations and illumination, under which, the L2 loss function is conventionally dominated by errors from those facial components on which the landmarks are hard predicted. In this paper, a novel branched convolutional neural network incorporated with Jacobian deep regression framework, hereafter referred to as BCNN-JDR, is proposed to solve the facial landmark detection problem. Our proposed framework consists of two parts: initialization stage and cascaded refinement stages. We firstly exploit branched convolutional neural networks as the robust initializer to estimate initial shape, which is incorporated with the knowledge of component-aware branches. By virtue of the component-aware branches mechanism, BCNN can effectively alleviate this issue of the imbalance errors among facial components and provide the robust initial face shape. Following the BCNN, a sequence of refinement stages are cascaded to fine-tune the initial shape within a narrow range. In each refinement stage, the local texture information is adopted to fit the facial local nonlinear variation. Moreover, our entire framework is jointly optimized via the Jacobian deep regression optimization strategy in an end-to-end manner. Jacobian deep regression optimization strategy has an ability to backward propagate the training error of the last stage to all previous stages, which implements a global optimization approach to our proposed framework. Experimental results on benchmark datasets demonstrate that the proposed BCNN-JDR is robust against uncontrolled conditions and outperforms the state-of-the-art approaches.

A hierarchically trained generative network for robust facial symmetrization.

Face symmetrization has extensive applications in both medical and academic fields, such as facial disorder diagnosis. Human face possesses an important characteristic, which is as known as symmetry. However, in many scenarios, the perfect symmetry doesn’t exist in human faces, which yields a large number of studies around this topic. For example, facial palsy evaluation, facial beauty evaluation based on facial symmetry analysis, and many among others. Currently, there are still very limited researches dedicated for automatic facial symmetrization. Most of the existing studies only utilized their own implantations for facial symmetrization to assist their interdisciplinary academic researches. Limitations thus can be noticed in their methods, such as the requirements for manual interventions. Furthermore, most existing methods utilize facial landmark detection algorithms for automatic facial symmetrization. Though accuracies of the landmark detection algorithms are promising, the uncontrolled conditions in the facial images can still negatively impact the performance of the symmetrical face production. To this end, this paper presents a joint-loss enhanced deep generative network model for automatic facial symmetrization, which is achieved by a full facial image analysis. The joint-loss consists of a pair of adversarial losses and an identity loss. The adversarial losses try to make the generated symmetrical face as realistic as possible, while the identity loss helps to constrain the output to have the same identity of the person in the original input as much as possible. Rather than an end-to-end learning strategy, the proposed model is constructed by a multi-stage training process, which avoids the demand for a large size of the symmetrical face as training data. Experiments are conducted with comparisons with several existing methods based on some of the most popular facial landmark detection algorithms. Competitive results of the proposed method are demonstrated.

Unobtrusive Respiratory Flow Monitoring Using a Thermopile Array: A Feasibility Study

Low-resolution thermal cameras have already been used in the detection of respiratory flow. However, microbolometer technology has a high production cost compared to thermopile arrays. In this work, the feasibility of using a thermopile array to detect respiratory flow has been investigated in multiple settings. To prove the concept, we tested the detector on six healthy subjects. Our method automatically selects the region-of-interest by discriminating between sensor elements that output noise and flow-induced signals. The thermopile array yielded an average root mean squared error of 1.59 b r e a t h s p e r m i n u t e . Parameters such as distance, breathing rate, orientation, and oral or nasal breathing resulted in being fundamental in the detection of respiratory flow. The paper provides the proof-of-concept that low-cost thermopile-arrays can be used to monitor respiratory flow in a lab setting and without the need for facial landmark detection. Further development could provide a more attractive alternative for the earlier bolometer-based proposals.

Light Fields for Face Analysis.

The term “plenoptic” comes from the Latin words plenus (“full”) + optic. The plenoptic function is the 7-dimensional function representing the intensity of the light observed from every position and direction in 3-dimensional space. Thanks to the plenoptic function it is thus possible to define the direction of every ray in the light-field vector function. Imaging systems are rapidly evolving with the emergence of light-field-capturing devices. Consequently, existing image-processing techniques need to be revisited to match the richer information provided. This article explores the use of light fields for face analysis. This field of research is very recent but already includes several works reporting promising results. Such works deal with the main steps of face analysis and include but are not limited to: face recognition; face presentation attack detection; facial soft-biometrics classification; and facial landmark detection. This article aims to review the state of the art on light fields for face analysis, identifying future challenges and possible applications.

A fear detection method based on palpebral fissure

Human emotions, such as smiling or laughing, can be expressed in various forms through the face whenever there are stimuli. These changing faces can reflect the emotional states that are used to identify a normal or an abnormal behaviour. This research aims to study the patterns in human faces and identify the areas of interest (AOI), which is called Facial Landmark Detection (FLD). The investigation of the external elements of eyes is performed, and it consists of the interpalpebral fissure (IPF), the palpebral fissure length (PFL), and the palpebral fissure region (PFR). These elements are applied to classify the emotional differences between neutral and fearful emotions. A method for emotional classification was designed according to the changing values of the IPF, PFL, and PFR. An ID3 algorithm was used to classify the emotions. Three hundred sixty images were derived from horror-thriller-murder movies based on IMDb. This data set was utilized to generate the proposed pattern. This pattern was used to classify the emotions using a decision tree technique that led to the development of an emotional classification model. The accuracy of the emotional classification model between neutral and fearful emotions was 92.50%, thus proving that the proposed model is efficient.

Facial Landmark Detection based on Cascade Neural Network

A three-stage cascade convolutional neural network is proposed in this paper to learn the facial features. In order to improve the robustness and accuracy of facial landmark detection, several parallel sub-networks are involved in each stage of network. The convolutional neural network performs global advanced feature extraction for the entire facial region in the preceding stage, so as to obtain more accurate positioning. Since the first-stage input covers the whole image, the global facial information can be effectively used to weaken or even avoid the detection errors caused by occlusion, large posture changes, extreme illumination and etc. During facial detection, the first-stage network will make initial prediction of facial landmarks, and then get the facial area narrowed down and accurate according to the prediction results; the second-stage network will take the narrowed-down facial area above as the input, the three parallel convolution networks are used to classify the facial landmarks for quadratic estimates, so as to improve the accuracy of prediction results based on the results of first-stage network. The input of the third-stage network is limited to the small neighborhood of the second-stage prediction results, which can make the prediction results more accurate. The experimental data shows that our method has certain advantages in detection accuracy and reliability.

Deep coupling neural network for robust facial landmark detection

Facial landmark detection aims at locating a sparse set of fiducial facial key-points. Two significant issues (i.e., Intra-Dataset Variation and Inter-Dataset Variation) remain in datasets which dramatically lead to performance degradation. Specifically, dataset variations will lead to severe over-fitting easily and perform poor generalization in recent in-the-wild datasets which severely harm the robustness of facial landmark detection algorithm. In this study, we show that model robustness can be significantly improved by leveraging rich variations within and between different datasets. This is non-trivial because of the serious data bias within one certain dataset and inconsistent landmark definitions between different datasets, which make it an extraordinarily tough task.To address the mentioned problems, we proposed a novel Deep Coupling Neural Network (DCNN), which consists of two strong coupling sub-networks, e.g., Dataset-Across Network (DA-Net) and Candidate-Decision Network (CD-Net). In particular, DA-Net takes advantage of different characteristics and distributions across different datasets, while CD-Net makes a final decision on candidate hypotheses given by DA-Net to leverage variations within one certain dataset. Extensive evaluations show that our approach dramatically outperforms state-of-the-art methods on the challenging 300-W and WFLW dataset.

Survey on the approaches based geometric information for 3D face landmarks detection

Facial landmarks detection is an important and basic step in many face analysis applications. For this reason, it is considered a challenging task as the final results of the analysis depend on the accuracy of the landmarks detection. Decades of research have investigated approaches for two-dimensional (2D) facial landmarks detection but; however, the good obtained results, they still suffer from some weakness regarding the pose and illumination variations. Recently, the large availability of 3D scans makes the use of 3D face models easier hence, overcome the problems caused using 2D images. Many papers have studied the problem of 3D facial landmarks detection; nevertheless, there is a lack of literature reviews allowing an overview of the studies and researches related to the 3D face landmarks detection. In this study, the authors present a detailed survey of the latest (2010-2018) approaches based geometric information for 3D face landmarks detection, including the limitations and strengths of each work.