Facial Keypoints Research Articles

Training‐based head pose estimation under monocular vision

Although many 3D head pose estimation methods based on monocular vision can achieve an accuracy of 5°, how to reduce the number of required training samples and how to not to use any hardware parameters as input features are still among the biggest challenges in the field of head pose estimation. To aim at these challenges, the authors propose an accurate head pose estimation method which can act as an extension to facial key point detection systems. The basic idea is to use the normalised distance between key points as input features, and to use ℓ1‐minimisation to select a set of sparse training samples which can reflect the mapping relationship between the feature vector space and the head pose space. The linear combination of the head poses corresponding to these samples represents the head pose of the test sample. The experiment results show that the authors’ method can achieve an accuracy of 2.6° without any extra hardware parameters or information of the subject. In addition, in the case of large head movement and varying illumination, the authors’ method is still able to estimate the head pose.

Improving facial analysis and performance driven animation through disentangling identity and expression

We present techniques for improving performance driven facial animation, emotion recognition, and facial key-point or landmark prediction using learned identity invariant representations. Established approaches to these problems can work well if sufficient examples and labels for a particular identity are available and factors of variation are highly controlled. However, labeled examples of facial expressions, emotions and key-points for new individuals are difficult and costly to obtain. In this paper we improve the ability of techniques to generalize to new and unseen individuals by explicitly modeling previously seen variations related to identity and expression. We use a weakly-supervised approach in which identity labels are used to learn the different factors of variation linked to identity separately from factors related to expression. We show how probabilistic modeling of these sources of variation allows one to learn identity-invariant representations for expressions which can then be used to identity-normalize various procedures for facial expression analysis and animation control. We also show how to extend the widely used techniques of active appearance models and constrained local models through replacing the underlying point distribution models which are typically constructed using principal component analysis with identity–expression factorized representations. We present a wide variety of experiments in which we consistently improve performance on emotion recognition, markerless performance-driven facial animation and facial key-point tracking.

A reduced feature set for driver head pose estimation

Evaluation of driving performance is of utmost importance in order to reduce road accident rate. Since driving ability includes visual-spatial and operational attention, among others, head pose estimation of the driver is a crucial indicator of driving performance. This paper proposes a new automatic method for coarse and fine head's yaw angle estimation of the driver. We rely on a set of geometric features computed from just three representative facial keypoints, namely the center of the eyes and the nose tip. With these geometric features, our method combines two manifold embedding methods and a linear regression one. In addition, the method has a confidence mechanism to decide if the classification of a sample is not reliable. The approach has been tested using the CMU-PIE dataset and our own driver dataset. Despite the very few facial keypoints required, the results are comparable to the state-of-the-art techniques. The low computational cost of the method and its robustness makes feasible to integrate it in massive consume devices as a real time application.

Efficient quantitative assessment of facial paralysis using iris segmentation and active contour-based key points detection with hybrid classifier.

BackgroundFacial palsy or paralysis (FP) is a symptom that loses voluntary muscles movement in one side of the human face, which could be very devastating in the part of the patients. Traditional methods are solely dependent to clinician’s judgment and therefore time consuming and subjective in nature. Hence, a quantitative assessment system becomes apparently invaluable for physicians to begin the rehabilitation process; and to produce a reliable and robust method is challenging and still underway.MethodsWe introduce a novel approach for a quantitative assessment of facial paralysis that tackles classification problem for FP type and degree of severity. Specifically, a novel method of quantitative assessment is presented: an algorithm that extracts the human iris and detects facial landmarks; and a hybrid approach combining the rule-based and machine learning algorithm to analyze and prognosticate facial paralysis using the captured images. A method combining the optimized Daugman’s algorithm and Localized Active Contour (LAC) model is proposed to efficiently extract the iris and facial landmark or key points. To improve the performance of LAC, appropriate parameters of initial evolving curve for facial features’ segmentation are automatically selected. The symmetry score is measured by the ratio between features extracted from the two sides of the face. Hybrid classifiers (i.e. rule-based with regularized logistic regression) were employed for discriminating healthy and unhealthy subjects, FP type classification, and for facial paralysis grading based on House-Brackmann (H-B) scale.ResultsQuantitative analysis was performed to evaluate the performance of the proposed approach. Experiments show that the proposed method demonstrates its efficiency.ConclusionsFacial movement feature extraction on facial images based on iris segmentation and LAC-based key point detection along with a hybrid classifier provides a more efficient way of addressing classification problem on facial palsy type and degree of severity. Combining iris segmentation and key point-based method has several merits that are essential for our real application. Aside from the facial key points, iris segmentation provides significant contribution as it describes the changes of the iris exposure while performing some facial expressions. It reveals the significant difference between the healthy side and the severe palsy side when raising eyebrows with both eyes directed upward, and can model the typical changes in the iris region.

Hallucinating faces based on adaptive neighborhood selection and dual tree complex wavelet transform

In this paper, a novel face hallucination method is proposed. Instead of utilizing the gray values of pixels directly, the feature vector extracted using dual tree complex wavelet transform (DTCWT) is exploited to describe the directional detailed information of facial image. Then, the adaptive neighborhood selection (ANS) algorithm is used to preserve neighborhood relationship between the low-resolution and high-resolution face by selecting the optimal neighbors of each facial patch from the feature space. The selected neighborhood can well reflect the local geometrical structure of face manifold, so that the linear subspace determined by the optimal linear fitting can approximate the local facial geometry with a higher accuracy. In order to reduce the effect of misalignment, a face alignment method based on three facial keypoints is also used. Experimental results show that the proposed face hallucination method outperforms other state-of-the-art methods in terms of visual inspection and objective evaluations, especially when the facial images are compressed and have low-resolution.

How to predict the global instantaneous feeling induced by a facial picture?

Picture selection is a time-consuming task for humans and a real challenge for machines, which have to retrieve complex and subjective information from image pixels. An automated system that infers human feelings from digital portraits would be of great help for profile picture selection, photo album creation or photo editing. In this work, two models of facial pictures evaluation are defined. The first one predicts the overall aesthetic quality of a facial image, and the second one answers the question “Among a set of facial pictures of a given person, on which picture does the person look like the most friendly?”. Aesthetic quality is evaluated by the computation of 15 features that encode low-level statistics in different image regions (face, eyes, and mouth). Relevant features are automatically selected by a feature ranking technique, and the outputs of 4 learning algorithms are fused in order to make a robust and accurate prediction of the image quality. Results are compared with recent works and the proposed algorithm obtains the best performance. The same pipeline is considered to evaluate the likability of a facial picture, with the difference that the estimation is based on high-level attributes such as gender, age, and smile. Performance of these attributes is compared with previous techniques that mostly rely on facial keypoint positions, and it is shown that it is possible to obtain likability predictions that are close to human perception. Finally, a combination of both models that selects a likable facial image of good quality for a given person is described.

Design and learn distinctive features from pore-scale facial keypoints

Establishing correct correspondences between two faces with different viewpoints has played an important role in 3D face reconstruction and other computer-vision applications. Usually, face images are considered to lack sufficient distinctive features to establish a large number of correspondences on uncalibrated images. In this paper, we investigate pore-scale facial features, which are formed from pores, fine wrinkles, and hair. These features have many characteristics that make them suitable for matching facial images under different variations. Using both biological observation and computer-vision consideration, a new framework is devised for pore-scale facial-feature extraction and matching. The matching difficulty under various skin appearances of different subjects and imaging distortion is also analyzed. For further improving the matching performance and tackling distortions such as varying illuminations and unfocused blurring, a pore-to-pore correspondences dataset is established for training a more distinctive and compact descriptor. Experiments are conducted on a face database containing 105 subjects, and the results prove that the pore-scale features are highly distinctive; face images with a minimum resolution of 600×700 (0.4 mega) pixels contain sufficient details to perform a reliable matching in different poses. Generally, our algorithm can establish between 500 and 2000 correct correspondences on a pair of uncalibrated face images of the same person. Furthermore, the proposed methods can be applied to face recognition, 3D reconstruction, etc.

22-6 選択された顔キーポイント特徴に基づく個人依存しにくい喜び表情認識(第22部門 ヒューマンインフォメーション3)

22-6 選択された顔キーポイント特徴に基づく個人依存しにくい喜び表情認識(第22部門 ヒューマンインフォメーション3)

3D landmarking in multiexpression face analysis: a preliminary study on eyebrows and mouth.

The application of three-dimensional (3D) facial analysis and landmarking algorithms in the field of maxillofacial surgery and other medical applications, such as diagnosis of diseases by facial anomalies and dysmorphism, has gained a lot of attention. In a previous work, we used a geometric approach to automatically extract some 3D facial key points, called landmarks, working in the differential geometry domain, through the coefficients of fundamental forms, principal curvatures, mean and Gaussian curvatures, derivatives, shape and curvedness indexes, and tangent map. In this article we describe the extension of our previous landmarking algorithm, which is now able to extract eyebrows and mouth landmarks using both old and new meshes. The algorithm has been tested on our face database and on the public Bosphorus 3D database. We chose to work on the mouth and eyebrows as a separate study because of the role that these parts play in facial expressions. In fact, since the mouth is the part of the face that moves the most and affects mainly facial expressions, extracting mouth landmarks from various facial poses means that the newly developed algorithm is pose-independent. This journal requires that authors assign a level of evidence to each submission to which Evidence-Based Medicine rankings are applicable. This excludes Review Articles, Book Reviews, and manuscripts that concern Basic Science, Animal Studies, Cadaver Studies, and Experimental Studies. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors http://www.springer.com/00266 .

Automatic recognition of facial movement for paralyzed face.

Facial nerve paralysis is a common disease due to nerve damage. Most approaches for evaluating the degree of facial paralysis rely on a set of different facial movements as commanded by doctors. Therefore, automatic recognition of the patterns of facial movement is fundamental to the evaluation of the degree of facial paralysis. In this paper, a novel method named Active Shape Models plus Local Binary Patterns (ASMLBP) is presented for recognizing facial movement patterns. Firstly, the Active Shape Models (ASMs) are used in the method to locate facial key points. According to these points, the face is divided into eight local regions. Then the descriptors of these regions are extracted by using Local Binary Patterns (LBP) to recognize the patterns of facial movement. The proposed ASMLBP method is tested on both the collected facial paralysis database with 57 patients and another publicly available database named the Japanese Female Facial Expression (JAFFE). Experimental results demonstrate that the proposed method is efficient for both paralyzed and normal faces.

3D facial expression recognition using SIFT descriptors of automatically detected keypoints

Methods to recognize humans’ facial expressions have been proposed mainly focusing on 2D still images and videos. In this paper, the problem of person-independent facial expression recognition is addressed using the 3D geometry information extracted from the 3D shape of the face. To this end, a completely automatic approach is proposed that relies on identifying a set of facial keypoints, computing SIFT feature descriptors of depth images of the face around sample points defined starting from the facial keypoints, and selecting the subset of features with maximum relevance. Training a Support Vector Machine (SVM) for each facial expression to be recognized, and combining them to form a multi-class classifier, an average recognition rate of 78.43% on the BU-3DFE database has been obtained. Comparison with competitor approaches using a common experimental setting on the BU-3DFE database shows that our solution is capable of obtaining state of the art results. The same 3D face representation framework and testing database have been also used to perform 3D facial expression retrieval (i.e., retrieve 3D scans with the same facial expression as shown by a target subject), with results proving the viability of the proposed solution.

Pupil Localization for Multi-View Eyeballs by ASM and Eye Gray Distribution

This paper presents a new approach for pupil localization in multi-view eyeballs under ordinary light conditions. There are two key steps. The first step is eye detection, which is obtained by facial key points using Active Shape Mode. Face detection is also used to provide the initial search area and enhance the searching speed for facial key points. The second step is the pupil localization process. In this stage, eye gray distribution obtained by sliding window technique is used to provide the coarse position of pupil, and similar local area analysis method is employed to provide pupil's fine position. A set of experiments on different eyeball positions are also presented. The experimental results show that the proposed method can perform well when the eye keeps normal and narrowed slightly, or gazes left, right and up. While the eye looks down, the localization effect needs to be improved further.

A Novel Smart Household Control System by Computer Vision

For the disable people or the aged people, they need extra help in their daily household life. Various modern devices have been developed to improve their living quality. These devices always have automatic functions which are considered crucial for their independent life. Different to the traditional automatic household appliance such as manually infrared remote-controller and automatic voice-operated switch, this paper presents a novel approach based on computer vision to realize intelligent household control. This proposed system uses Adaboost and ASM algorithm for face detection and feature point calibration. Based on the facial key points, the head postures and mouth states are calculated and then coded into a special data packet. Finally, the packet is sent by an infrared carrier to the intelligent appliance with infrared receiver to control their operation manner. By using computer vision technology with infrared communication, the system is especially effective for the disable people such as the upper-body paraplegic and the people with hand dysfunction.

Automatic localization of facial key-points for 3D face modeling

In view of the key-points acquirement problem in morphable way of 3D face modeling,an automatic localization method was proposed.First of all,according to the demand,the locations of key-points were determined and classified.Secondly,by using the S component in HIS color space,the head was detected.Thirdly,combined with the structure of face features and texture,the features' parts were extracted.Then,using different methods,the 2D coordinates of the different key-points were gained at the features' parts from frontal and side photos respectively.Finally,merging the 2D coordinates from both views,the 3D coordinates of the key-points were obtained.The experimental results show that the proposed method has good performance in speed and accuracy.