Facial Points Research Articles

Facial expression classification using salient pattern driven integrated geometric and textual features

Facial expression classification aims to recognize human emotion via face images. The major challenge of facial expression classification is how to extract discriminative features from the human face images to differentiate various emotions. To tackle this challenge, a new feature extraction approach is proposed in this paper. The proposed approach defines a new set of salient patterns at the facial keypoint locations. This is in contrast to conventional approaches that either represent the whole face image as a regular grid or use local image patches centered at all facial key point locations. Driven by the proposed salient patterns, both the geometric and textual features are extracted and then concatenated and further incorporated into a machine learning framework to perform facial expression classification. The proposed approach is evaluated in the well-known CK + benchmark dataset to demonstrate its superior performance.

RED-Net: A Recurrent Encoder–Decoder Network for Video-Based Face Alignment

We propose a novel method for real-time face alignment in videos based on a recurrent encoder–decoder network model. Our proposed model predicts 2D facial point heat maps regularized by both detection and regression loss, while uniquely exploiting recurrent learning at both spatial and temporal dimensions. At the spatial level, we add a feedback loop connection between the combined output response map and the input, in order to enable iterative coarse-to-fine face alignment using a single network model, instead of relying on traditional cascaded model ensembles. At the temporal level, we first decouple the features in the bottleneck of the network into temporal-variant factors, such as pose and expression, and temporal-invariant factors, such as identity information. Temporal recurrent learning is then applied to the decoupled temporal-variant features. We show that such feature disentangling yields better generalization and significantly more accurate results at test time. We perform a comprehensive experimental analysis, showing the importance of each component of our proposed model, as well as superior results over the state of the art and several variations of our method in standard datasets.

Facial Landmark Detection: A Literature Survey

The locations of the fiducial facial landmark points around facial components and facial contour capture the rigid and non-rigid facial deformations due to head movements and facial expressions. They are hence important for various facial analysis tasks. Many facial landmark detection algorithms have been developed to automatically detect those key points over the years, and in this paper, we perform an extensive review of them. We classify the facial landmark detection algorithms into three major categories: holistic methods, Constrained Local Model (CLM) methods, and the regression-based methods. They differ in the ways to utilize the facial appearance and shape information. The holistic methods explicitly build models to represent the global facial appearance and shape information. The CLMs explicitly leverage the global shape model but build the local appearance models. The regression-based methods implicitly capture facial shape and appearance information. For algorithms within each category, we discuss their underlying theories as well as their differences. We also compare their performances on both controlled and in the wild benchmark datasets, under varying facial expressions, head poses, and occlusion. Based on the evaluations, we point out their respective strengths and weaknesses. There is also a separate section to review the latest deep learning-based algorithms. The survey also includes a listing of the benchmark databases and existing software. Finally, we identify future research directions, including combining methods in different categories to leverage their respective strengths to solve landmark detection "in-the-wild".

Deep Recurrent Regression for Facial Landmark Detection

We propose a novel end-to-end deep architecture for face landmark detection, based on a deep convolutional and deconvolutional network followed by carefully designed recurrent network structures. The pipeline of this architecture consists of three parts. Through the first part, we encode an input face image to resolution-preserved deconvolutional feature maps via a deep network with stacked convolutional and deconvolutional layers. Then, in the second part, we estimate the initial coordinates of the facial key points by an additional convolutional layer on top of these deconvolutional feature maps. In the last part, by using the deconvolutional feature maps and the initial facial key points as input, we refine the coordinates of the facial key points by a recurrent network that consists of multiple long short-term memory components. Extensive evaluations on several benchmark data sets show that the proposed deep architecture has superior performance against the state-of-the-art methods.

Lip Symmetry Evaluation Of The Gentur's Method Compared To Fisher Techniques For Unilateral Cleft Lip Repair


 
 
 
 Background : Cleft lip and palate are the most common congenital anomalies that were found in plastic surgery. There are so many techniques for unilateral cleft lip repair. Rotation-advancement method by Gentur based on Millard technique has become the most widely used in unilateral cleft lip repair in RSCM. The Fisher technique repair is a modified technique based on approximation of anatomical subunit of the lip. The purpose of this study is to objectively compare and evaluate the lip symmetry of these two techniques.
 Method : Two senior board-certified plastic surgeons will perform different surgical techniques for the unilateral cleft lip: rotation-advancement technique by Gentur and Fisher technique. This study prospectively analyzed preoperative and postoperative of randomized single blinded patients who underwent unilateral cleft lip repair performed by each surgeon in 2016. Using caliper, facial points on the cleft and non-cleft sides were measured, including height and symmetry of Cupid’s bow, width and height of the nasal vestibule, height of the vermilion, and alar base position. Ratios of cleft side to non cleft side measurements were calculated to standardize comparisons between patients.
 Result : From July-October 2016, 14 patients performed surgery as preliminary data, showed that there are statistically difference in length of design and surgery time. Preoperative, comparable of cupid’s bow and vermillion showed statistically difference. Although, we found no statistically difference in postoperative ratio.
 Conclusion : Lip symmetry outcomes after cheiloplasty procedure are same between Gentur method and Fisher technique.
 
 
 

Automated Analysis and Prediction of Job Interview Performance

We present a computational framework for automatically quantifying verbal and nonverbal behaviors in the context of job interviews. The proposed framework is trained by analyzing the videos of 138 interview sessions with 69 internship-seeking undergraduates at the Massachusetts Institute of Technology (MIT). Our automated analysis includes facial expressions (e.g., smiles, head gestures, facial tracking points), language (e.g., word counts, topic modeling), and prosodic information (e.g., pitch, intonation, and pauses) of the interviewees. The ground truth labels are derived by taking a weighted average over the ratings of nine independent judges. Our framework can automatically predict the ratings for interview traits such as excitement, friendliness, and engagement with correlation coefficients of 0.70 or higher, and can quantify the relative importance of prosody, language, and facial expressions. By analyzing the relative feature weights learned by the regression models, our framework recommends to speak more fluently, use fewer filler words, speak as “we” (versus “I”), use more unique words, and smile more. We also find that the students who were rated highly while answering the first interview question were also rated highly overall (i.e., first impression matters). Finally, our MIT Interview dataset is available to other researchers to further validate and expand our findings.

Grey Wolf optimisation‐based feature selection and classification for facial emotion recognition

The channels used to convey the human emotions consider actions, behaviours, poses, facial expressions, and speech. An immense research has been carried out to analyse the relationship between the facial emotions and these channels. The goal of this study is to develop a system for Facial Emotion Recognition (FER) that can analyse the elemental facial expressions of human, such as normal, smile, sad, surprise, anger, fear, and disgust. The recognition process of the proposed FER system is categorised into four processes, namely pre-processing, feature extraction, feature selection, and classification. After preprocessing, scale invariant feature transform -based feature extraction method is used to extract the features from the facial point. Further, a meta-heuristic algorithm called Grey Wolf optimisation (GWO) is used to select the optimal features. Subsequently, GWO-based neural network (NN) is used to classify the emotions from the selected features. Moreover, an effective performance analysis of the proposed as well as the conventional methods such as convolutional neural network, NN-Levenberg-Marquardt, NN-Gradient Descent, NN-Evolutionary Algorithm, NN-firefly, and NN-Particle Swarm Optimisation is provided by evaluating few performance measures and thereby, the effectiveness of the proposed strategy over the conventional methods is validated.

Explicit Shape Regression With Characteristic Number for Facial Landmark Localization

Robustly localizing facial landmarks plays a very important role in many multimedia and vision applications. Most recently proposed regression-based methods prevailing in the community lack explicit shape constraints for faces and require a large number of facial images to cover great appearance variations. To address these limitations, this paper introduces a novel projective invariant called characteristic number (CN) to explicitly characterize the intrinsic geometries of facial points shared by human faces. It can be verified that the shape priors from CN are inherently invariant to pose changes. By further developing a shape-to-gradient regression framework, we provide a robust and efficient landmark detector for facial images in the wild. The computation of our model can be successfully addressed by learning the descent directions using point-CN pairs without the need for large collections for appearance training. As a nontrivial byproduct, this paper also builds a face dataset, where each face has 15 well-defined viewpoints (poses) to quantitatively analyze the effects of different poses on localization methods. Extensive experiments on challenging benchmarks and our newly built dataset demonstrate the effectiveness of our proposed detector against other state-of-the-art approaches.

Combining Data-Driven and Model-Driven Methods for Robust Facial Landmark Detection

Facial landmark detection is an important yet challenging task for real-world computer vision applications. This paper proposes an effective and robust approach for facial landmark detection by combining data- and model-driven methods. First, a fully convolutional network (FCN) is trained to compute response maps of all facial landmark points. Such a data-driven method could make full use of holistic information in a facial image for global estimation of facial landmarks. After that, the maximum points in the response maps are fitted with a pre-trained point distribution model (PDM) to generate the initial facial shape. This model-driven method is able to correct the inaccurate locations of outliers by considering the shape prior information. Finally, a weighted version of regularized landmark mean-shift (RLMS) is employed to fine-tune the facial shape iteratively. This estimation-correction-tuning process perfectly combines the advantages of the global robustness of the data-driven method (FCN), outlier correction capability of the model-driven method (PDM), and non-parametric optimization of RLMS. Results of extensive experiments demonstrate that our approach achieves state-of-the-art performances on challenging data sets, including 300W, AFLW, AFW, and COFW. The proposed method is able to produce satisfying detection results on face images with exaggerated expressions, large head poses, and partial occlusions.

Emotion Recognition from Facial Expression Based on Fiducial Points Detection and using Neural Network

The importance of emotion recognition lies in the role that emotions play in our everyday lives. Emotions have a strong relationship with our behavior. Thence, automatic emotion recognition, is to equip the machine of this human ability to analyze, and to understand the human emotional state, in order to anticipate his intentions from facial expression. In this paper, a new approach is proposed to enhance accuracy of emotion recognition from facial expression, which is based on input features deducted only from fiducial points. The proposed approach consists firstly on extracting 1176 dynamic features from image sequences that represent the proportions of euclidean distances between facial fiducial points in the first frame, and faicial fiducial points in the last frame. Secondly, a feature selection method is used to select only the most relevant features from them. Finally, the selected features are presented to a Neural Network (NN) classifier to classify facial expression input into emotion. The proposed approach has achieved an emotion recognition accuracy of 99% on the CK+ database, 84.7% on the Oulu-CASIA VIS database, and 93.8% on the JAFFE database.

An Evaluation of Mandibular Dental and Basal Arch Dimensions in Class I and Class II Division 1 Adult Syrian Patients using Cone-beam Computed Tomography

Aim: The aim of this study is (1) to inspect any difference in mandibular arch widths between males and females in class I and class II division 1 (class malocclusions using cone-beam computed tomography (CBCT), (2) to compare the mandibular dental and basal widths between the two groups, and (3) to investigate any possible correlation between dental and basal arch widths in both groups. Materials and methods: The CBCT images of 68 patients aged between 18 and 25 years consisted of 34 class I (17 males and 17 females) and 34 class (17 males and 17 females) who were recruited at the Department of Orthodontics, University of Damascus Dental School (Syria). Using on-demand three-dimensional (3D) on axial views, facial axis points for dental measurements and basal bone center (BBC) points for basal measurements were identified on lower canines and first molars. Dental and basal intercanine width (ICW) and intermolar width (IMW) were measured. Results: Independent t-test showed a statistically significant difference between males and females in several variables in both groups and a statistically significant difference between class I and class groups in the basal ICW for both genders and in the dental ICW for females only (p < 0.05). In class I group, Pearson's correlation coefficients between dental and basal measurements showed a strong correlation in the IMW for both genders (r > 0.73; p < 0.01) and a moderate correlation in females' ICW (r = 0.67; p < 0.01). In the class group, a moderate correlation in females' IMW (r = 0.67; p < 0.01) was found. Conclusion: Females compared with males had narrower dimensions. Class I patients had larger ICW than class II-1 patients in all measurements and had narrower IMW than class in most measurements for both genders. There were moderate-to-strong correlations between dental and basal dimensions. BBC points might be landmarks that accurately represent the basal bone arch. Clinical significance: CBCT-based assessments of dental and basal arch dimensions provide a great opportunity to accurately evaluate these aspects, to enhance clinicians' decisions regarding proper tooth movements, and to achieve good dentoalveolar intra-arch harmony. Keywords: Basal arch, Class I, Class Cone-beam computed tomography, Dental arch, Intercanine width, Intermolar width.

Real-time Driver Drowsiness Detection for Android Application Using Deep Neural Networks Techniques

Road crashes and related forms of accidents are a common cause of injury and death among the human population. According to 2015 data from the World Health Organization, road traffic injuries resulted in approximately 1.25 million deaths worldwide, i.e. approximately every 25 seconds an individual will experience a fatal crash. While the cost of traffic accidents in Europe is estimated at around 160 billion Euros, driver drowsiness accounts for approximately 100,000 accidents per year in the United States alone as reported by The American National Highway Traffic Safety Administration (NHTSA). In this paper, a novel approach towards real-time drowsiness detection is proposed. This approach is based on a deep learning method that can be implemented on Android applications with high accuracy. The main contribution of this work is the compression of heavy baseline model to a lightweight model. Moreover, minimal network structure is designed based on facial landmark key point detection to recognize whether the driver is drowsy. The proposed model is able to achieve an accuracy of more than 80%.

Cross-Domain Color Facial Expression Recognition Using Transductive Transfer Subspace Learning

Facial expression recognition across domains, e.g., training and testing facial images come from different facial poses, is very challenging due to the different marginal distributions between training and testing facial feature vectors. To deal with such challenging cross-domain facial expression recognition problem, a novel transductive transfer subspace learning method is proposed in this paper. In this method, a labelled facial image set from source domain is combined with an unlabelled auxiliary facial image set from target domain to jointly learn a discriminative subspace and make the class labels prediction of the unlabelled facial images, where a transductive transfer regularized least-squares regression (TTRLSR) model is proposed to this end. Then, based on the auxiliary facial image set, we train a SVM classifier for classifying the expressions of other facial images in the target domain. Moreover, we also investigate the use of color facial features to evaluate the recognition performance of the proposed facial expression recognition method, where color scale invariant feature transform (CSIFT) features associated with 49 landmark facial points are extracted to describe each color facial image. Finally, extensive experiments on BU-3DFE and Multi-PIE multiview color facial expression databases are conducted to evaluate the cross-database & cross-view facial expression recognition performance of the proposed method. Comparisons with state-of-the-art domain adaption methods are also included in the experiments. The experimental results demonstrate that the proposed method achieves much better recognition performance compared with the state-of-the-art methods.

Facial expression recognition based on dual-feature fusion and improved random forest classifier

Facial expression recognition (FER) is an important means for machines to understand the changes in the facial expression of human beings. Expression recognition using single-modal facial images, such as gray scale, may suffer from illumination changes and the lack of detailed expression-related information. In this study, multi-modal facial images, such as facial gray scale, depth, and local binary pattern (LBP), are used to recognize six basic facial expressions, namely, happiness, sadness, anger, disgust, fear, and surprise. Facial depth images are used for robust face detection initially. The deep geometric feature is represented by point displacement and angle variation in facial landmark points with the help of depth information. The local appearance feature, which is obtained by concatenating LBP histograms of expression-prominent patches, is utilized to recognize those expression changes that are difficult to capture by only the geometric changes. Thereafter, an improved random forest classifier based on feature selection is used to recognize different facial expressions. Results of comparative evaluations in benchmarking datasets show that the proposed method outperforms several state-of-the-art FER approaches that are based on hand-crafted features. The capability of the proposed method is comparable to that of the popular convolutional neural-network-based FER approach but with fewer demands for training data and a high-performance hardware platform.

Derivation of simple rules for complex flow vector fields on the lower part of the human face for robot face design

It is quite difficult for android robots to replicate the numerous and various types of human facial expressions owing to limitations in terms of space, mechanisms, and materials. This situation could be improved with greater knowledge regarding these expressions and their deformation rules, i.e. by using the biomimetic approach. In a previous study, we investigated 16 facial deformation patterns and found that each facial point moves almost only in its own principal direction and different deformation patterns are created with different combinations of moving lengths. However, the replication errors caused by moving each control point of a face in only their principal direction were not evaluated for each deformation pattern at that time. Therefore, we calculated the replication errors in this study using the second principal component scores of the 16 sets of flow vectors at each point on the face. More than 60% of the errors were within 1 mm, and approximately 90% of them were within 3 mm. The average error was 1.1 mm. These results indicate that robots can replicate the 16 investigated facial expressions with errors within 3 mm and 1 mm for about 90% and 60% of the vectors, respectively, even if each point on the robot face moves in only its own principal direction. This finding seems promising for the development of robots capable of showing various facial expressions because significantly fewer types of movements than previously predicted are necessary.

Modern physiognomy: an investigation on predicting personality traits and intelligence from the human face

The human behavior of evaluating other individuals with respect to their personality traits and intelligence by evaluating their faces plays a crucial role in human relations. These trait judgments might influence important social outcomes in our lives such as elections and court sentences. Previous studies have reported that human can make valid inferences for at least four personality traits. In addition, some studies have demonstrated that facial trait evaluation can be learned using machine learning methods accurately. In this work, we experimentally explore whether self-reported personality traits and intelligence can be predicted reliably from a facial image. More specifically, the prediction problem is separately cast in two parts: a classification task and a regression task. A facial structural feature is constructed from the relations among facial salient points, and an appearance feature is built by five texture descriptors. In addition, a minutia-based fingerprint feature from a fingerprint image is also explored. The classification results show that the personality traits Rule-consciousness and Vigilance can be predicted reliably, and that the traits of females can be predicted more accurately than those of male. However, the regression experiments show that it is difficult to predict scores for individual personality traits and intelligence. The residual plots and the correlation results indicate no evident linear correlation between the measured scores and the predicted scores. Both the classification and the regression results reveal that Rule-consciousness and Tension can be reliably predicted from the facial features, while Social boldness gets the worst prediction results. The experiments results show that it is difficult to predict intelligence from either the facial features or the fingerprint feature, a finding that is in agreement with previous studies.

Modification of Chin Projection and Aesthetics With OnabotulinumtoxinA Injection.

Because of the soft-tissue envelope and the dimensional complexity of the muscular contraction of the mentalis muscle, the projection and shape of the chin is not determined by only the bony projection. In a subset of patients with a seemingly underprojected chin, a hyperactive, high-riding mentalis muscle contributes to a blunted chin contour. To evaluate the use of onabotulinumtoxinA (botulinum toxin A) for improving chin aesthetics in patients with an underprojected bony chin and a high-riding hyperactive mentalis muscle. This case series and photographic analysis included 11 patients presenting with an underprojected bony chin and a high-riding hyperactive mentalis muscle at a private facial plastic surgery practice from August 25, 2006, to November 10, 2012. Data were analyzed from November 13, 2012, to April 9, 2013. Injection with 12 to 15 U of onabotulinumtoxinA into the mentalis muscle. Photographic analysis of the vertical and horizontal positions of the pogonion relative to fixed facial points before and after injection. A patient satisfaction scale was also used to assess improvement in overall chin aesthetic. Eleven patients (3 men and 8 women; mean [SD] age, 46.3 [16.4] years) participated in the study, including 2 who had undergone prior chin implantation. The vertical position of the pogonion was more inferior after injection in 10 of 11 patients (mean [SD] vertical position, 1.36 [0.18] preinjection and 1.44 [0.18] postinjection; P = .005). Although the horizontal position of the pogonion changed in all patients, this change was not statistically significant (mean [SD] horizontal position, 0.11 [0.13] preinjection and 0.14 [0.13] postinjection; P = .32). All patients experienced improvement in their overall chin aesthetic and a subjective decrease in chin tension. Chin contour was improved, with soft-tissue volume overall displaced more inferiorly with a rounded rather than blunted appearance. No adverse effects were reported after injection. OnabotulinumtoxinA is effective at improving chin aesthetics by altering the position of the pogonion. In this first photographic analysis to date of the influence of onabotulinumtoxinA treatment, patients demonstrated a measurable change in the position of the pogonion, in addition to improvements to the chin contour on profile. This nonsurgical mentoplasty may be used as an independent procedure or as an adjunct to optimize traditional chin augmentation in carefully selected patients. 4.

Compressing and Accelerating Neural Network for Facial Point Localization

State-of-the-art deep neural networks (DNNs) have greatly improved the accuracy of facial landmark localization. However, DNN models usually have a huge number of parameters which cause high memory cost and computational complexity. To address this issue, a novel method is proposed to compress and accelerate large DNN models while maintaining the performance. It includes three steps: (1) importance-based pruning: compared with traditional connection pruning, weight correlations are introduced to find and prune unimportant neurons or connections. (2) Product quantization: product quantization helps to enforce weights shared. With the same size codebook, product quantization can achieve higher compression rate than scalar quantization. (3) Network retraining: to reduce compression difficulty and performance degradation, the network is retrained iteratively after compressing one layer at a time. Besides, all pooling layers are removed and the strides of their neighbor convolutional layers are increased to accelerate the network simultaneously. The experimental results of compressing a VGG-like model demonstrate the effectiveness of our proposed method, which achieves 26 × compression and 4 × acceleration while the root mean squared error (RMSE) increases by just 3.6%.

Face alignment in-the-wild: A Survey

Over the last two decades, face alignment or localizing fiducial facial points on 2D images has received increasing attention owing to its comprehensive applications in automatic face analysis. However, such a task has proven extremely challenging in unconstrained environments due to many confounding factors, such as pose, occlusions, expression and illumination. While numerous techniques have been developed to address these challenges, this problem is still far away from being solved. In this survey, we present an up-to-date critical review of the existing literatures on face alignment, focusing on those methods addressing overall difficulties and challenges of this topic under uncontrolled conditions. Specifically, we categorize existing face alignment techniques, present detailed descriptions of the prominent algorithms within each category, and discuss their advantages and disadvantages. Furthermore, we organize special discussions on the practical aspects of face alignment in-the-wild, towards the development of a robust face alignment system. In addition, we show performance statistics of the state of the art, and conclude this paper with several promising directions for future research.

Facial landmark localization by enhanced convolutional neural network

Facial landmark localization is important to many facial recognition and analysis tasks, such as face attributes analysis, head pose estimation, 3D face modeling, and facial expression analysis. In this paper, we propose a new approach to localizing landmarks in facial image by deep convolutional neural network (DCNN). We make two enhancements on the CNN to adapt it to the feature localization task as follows. First, we replace the commonly used max pooling by depth-wise convolution to obtain better localization performance. Second, we define a response map for each facial points as a 2D probability map indicating the presence likelihood, and train our model with a KL divergence loss. To obtain robust localization results, our approach first takes the expectations of the response maps of enhanced CNN and then applies auto-encoder model to the global shape vector, which is effective to rectify the outlier points by the prior global landmark configurations. The proposed ECNN method achieves 5.32% mean error on the experiments on the 300-W dataset, which is comparable to the state-of-the-art performance on this standard benchmark, showing the effectiveness of our methods.