Facial Landmark Localization Research Articles

Efficient Facial Landmark Localization Based on Binarized Neural Networks

Facial landmark localization is a significant yet challenging computer vision task, whose accuracy has been remarkably improved due to the successful application of deep Convolutional Neural Networks (CNNs). However, CNNs require huge storage and computation overhead, thus impeding their deployment on computationally limited platforms. In this paper, to the best of our knowledge, it is the first time that an efficient facial landmark localization is implemented via binarized CNNs. We introduce a new network architecture to calculate the binarized models, referred to as Amplitude Convolutional Networks (ACNs), based on the proposed asynchronous back propagation algorithm. We can efficiently recover the full-precision filters only using a single factor in an end-to-end manner, and the efficiency of CNNs for facial landmark localization is further improved by the extremely compressed 1-bit ACNs. Our ACNs reduce the storage space of convolutional filters by a factor of 32 compared with the full-precision models on dataset LFW+Webface, CelebA, BioID and 300W, while achieving a comparable performance to the full-precision facial landmark localization algorithms.

CenterFace: Joint Face Detection and Alignment Using Face as Point

Face detection and alignment in unconstrained environment is always deployed on edge devices which have limited memory storage and low computing power. This paper proposes a one-stage method named CenterFace to simultaneously predict facial box and landmark location with real-time speed and high accuracy. The proposed method also belongs to the anchor-free category. This is achieved by (a) learning face existing possibility by the semantic maps, (b) learning bounding box, offsets, and five landmarks for each position that potentially contains a face. Specifically, the method can run in real time on a single CPU core and 200 FPS using NVIDIA 2080TI for VGA-resolution images and can simultaneously achieve superior accuracy (WIDER FACE Val/Test-Easy: 0.935/0.932, Medium: 0.924/0.921, Hard: 0.875/0.873, and FDDB discontinuous: 0.980 and continuous: 0.732).

An improved boundary-aware face alignment using stacked dense U-Nets

Facial landmark localization is still a challenge task in the unconstrained environment with influences of significant variation conditions such as facial pose, shape, expression, illumination, and occlusions. In this work, we present an improved boundary-aware face alignment method by using stacked dense U-Nets. The proposed method consists of two stages: a boundary heatmap estimation stage to learn the facial boundary lines and a facial landmark localization stage to predict the final face alignment result. With the constraint of boundary lines, facial landmarks are unified as a whole facial shape. Hence, the unseen landmarks in a shape with occlusions can be better estimated by message passing with other landmarks. By introducing the stacked dense U-Nets for feature extraction, the capacity of the model is improved. Experiments and comparisons on public datasets show that the proposed method obtains better performance than the baselines, especially for facial images with large pose variation, shape variation, and occlusions.

Multi-pose face recognition using Cascade Alignment Network and incremental clustering

The variety of face pose has imposed significant challenges on the existing face recognition. A novel approach has been proposed for multi-pose face recognition using LSTM and CNN-based cascade alignment network (LCCAN) with incremental clustering strategy. LCCAN is used to leverage the memory function of LSTM and explore the spatial contextual information between facial landmarks. The coarse facial landmark locations have been gotten at first in LCCAN. CNNs are utilized to refine facial landmarks as mentioned above. Then, the facial landmarks are used as facial orientation descriptors. In order to fit in with the dynamic updating of diversified facial poses, dynamic adaptive incremental clustering strategy with correntropy-induced metric has been developed to construct facial pose pool. Multi-pose face recognition is implemented by building face recognition classification models on different poses. Experimental results demonstrate that the effectiveness of the proposed method is superior to the investigated state of the arts.

Deep 3D Facial Landmark Localization on position maps

3D facial landmark detection is a crucial step for many computer vision applications, such as 3D facial expression analysis, 3D face recognition, and 3D cephalometry. Pose variations, expression changes and self-occlusion yet make 3D facial landmark detection a very challenging task. In this paper, we propose a novel 3D Facial Landmark Localization Network (3DLLN), which is robust to the above challenges. Different from existing methods, the proposed 3DLLN utilizes the position maps as an intermediate representation, from which 3DLLN detects 3D landmark coordinates. Further, we demonstrate the usage of a deep regression architecture to improve the accuracy and robustness of a large number of landmarks. The proposed scheme is evaluated on two public datasets FRGCv2 subset and BU-3DFE, and is shown to achieve state-of-the-art results.

Face Identification using Histogram

Various experiments or methods can be used for face recognition and detection however two of the main contain an experiment that evaluates the impact of facial landmark localization in the face recognition performance and the second experiment evaluates the impact of extracting the HOG from a regular grid and at multiple scales. We observe the question of feature sets for robust visual object recognition. The Histogram of Oriented Gradients outperform other existing methods like edge and gradient based descriptors. We observe the influence of each stage of the computation on performance, concluding that fine-scale gradients, relatively coarse spatial binning, fine orientation binning and high- quality local contrast normalization in overlapping descriptor patches are all important for good results. Comparative experiments show that though HOG is simple feature descriptor, the proposed HOG feature achieves good results with much lower computational time.

A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation

Action Units (AUs) are geometrically-based atomic facial muscle movements known to produce appearance changes at specific facial locations. Motivated by this observation we propose a novel AU modelling problem that consists of jointly estimating their localisation and intensity. To this end, we propose a simple yet efficient approach based on Heatmap Regression that merges both problems into a single task. A Heatmap models whether an AU occurs or not at a given spatial location. To accommodate the joint modelling of AUs intensity, we propose variable size heatmaps, with their amplitude and size varying according to the labelled intensity. Using Heatmap Regression, we can inherit from the progress recently witnessed in facial landmark localisation. Building upon the similarities between both problems, we devise a transfer learning approach where we exploit the knowledge of a network trained on large-scale facial landmark datasets. In particular, we explore different alternatives for transfer learning through a) fine-tuning, b) adaptation layers, c) attention maps, and d) reparametrisation. Our approach effectively inherits the rich facial features produced by a strong face alignment network, with minimal extra computational cost. We empirically validate that our system sets a new state-of-the-art on three popular datasets, namely BP4D, DISFA, and FERA2017.

FAN-Face: a Simple Orthogonal Improvement to Deep Face Recognition

It is known that facial landmarks provide pose, expression and shape information. In addition, when matching, for example, a profile and/or expressive face to a frontal one, knowledge of these landmarks is useful for establishing correspondence which can help improve recognition. However, in prior work on face recognition, facial landmarks are only used for face cropping in order to remove scale, rotation and translation variations. This paper proposes a simple approach to face recognition which gradually integrates features from different layers of a facial landmark localization network into different layers of the recognition network. To this end, we propose an appropriate feature integration layer which makes the features compatible before integration. We show that such a simple approach systematically improves recognition on the most difficult face recognition datasets, setting a new state-of-the-art on IJB-B, IJB-C and MegaFace datasets.

Toward an Automatic System for Computer-Aided Assessment in Facial Palsy.

Importance: Quantitative assessment of facial function is challenging, and subjective grading scales such as House-Brackmann, Sunnybrook, and eFACE have well-recognized limitations. Machine learning (ML) approaches to facial landmark localization carry great clinical potential as they enable high-throughput automated quantification of relevant facial metrics from photographs and videos. However, the translation from research settings to clinical application still requires important improvements. Objective: To develop a novel ML algorithm for fast and accurate localization of facial landmarks in photographs of facial palsy patients and utilize this technology as part of an automated computer-aided diagnosis system. Design, Setting, and Participants: Portrait photographs of 8 expressions obtained from 200 facial palsy patients and 10 healthy participants were manually annotated by localizing 68 facial landmarks in each photograph and by 3 trained clinicians using a custom graphical user interface. A novel ML model for automated facial landmark localization was trained using this disease-specific database. Algorithm accuracy was compared with manual markings and the output of a model trained using a larger database consisting only of healthy subjects. Main Outcomes and Measurements: Root mean square error normalized by the interocular distance (NRMSE) of facial landmark localization between prediction of ML algorithm and manually localized landmarks. Results: Publicly available algorithms for facial landmark localization provide poor localization accuracy when applied to photographs of patients compared with photographs of healthy controls (NRMSE, 8.56 ± 2.16 vs. 7.09 ± 2.34, p ≪ 0.01). We found significant improvement in facial landmark localization accuracy for the facial palsy patient population when using a model trained with a relatively small number photographs (1440) of patients compared with a model trained using several thousand more images of healthy faces (NRMSE, 6.03 ± 2.43 vs. 8.56 ± 2.16, p ≪ 0.01). Conclusions and Relevance: Retraining a computer vision facial landmark detection model with fewer than 1600 annotated images of patients significantly improved landmark detection performance in frontal view photographs of this population. The new annotated database and facial landmark localization model represent the first steps toward an automatic system for computer-aided assessment in facial palsy. Level of Evidence: 4.

A Local-Global Aggregate Network for Facial Landmark Localization

Facial landmark localization plays a critical role in many face analysis tasks. In this paper, we present a novel local-global aggregate network (LGA-Net) for robust facial landmark localization of faces in the wild. The network consists of two convolutional neural network levels which aggregate local and global information for better prediction accuracy and robustness. Experimental results show our method overcomes typical problems of cascaded networks and outperforms state-of-the-art methods on the 300-W [1] benchmark.

Robust Facial Landmark Detection via Heatmap-Offset Regression.

Facial landmark detection aims at localizing multiple keypoints for a given facial image, which usually suffers from variations caused by arbitrary pose, diverse facial expression and partial occlusion. In this paper, we develop a two-stage regression network for facial landmark detection on unconstrained conditions. Our model consists of a Structural Hourglass Network (SHN) for detecting the initial locations of all facial landmarks based on heatmap generation, and a Global Constraint Network (GCN) for further refining the detected locations based on offset estimation. Specifically, SHN introduces an improved Inception-ResNet unit as basic building block, which can effectively improve the receptive field and learn contextual feature representations. In the meanwhile, a novel loss function with adaptive weight is proposed to make the whole model focus on the hard landmarks precisely. GCN attempts to explore the spatial contextual relationship between facial landmarks and refine the initial locations of facial landmarks by optimizing the global constraint. Moreover, we develop a pre-processing network to generate features with different scales, which will be transmitted to SHN and GCN for effective feature representations. Different from existing models, the proposed method realizes the heatmap-offset framework, which combines the outputs of heatmaps generated by SHN and coordinates estimated by GCN, to obtain an accurate prediction. The extensive experimental results on several challenging datasets, including 300W, COFW, AFLW, and 300-VW confirm that our method achieve competitive performance compared with the state-of-the-art algorithms.

Rectified Wing Loss for Efficient and Robust Facial Landmark Localisation with Convolutional Neural Networks

Efficient and robust facial landmark localisation is crucial for the deployment of real-time face analysis systems. This paper presents a new loss function, namely Rectified Wing (RWing) loss, for regression-based facial landmark localisation with Convolutional Neural Networks (CNNs). We first systemically analyse different loss functions, including L2, L1 and smooth L1. The analysis suggests that the training of a network should pay more attention to small-medium errors. Motivated by this finding, we design a piece-wise loss that amplifies the impact of the samples with small-medium errors. Besides, we rectify the loss function for very small errors to mitigate the impact of inaccuracy of manual annotation. The use of our RWing loss boosts the performance significantly for regression-based CNNs in facial landmarking, especially for lightweight network architectures. To address the problem of under-representation of samples with large pose variations, we propose a simple but effective boosting strategy, referred to as pose-based data balancing. In particular, we deal with the data imbalance problem by duplicating the minority training samples and perturbing them by injecting random image rotation, bounding box translation and other data augmentation strategies. Last, the proposed approach is extended to create a coarse-to-fine framework for robust and efficient landmark localisation. Moreover, the proposed coarse-to-fine framework is able to deal with the small sample size problem effectively. The experimental results obtained on several well-known benchmarking datasets demonstrate the merits of our RWing loss and prove the superiority of the proposed method over the state-of-the-art approaches.

Simultaneous multi-descent regression and feature learning for facial landmarking in depth images

Face alignment (or facial landmarking) is an important task in many face-related applications, ranging from registration, tracking, and animation to higher-level classification problems such as face, expression, or attribute recognition. While several solutions have been presented in the literature for this task so far, reliably locating salient facial features across a wide range of posses still remains challenging. To address this issue, we propose in this paper a novel method for automatic facial landmark localization in 3D face data designed specifically to address appearance variability caused by significant pose variations. Our method builds on recent cascaded regression-based methods to facial landmarking and uses a gating mechanism to incorporate multiple linear cascaded regression models each trained for a limited range of poses into a single powerful landmarking model capable of processing arbitrary-posed input data. We develop two distinct approaches around the proposed gating mechanism: (1) the first uses a gated multiple ridge descent mechanism in conjunction with established (hand-crafted) histogram of gradients features for face alignment and achieves state-of-the-art landmarking performance across a wide range of facial poses and (2) the second simultaneously learns multiple-descent directions as well as binary features that are optimal for the alignment tasks and in addition to competitive landmarking results also ensures extremely rapid processing. We evaluate both approaches in rigorous experiments on several popular datasets of 3D face images, i.e., the FRGCv2 and Bosphorus 3D face datasets and image collections F and G from the University of Notre Dame. The results of our evaluation show that both approaches compare favorably to the state-of-the-art, while exhibiting considerable robustness to pose variations.

Facial Landmark Machines: A Backbone-Branches Architecture With Progressive Representation Learning

Facial landmark localization plays a critical role in face recognition and analysis. In this paper, we propose a novel cascaded backbone-branches fully convolutional neural network (BB-FCN) for rapidly and accurately localizing facial landmarks in unconstrained and cluttered settings. Our proposed BB-FCN generates facial landmark response maps directly from raw images without any preprocessing. BB-FCN follows a coarse-to-fine cascaded pipeline, which consists of a backbone network to roughly detect the locations of all facial landmarks and one branch network for each type of detected landmark to further refine its location. Furthermore, to facilitate the facial landmark localization under unconstrained settings, we propose a large-scale benchmark named SYSU16K, which contains 16 000 faces with large variations in pose, expression, illumination, and resolution. Extensive experimental evaluations demonstrate that our proposed BB-FCN can significantly outperform the state of the art under both constrained (i.e., within detected facial regions only) and unconstrained settings. We further confirm that high-quality facial landmarks localized with our proposed network can also improve the precision and recall of face detection.

Facial expression recognition based on Electroencephalogram and facial landmark localization.

Facial expression recognition plays an essential role in affective computing, mental illness diagnosis and rehabilitation. Therefore, facial expression recognition has attracted more and more attention over the years. The goal of this paper was to improve the accuracy of the Electroencephalogram (EEG)-based facial expression recognition. In this paper, we proposed a fusion facial expression recognition method based on EEG and facial landmark localization. The EEG signal processing and facial landmark localization are the two key parts. The raw EEG signals is preprocessed by discrete wavelet transform (DWT). The energy feature vector is composed of energy features of the reconstructed signal. For facial landmark localization, images of the subjects' facial expression are processed by facial landmark localization, and the facial features are calculated by landmarks of essence. In this research, we fused the energy feature vector and facial feature vector, and classified the fusion feature vector with the support vector machine (SVM). From the experiments, we found that the accuracy of facial expression recognition was increased 4.16% by fusion method (86.94 ± 4.35%) than EEG-based facial expression recognition (82.78 ± 5.78%). The proposed method obtain a higher accuracy and a stronger generalization capability.

Robust Facial Landmark Localization Based on Two-Stage Cascaded Pose Regression

In this paper, we propose a two-stage cascaded pose regression for facial landmark localization under occlusion. In the first stage, a global cascaded pose regression with robust initialization is performed to get localization results for the original face and its mirror image. The localization difference between the original image and the mirror image is used to determine whether the localization of each landmark is reliable, while unreliable localization with a large difference can be adjusted. In the second stage, the global results are divided into four parts, which are further refined by local regressions. Finally, the four refined local results are integrated and adjusted to get the final output.

Sign-correlation cascaded regression for face alignment

Face alignment plays an important role in many applications such as face recognition and face reconstruction. Current regression based approaches can ease the multi-pose face alignment problem, but they fail to deal with the multiple local minima problem directly. To improve the performance of multi-pose facial landmark localization, in this paper we propose a sign correlation supervised descent method (SC-SDM) based on a nonlinear optimization theory. SC-SDM analyses the sign correlation between features and shapes and project both of them into a mutual sign-correlation subspace. By partitioning the whole multi-pose samples into a series of pose-consistent subsets, a group of models are learned from each subset. The experiments using the public multi-pose datasets has validated the partition and proved that SC-SDM can accurately separate samples into pose-consistent subsets, which reveals their latent relationships to pose. The comparison with state-of-the-art methods demonstrates that SC-SDM outperforms them, especially in uncontrolled conditions with various poses.

Deep multi-center learning for face alignment

Facial landmarks are highly correlated with each other since a certain landmark can be estimated by its neighboring landmarks. Most of the existing deep learning methods only use one fully-connected layer called shape prediction layer to estimate the locations of facial landmarks. In this paper, we propose a novel deep learning framework named Multi-Center Learning with multiple shape prediction layers for face alignment. In particular, each shape prediction layer emphasizes on the detection of a certain cluster of semantically relevant landmarks respectively. Challenging landmarks are focused firstly, and each cluster of landmarks is further optimized respectively. Moreover, to reduce the model complexity, we propose a model assembling method to integrate multiple shape prediction layers into one shape prediction layer. Extensive experiments demonstrate that our method is effective for handling complex occlusions and appearance variations with real-time performance. The code for our method is available at https://github.com/ZhiwenShao/MCNet-Extension.

Adversarial Learning Semantic Volume for 2D/3D Face Shape Regression in the Wild.

Regression based methods have revolutionized 2D landmark localization with the exploitation of deep neural networks and massive annotated datasets in the wild. However, it remains challenging for 3D landmark localization due to the lack of annotated datasets and the ambiguous nature of landmarks under 3D perspective. This paper revisits regression based methods and proposes an adversarial voxel and coordinate regression framework for 2D and 3D facial landmark localization in real-world scenarios. First, a semantic volumetric representation is introduced to encode the per-voxel likelihood of positions being the 3D landmarks. Then, an end-to-end pipeline is designed to jointly regress the proposed volumetric representation and the coordinate vector. Such a pipeline not only enhances the robustness and accuracy of the predictions but also unifies the 2D and 3D landmark localization so that 2D and 3D datasets could be utilized simultaneously. Further, an adversarial learning strategy is exploited to distill 3D structure learned from synthetic datasets to real-world datasets under weakly supervised settings, where an auxiliary regression discriminator is proposed to encourage the network to produce plausible predictions for both synthetic and real-world images. The effectiveness of our method is validated on benchmark datasets 3DFAW and AFLW2000-3D for both 2D and 3D facial landmark localization tasks. Experimental results show that the proposed method achieves significant improvements over previous state-of-the-art methods.

A low-cost photorealistic CG dataset rendering pipeline for facial landmark localization

Face analysis has been a hot research field in computer vision for decades. The dataset is of vital importance for modern machine learning methods. The paper proposes a flexible CG (Computer Graphics) rendering pipe-line for creating facial image datasets together with automatic ground truth labelling. The proposed pipe-line could produce a huge amount of labelled data fast and in low cost compared to traditional dataset creation methods which need high cost hardware and longtime manual ground truth labelling. The paper also proposes a data capture setup in the CG environment for creating the dataset for facial landmark localization. The effectiveness of the proposed method is verified by cross validation with Multi-PIE dataset. For creating a high quality training dataset, some of the varying factors of the dataset should be considered. The paper analyzes a few varying factors for accurate eye landmark localization, such as eye closure levels, eye and eyebrow shapes and wearing glasses. Based on the benefits of the proposed CG rendering pipe-line, the paper implemented a facial landmark localization system across large face rotation by integrating off-the-shelves algorithms. The experiments on Multi-PIE and real persons show that the implemented system could localize facial landmarks accurately across [−90°, +90°] in yaw rotation in real time.