Face Alignment Research Articles

An improved boundary-aware face alignment using stacked dense U-Nets

Facial landmark localization is still a challenge task in the unconstrained environment with influences of significant variation conditions such as facial pose, shape, expression, illumination, and occlusions. In this work, we present an improved boundary-aware face alignment method by using stacked dense U-Nets. The proposed method consists of two stages: a boundary heatmap estimation stage to learn the facial boundary lines and a facial landmark localization stage to predict the final face alignment result. With the constraint of boundary lines, facial landmarks are unified as a whole facial shape. Hence, the unseen landmarks in a shape with occlusions can be better estimated by message passing with other landmarks. By introducing the stacked dense U-Nets for feature extraction, the capacity of the model is improved. Experiments and comparisons on public datasets show that the proposed method obtains better performance than the baselines, especially for facial images with large pose variation, shape variation, and occlusions.

Challenge and Algorithm of Face Alignment Development

Face alignment is a key step in face recognition. The location of face feature points is located in the face image, and the difference between different faces is reduced by geometric transformation. This is the basic condition of face information processing, such as expression recognition, face tracking, head pose estimation and so on. Due to the interference of expression, illumination, shading and other factors, face alignment has a great challenge and is becoming the developmental direction. Different algorithms can solve different problems at different levels. Deep learning algorithm can solve the shortcomings of traditional algorithm, improve the accuracy of face alignment, and promote the development of face alignment.

Gradient Shape Model

For years, the so-called Constrained Local Model (CLM) and its variants have been the gold standard in face alignment tasks. The CLM combines an ensemble of local feature detectors whose locations are regularized by a shape model. Fitting such a model typically consists of an exhaustive local search using the detectors and a global optimization that finds the CLM’s parameters that jointly maximize all the responses. However, one major drawback of CLMs is the inefficiency of the local search, which relies on a large amount of expensive convolutions. This paper introduces the Gradient Shape Model (GSM), a novel approach that addresses this limitation. We are able to align a similar CLM model without the need for any convolutions at all. We also use true analytical gradient and Hessian matrices, which are easy to compute, instead of their approximations. Our formulation is very general, allowing an optional 3D shape term to be seamlessly included. Additionally, we expand the GSM formulation through a cascade regression framework. This revised technique allows a substantially reduction in the complexity/dimensionality of the data term, making it possible to compute a denser, more accurate, regression step per cascade level. Experiments in several standard datasets show that our proposed models perform faster than state-of-the-art CLMs and better than recent cascade regression approaches.

Learning spatial-temporal deformable networks for unconstrained face alignment and tracking in videos

In this paper, we propose a spatial-temporal deformable networks approach to investigate both problems of face alignment in static images and face tracking in videos under unconstrained environments. Unlike conventional feature extractions which cannot explicitly exploit augmented spatial geometry for various facial shapes, in our approach, we propose a deformable hourglass networks (DHGN) method, which aims to learn a deformable mask to reduce the variances of facial deformation and extract attentional facial regions for robust feature representation. However, our DHGN is limited to extract only spatial appearance features from static facial images, which cannot explicitly exploit the temporal consistency information across consecutive frames in videos. For efficient temporal modeling, we further extend our DHGN to a temporal DHGN (T-DHGN) paradigm particularly for video-based face alignment. To this end, our T-DHGN principally incorporates with a temporal relational reasoning module, so that the temporal order relationship among frames is encoded in the relational feature. By doing this, our T-DHGN reasons about the temporal offsets to select a subset of discriminative frames over time steps, thus allowing temporal consistency information memorized to flow across frames for stable landmark tracking in videos. Compared with most state-of-the-art methods, our approach achieves superior performance on folds of widely-evaluated benchmarking datasets. Code will be made publicly available upon publication.

Towards End-to-End Face Recognition Method through Spatial Transformer

Face recognition technology is widely applied in daily life, but in most methods, similarity or affine transformation is employed to align face images according to five facial landmarks. The face alignment module is implemented independently, thus it’s difficult in end-to-end training. In this paper, the main purpose is to design a towards end-to-end trainable face recognition method based on indoor scenes. Due to that spatial transformer can implement any parametrizable transformation, we joint it with recognition network, making end-to-end training possible. Simultaneously, any prior knowledge on facial landmarks isn’t required. The model jointly with spatial transformer can achieve 0.3% higher accuracy than similarity transformation. Most downsampling methods ignore the sampling theorem, making convolutional networks not shift-invariant. We replace max-pooling by MaxBlurPool in spatial transformer network, and the accuracy is improved by 0.25%.

An incrementally cascaded broad learning framework to facial landmark tracking

Facial landmark tracking often adopts static models to generically fit per frame of video. This is considered inappropriate since such models ignore the informative correlation between previous and current frames. Moreover, most of these methods fail to balance the speed and accuracy for video-based facial landmark tracking simultaneously. In this paper, we propose an efficient online framework for video-based face alignment, named Incrementally Cascaded Broad Learning framework (ICBL). ICBL aims to continuously enhance the prediction capability of tracking model for sequential data. It is capable of learning the spatial appearance on specific-person statistics from continuous facial frames and using such knowledge to incrementally tune a cascade of regressors in parallel. To achieve this goal, we approximate the facial shape space by sampling from a dynamic distribution which is continuously updated by person-specific statistics from the tracked facial frames. This dramatically facilitates cascade regression to incrementally update all cascade-regressors in parallel, thus allowing a fast update of the whole model. Furthermore, we successfully incorporate both the linear and non-linear mappings into our parallel cascade framework and introduce Broad Learning (BL) algorithm as a solution for them simultaneously. Experimental results on the most popular and large-scale benchmark for facial landmark tracking show highly competitive performance of proposed ICBL in comparisons with the state-of-the-arts. The code of our ICBL framework has been available fromhttps://github.com/CaifengLiu/Facial-landmark-tracking-by-ICBL.

A 460 GOPS/W Improved Mnemonic Descent Method-Based Hardwired Accelerator for Face Alignment

The mnemonic descent method (MDM) algorithm is the first end-to-end recurrent convolutional system for high-accuracy face alignment. However, the heavy computational complexity and high memory access demands make it difficult to satisfy the requirements of real-time applications. To address this problem, an improved MDM (I-MDM) algorithm is proposed for efficient hardware implementation based on several hardware-oriented optimizations. First, a patch merging mechanism is introduced to dynamically cluster and eliminate redundant landmarks, which significantly reduces computational complexity with minimal accuracy loss. Second, a dedicated convolutional layer is inserted to halve the number of computations and memory access of the subsequent fully connected layer, yielding a 4.42% decrease in the failure rate. Third, a lightweight preprocessing method named dual regressors is proposed to reinitialize face images, which can greatly improve the overall accuracy. Moreover, compared with a similar method, the DR method can reduce computations and memory storage by nearly 99.9%. Overall and compared with the MDM algorithm, I-MDM not only reduces the number of computations by 23.5% but also decreases the failure rate by 17.9% on the 300 W test set. Based on the proposed I-MDM algorithm, an I-MDM-based hardwired accelerator is presented using the TSMC 65 nm CMOS process. First, compared with similar solutions, the gradient calculation operation is rearranged and loaded pixels are reused in the HoG feature extraction to eliminate all division operations and 25% off-chip memory access. Second, patch-independent central activations are used to enable patch-level pipelined operations, yielding a 2× acceleration in the overall process. This accelerator achieves 460 GOPS/W energy efficiency at 330 MHz, which is 38× higher than the most recent face alignment accelerator with the same process.

Joint learning for face alignment and face transfer with depth image

Face alignment and cross-modal face transfer are two important tasks for automatic face analysis in computer vision. Over the years, they have been extensively studied. Recently, deep neural networks have attracted much research attention for both face alignment and face transfer. With the prevalence of the consumer depth sensor, depth-based face alignment and cross-modal (image and depth) are increasingly important. Different from existing RGB- image based tasks, the main challenge of depth-based tasks is the lack of annotated data. To address the challenge, we observe that these two tasks are closely related and their learning processes may benefit each other. This paper develops a joint multi-task learning algorithm for both depth-based face alignment and face transfer using the deep convolutional neural network (CNN). The proposed approach allows the CNN model to simultaneously share visual knowledge and information between two tasks. We use a dataset of 10,000 face depth images for validation. Our experiments show that the proposed approach outperforms state-of-the-art algorithms. The results also show that learning these two related tasks simultaneously improves the performance of each individual task.

Multi-view face recognition using deep neural networks

F Face recognition has been widely used in modern intelligent systems, such as smart video surveillance, online payment, and intelligent access control system. Existing face recognition algorithms are prone to be attacked by various face presentation attacks (face-PAs), such as printed paper, video replay, and silicone masks. To optimally handle the aforementioned problems, we formulate a novel deep architecture to increase the accuracy of multi-view human face recognition. In particular, in the first place, a novel deep neural network is built for deeply encoding the face regions, where a novel face alignment algorithm is employed to localize the key points inside faces. Subsequently, we utilize the well-known PCA for reducing the dimensionality of the deep features and simultaneously, removing the redundant and contaminated visual features. Thereafter, we propose a joint Bayesian framework in order to evaluate the similarity of feature vectors and highly competitive face classification accuracy can be achieved. Comprehensive experiments were conducted on our compiled CAS-PEAL dataset and achieved a 98.52% face recognition performance. Moreover, our proposed face recognition system can robustly handle various face recognition attack under various contexts.

3D Face Reconstruction From A Single Image Assisted by 2D Face Images in the Wild

3D face reconstruction from a single 2D image is a challenging problem with broad applications. Recent methods typically aim to learn a CNN-based 3D face model that regresses coefficients of 3D Morphable Model (3DMM) from 2D images to render 3D face reconstruction or dense face alignment. However, the shortage of training data with 3D annotations considerably limits performance of those methods. To alleviate this issue, we propose a novel 2D-assisted self-supervised learning (2DASL) method that can effectively use "in-the-wild" 2D face images with noisy landmark information to substantially improve 3D face model learning. Specifically, taking the sparse 2D facial landmarks as additional information, 2DSAL introduces four novel self-supervision schemes that view the 2D landmark and 3D landmark prediction as a self-mapping process, including the 2D and 3D landmark self-prediction consistency, cycle-consistency over the 2D landmark prediction and self-critic over the predicted 3DMM coefficients based on landmark predictions. Using these four self-supervision schemes, the 2DASL method significantly relieves demands on the the conventional paired 2D-to-3D annotations and gives much higher-quality 3D face models without requiring any additional 3D annotations. Experiments on multiple challenging datasets show that our method outperforms state-of-the-arts for both 3D face reconstruction and dense face alignment by a large margin.

Face Recognition System for Complex Surveillance Scenarios

In recent years, with the continuous development of the Internet and artificial intelligence, face recognition technology has also been widely used in many application scenarios. Facing complex surveillance scenarios, face recognition technology still faces great challenges. This paper focuses on implementing real-time and efficient face recognition systems in complex surveillance scenarios, such as insufficient lighting, small faces, dense crowds, and sides at 45 ° environment. The system is mainly based on RetinaFace for face detection and face alignment, and uses lightweight mobilenet (0.25) as the backbone network of RetinaFace. Facial feature extraction is based on deep residual neural network combined with ArcFace loss, and feature matching is performed by Euclidean distance. The experimental results show that the face recognition system has good real-time performance, accuracy and robustness.

Normalization of Facial Pose and Expression to Increase the Accuracy of Face Recognition System

Normalization of face poses and expressions is one of the critical steps to restore the raw appearance of faces in uncontrolled conditions (in the wild) and help improving the performance of face recognition systems. Researchers in the field of computer vision has for a long time trying to find methods to normalize of poses and expressions to gain accuracy improvement for the face recognition process. The process of normalization is related to the face alignment stage, which aims to find face landmarks (point markers of facial features). From this face landmarks, the poses and facial expressions that are different from canonical face obtained for later transformation. This paper discusses the method of normalizing poses and facial expressions that have been proposed by researchers from geometrical-based transformation to the current state of the art, which is a deep learning-based system.

Personal Guides: Heterogeneous Robots Sharing Personal Tours in Multi-Floor Environments.

GidaBot is an application designed to setup and run a heterogeneous team of robots to act as tour guides in multi-floor buildings. Although the tours can go through several floors, the robots can only service a single floor, and thus, a guiding task may require collaboration among several robots. The designed system makes use of a robust inter-robot communication strategy to share goals and paths during the guiding tasks. Such tours work as personal services carried out by one or more robots. In this paper, a face re-identification/verification module based on state-of-the-art techniques is developed, evaluated offline, and integrated into GidaBot’s real daily activities, to avoid new visitors interfering with those attended. It is a complex problem because, as users are casual visitors, no long-term information is stored, and consequently, faces are unknown in the training step. Initially, re-identification and verification are evaluated offline considering different face detectors and computing distances in a face embedding representation. To fulfil the goal online, several face detectors are fused in parallel to avoid face alignment bias produced by face detectors under certain circumstances, and the decision is made based on a minimum distance criterion. This fused approach outperforms any individual method and highly improves the real system’s reliability, as the tests carried out using real robots at the Faculty of Informatics in San Sebastian show.

A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation

Action Units (AUs) are geometrically-based atomic facial muscle movements known to produce appearance changes at specific facial locations. Motivated by this observation we propose a novel AU modelling problem that consists of jointly estimating their localisation and intensity. To this end, we propose a simple yet efficient approach based on Heatmap Regression that merges both problems into a single task. A Heatmap models whether an AU occurs or not at a given spatial location. To accommodate the joint modelling of AUs intensity, we propose variable size heatmaps, with their amplitude and size varying according to the labelled intensity. Using Heatmap Regression, we can inherit from the progress recently witnessed in facial landmark localisation. Building upon the similarities between both problems, we devise a transfer learning approach where we exploit the knowledge of a network trained on large-scale facial landmark datasets. In particular, we explore different alternatives for transfer learning through a) fine-tuning, b) adaptation layers, c) attention maps, and d) reparametrisation. Our approach effectively inherits the rich facial features produced by a strong face alignment network, with minimal extra computational cost. We empirically validate that our system sets a new state-of-the-art on three popular datasets, namely BP4D, DISFA, and FERA2017.

KPNet: Towards Minimal Face Detector

The small receptive field and capacity of minimal neural networks limit their performance when using them to be the backbone of detectors. In this work, we find that the appearance feature of a generic face is discriminative enough for a tiny and shallow neural network to verify from the background. And the essential barriers behind us are 1) the vague definition of the face bounding box and 2) tricky design of anchor-boxes or receptive field. Unlike most top-down methods for joint face detection and alignment, the proposed KPNet detects small facial keypoints instead of the whole face by in the bottom-up manner. It first predicts the facial landmarks from a low-resolution image via the well-designed fine-grained scale approximation and scale adaptive soft-argmax operator. Finally, the precise face bounding boxes, no matter how we define it, can be inferred from the keypoints. Without any complex head architecture or meticulous network designing, the KPNet achieves state-of-the-art accuracy on generic face detection and alignment benchmarks with only ∼ 1M parameters, which runs at 1000fps on GPU and is easy to perform real-time on most modern front-end chips.

Joint Super-Resolution and Alignment of Tiny Faces

Super-resolution (SR) and landmark localization of tiny faces are highly correlated tasks. On the one hand, landmark localization could obtain higher accuracy with faces of high-resolution (HR). On the other hand, face SR would benefit from prior knowledge of facial attributes such as landmarks. Thus, we propose a joint alignment and SR network to simultaneously detect facial landmarks and super-resolve tiny faces. More specifically, a shared deep encoder is applied to extract features for both tasks by leveraging complementary information. To exploit representative power of the hierarchical encoder, intermediate layers of a shared feature extraction module are fused to form efficient feature representations. The fused features are then fed to task-specific modules to detect landmarks and super-resolve face images in parallel. Extensive experiments demonstrate that the proposed model significantly outperforms the state-of-the-art in both landmark localization and SR of faces. We show a large improvement for landmark localization of tiny faces (i.e., 16 × 16). Furthermore, the proposed framework yields comparable results for landmark localization on low-resolution (LR) faces (i.e., 64 × 64) to existing methods on HR (i.e., 256 × 256). As for SR, the proposed method recovers sharper edges and more details from LR face images than other state-of-the-art methods, which we demonstrate qualitatively and quantitatively.

Towards Omni-Supervised Face Alignment for Large Scale Unlabeled Videos

In this paper, we propose a spatial-temporal relational reasoning networks (STRRN) approach to investigate the problem of omni-supervised face alignment in videos. Unlike existing fully supervised methods which rely on numerous annotations by hand, our learner exploits large scale unlabeled videos plus available labeled data to generate auxiliary plausible training annotations. Motivated by the fact that neighbouring facial landmarks are usually correlated and coherent across consecutive frames, our approach automatically reasons about discriminative spatial-temporal relationships among landmarks for stable face tracking. Specifically, we carefully develop an interpretable and efficient network module, which disentangles facial geometry relationship for every static frame and simultaneously enforces the bi-directional cycle-consistency across adjacent frames, thus allowing the modeling of intrinsic spatial-temporal relations from raw face sequences. Extensive experimental results demonstrate that our approach surpasses the performance of most fully supervised state-of-the-arts.

End-to-End Spatial Transform Face Detection and Recognition

BackgroundSeveral face detection and recognition methods have been proposed in the past decades that have excellent performance. The conventional face recognition pipeline comprises the following: (1) face detection, (2) face alignment, (3) feature extraction, and (4) similarity, which are independent of each other. The separate facial analysis stages lead to redundant model calculations, and are difficult for use in end-to-end training. MethodsIn this paper, we propose a novel end-to-end trainable convolutional network framework for face detection and recognition, in which a geometric transformation matrix is directly learned to align the faces rather than predicting the facial landmarks. In the training stage, our single CNN model is supervised only by face bounding boxes and personal identities, which are publicly available from WIDER FACE and CASIA-WebFace datasets. Our model is tested on Face Detection Dataset and Benchmark (FDDB) and Labeled Face in the Wild (LFW) datasets. ResultsThe results show 89.24% recall for face detection tasks and 98.63% accuracy for face recognition tasks.

Facial Landmark Detection Experiment

Facial landmark detection or face alignment is an important problem in computer vision. Different types of methods have been proposed to target this problem, with their advantages and disadvan- tages. This paper classified the past methods into three categories, briefly compared them, then attempted to reimplementing part of a recent work. Experiments were done with a common, current dataset to compare the method with other methods.

High-Precision Portrait Classification Based on MTCNN and Its Application on Similarity Judgement

Portrait classification is a complex course including at least face detection, recognization and compare each of which contains multi-tasks, facing plenty of various challenging questions due to askew poses, illuminations, occlusions, image blurring and small scale face in the pictures. Though deep learning methods, such as Convolutional Neural Network (CNN) family and You Only Look Once (YOLO) series, had boomed a large number of areas on object detection and accelerated the solving of these difficulties on image processing, they are not specially designed for the image classification and may require a great deal of resource, expensive computation and taxing annotation. In 2016, an innovative face detection model named Multi-task convolutional neural network (MTCNN) arose and triggered viral and wide spread. Its high efficient and accurate performance on both face detection and face alignment tasks, real time effect based on lightweight CNN as well as effective conducting online hard sample mining, all contribute to significant improvement to the challenges above. This paper introduces the MTCNN algorithm and applies it to the similarity judgement with two industrial real problems together with FaceNet model. In addition, some effective practical methods on increase precision of classification are also proposed to gain better effect.