Multi-view Face Research Articles

Multi-View Face Synthesis via Progressive Face Flow.

Existing GAN-based multi-view face synthesis methods rely heavily on "creating" faces, and thus they struggle in reproducing the faithful facial texture and fail to preserve identity when undergoing a large angle rotation. In this paper, we combat this problem by dividing the challenging large-angle face synthesis into a series of easy small-angle rotations, and each of them is guided by a face flow to maintain faithful facial details. In particular, we propose a Face Flow-guided Generative Adversarial Network (FFlowGAN) that is specifically trained for small-angle synthesis. The proposed network consists of two modules, a face flow module that aims to compute a dense correspondence between the input and target faces. It provides strong guidance to the second module, face synthesis module, for emphasizing salient facial texture. We apply FFlowGAN multiple times to progressively synthesize different views, and therefore facial features can be propagated to the target view from the very beginning. All these multiple executions are cascaded and trained end-to-end with a unified back-propagation, and thus we ensure each intermediate step contributes to the final result. Extensive experiments demonstrate the proposed divide-and-conquer strategy is effective, and our method outperforms the state-of-the-art on four benchmark datasets qualitatively and quantitatively.

Boosting Face Recognition under Drastic Views Using a Pose AutoAugment Manner

Face recognition under drastic pose drops rapidly due to the limited samples during the model training. In this paper, we propose a pose-autoaugment face recognition framework (PAFR) based on the training of a Convolutional Neural Network (CNN) with multi-view face augmentation. The proposed framework consists of three parts: face augmentation, CNN training, and face matching. The face augmentation part is composed of pose autoaugment and background appending for increasing the pose variations of each subject. In the second part, we train a CNN model with the generated facial images to enhance the pose-invariant feature extraction. In the third part, we concatenate the feature vectors of each face and its horizontally flipped face from the trained CNN model to obtain a robust feature. The correlation score between the two faces is computed by the cosine similarity of their robust features. Comparable experiments are demonstrated on Bosphorus and CASIA-3D databases.

High-Fidelity Face Manipulation With Extreme Poses and Expressions

Face manipulation has shown remarkable advances with the flourish of Generative Adversarial Networks. However, due to the difficulties of controlling structures and textures, it is challenging to model poses and expressions simultaneously, especially for the extreme manipulation at high-resolution. In this article, we propose a novel framework that simplifies face manipulation into two correlated stages: a boundary prediction stage and a disentangled face synthesis stage. The first stage models poses and expressions jointly via boundary images. Specifically, a conditional encoder-decoder network is employed to predict the boundary image of the target face in a semi-supervised way. Pose and expression estimators are introduced to improve the prediction performance. In the second stage, the predicted boundary image and the input face image are encoded into the structure and the texture latent space by two encoder networks, respectively. A proxy network and a feature threshold loss are further imposed to disentangle the latent space. Furthermore, due to the lack of high-resolution face manipulation databases to verify the effectiveness of our method, we collect a new high-quality Multi-View Face (MVF-HQ) database. It contains 120,283 images at $6000\times 4000$ resolution from 479 identities with diverse poses, expressions, and illuminations. MVF-HQ is much larger in scale and much higher in resolution than publicly available high-resolution face manipulation databases. We will release MVF-HQ soon to push forward the advance of face manipulation. Qualitative and quantitative experiments on four databases show that our method dramatically improves the synthesis quality.

Multi-View Face Detection in Open Environments using Gabor Features and Neural Networks

Multi-view face detection in open environments is a challenging task, due to the wide variations in illumination, face appearances and occlusion. In this paper, a robust method for multi-view face detection in open environments, using a combination of Gabor features and neural networks, is presented. Firstly, the effect of changing the Gabor filter parameters (orientation, frequency, standard deviation, aspect ratio and phase offset) for an image is analysed, secondly, the range of Gabor filter parameter values is determined and finally, the best values for these parameters are specified. A multilayer feedforward neural network with a back-propagation algorithm is used as a classifier. The input vector is obtained by convolving the input image and a Gabor filter, with both the angle and frequency values equal to π/2. The proposed algorithm is tested on 1,484 image samples with simple and complex backgrounds. The experimental results show that the proposed detector achieves great detection accuracy, by comparing it with several popular face-detection algorithms, such as OpenCV’s Viola-Jones detector.

Prominent facial feature and hybrid learning method-based advanced face detector robust to head-up, head-down, and arbitrary rotation cases

Face detection is important for computer vision. Until now, the Viola and Jones algorithm is still the most popular face detector embedded in digital cameras. Although it has good performance in frontal face detection, it may not perform well in detecting rotated, head-up, and head-down faces. In this paper, we propose a more robust face detector based on prominent feature extraction. We find that, in addition to using eyes and mouth, which have been adopted by other algorithms, noses and ears are also important clues for face detection. Particularly, for a profile face, only a single eye can be detected. For head-up or head-down faces, eyes and mouth may not be well detected. However, in these cases, noses and ears can still be detected well and we can apply these features to improve the accuracy of rotated, head-up, and head-down face detection. To well detect these prominent features, in addition to the Viola–Jones detector, we also apply edge and color information and use the refocus algorithm based on the convolutional neural network. Simulations conducted on several popular multi-view face databases show that the proposed face detector can attain higher detection rates and is robust to rotated, head-up, and head-down cases.

3D face reconstruction from mugshots: Application to arbitrary view face recognition

Mugshots while routinely acquired by law enforcement agencies are under utilized by automated face recognition systems. In this paper, we propose a regression based approach to reconstruct textured full 3D face models from multi-view mugshot images. Using landmarks from the input frontal and profile mugshots of a subject, our method reconstructs his⧹her 3D face shape via either linear or nonlinear regressors. The texture of the mugshot images is mapped to the reconstructed 3D face shape via an efficient seamless texture recovery scheme. Compared with existing 3D face reconstruction methods, the proposed method more effectively utilizes the three-view mugshot face images collected during booking. The reconstructed 3D faces are used to generate realistic multi-view face images to enlarge the gallery and facilitate arbitrary view face recognition. Evaluation experiments have been done on BFM and Bosphorus databases in terms of reconstruction accuracy, and on Multi-PIE and Color FERET databases in terms of recognition accuracy. The results show that the proposed method can reduce the 3D face reconstruction error of the best competitive method from 2.31 mm to 1.88 mm, and improve the recognition accuracy of state-of-the-art deep learning based face matchers by as much as ~4% on Multi-PIE and ~2% on Color FERET despite the high baseline set by them.

Multi-view face recognition using deep neural networks

F Face recognition has been widely used in modern intelligent systems, such as smart video surveillance, online payment, and intelligent access control system. Existing face recognition algorithms are prone to be attacked by various face presentation attacks (face-PAs), such as printed paper, video replay, and silicone masks. To optimally handle the aforementioned problems, we formulate a novel deep architecture to increase the accuracy of multi-view human face recognition. In particular, in the first place, a novel deep neural network is built for deeply encoding the face regions, where a novel face alignment algorithm is employed to localize the key points inside faces. Subsequently, we utilize the well-known PCA for reducing the dimensionality of the deep features and simultaneously, removing the redundant and contaminated visual features. Thereafter, we propose a joint Bayesian framework in order to evaluate the similarity of feature vectors and highly competitive face classification accuracy can be achieved. Comprehensive experiments were conducted on our compiled CAS-PEAL dataset and achieved a 98.52% face recognition performance. Moreover, our proposed face recognition system can robustly handle various face recognition attack under various contexts.

GDFace: Gated Deformation for Multi-View Face Image Synthesis

Photorealistic multi-view face synthesis from a single image is an important but challenging problem. Existing methods mainly learn a texture mapping model from the source face to the target face. However, they fail to consider the internal deformation caused by the change of poses, leading to the unsatisfactory synthesized results for large pose variations. In this paper, we propose a Gated Deformable Face Synthesis Network to model the deformation of faces that aids the synthesis of the target face image. Specifically, we propose a dual network that consists of two modules. The first module estimates the deformation of two views in the form of convolution offsets according to the input and target poses. The second one, on the other hand, leverages the predicted deformation offsets to create the target face image. In this way, pose changes are explicitly modeled in the face generator to cope with geometric transformation, by adaptively focusing on pertinent regions of the source image. To compensate offset estimation errors, we introduce a soft-gating mechanism that enables adaptive fusion between deformable features and primitive features. Extensive experimental results on five widely-used benchmarks show that our approach performs favorably against the state-of-the-arts on multi-view face synthesis, especially for large pose changes.

Recovering facial reflectance and geometry from multi-view images

While the problem of estimating shapes and diffuse reflectances of human faces from images has been extensively studied, there is relatively less work done on recovering the specular albedo. This paper presents a lightweight solution for inferring photorealistic facial reflectance and geometry. Our system processes video streams from two views of a subject, and outputs two reflectance maps for diffuse and specular albedos, as well as a vector map of surface normals. A model-based optimization approach is used, consisting of the three stages of multi-view face model fitting, facial reflectance inference and facial geometry refinement. Our approach is based on a novel formulation built upon the 3D morphable model (3DMM) for representing 3D textured faces in conjunction with the Blinn-Phong reflection model. It has the advantage of requiring only a simple setup with two video streams, and is able to exploit the interaction between the diffuse and specular reflections across multiple views as well as time frames. As a result, the method is able to reliably recover high-fidelity facial reflectance and geometry, which facilitates various applications such as generating photorealistic facial images under new viewpoints or illumination conditions.

Multi-View Face Recognition Via Well-Advised Pose Normalization Network

Numerous face frontalization methods based on 3D Morphable Model (3DMM) and Generative Adversarial Networks (GAN) have made great progress in multi-view face recognition. However, facial feature analysis and identity discrimination often suffer from failure frontalization results because of monotonous single-domain training and unpredictable input profile faces. To overcome the drawback, we present a novel approach named Well-advised Pose Normalization Network (WAPNN), which leverages multiple domains and extracts features considering their frontalization qualities wisely, to achieve a high accuracy on multi-view face recognition. Through multi-domain datasets, we design an end-to-end facial pose normalization network with adaptive weights on different objectives to exploit potentialities of various profile-front relationships. Meanwhile, the proposed method encourages intra-class compactness and inter-class separability between facial features by introducing quality-aware feature fusion. Experimental analyses show that our method effectively recovers frontal faces with good-quality textures and high identity-preserving, and significantly reduces the impact of various poses on face recognition under both constrained and wild environments.

Cost-Sensitive Canonical Correlation Analysis for Semi-Supervised Multi-View Learning

Canonical correlation analysis (CCA) is a cost insensitive method. It assumes the same loss for different classification errors and aims to attain a low error rate by maximizing the cross-view correlation. However, in some real-world applications, different classification errors will lead to unequal misclassification losses. In addition, in practice, only limited cost label information is available in training set due to the expensive costs of labelling. This paper aims to perform label propagation with CCA in a unified cost-sensitive learning framework. By learning jointly, both the label propagation and CCA can feed back to each other. Thus, more discriminative and cost-sensitive projections will be learned for feature fusion. We test the proposed method on the cost-sensitive application of door-locker system based on multi-view face recognition. The results in comparison with eight label propagation methods, eleven CCA related methods and eight cost-sensitive single-view methods demonstrate its effectiveness.

Multi-view local linear KNN classification: theoretical and experimental studies on image classification

When handling special multi-view scenarios where data from each view keep the same features, we may perhaps encounter two serious challenges: (1) samples from different views of the same class are less similar than those from the same view but different class, which sometimes happen in local way in both training and/or testing phases; (2) training an explicit prediction model becomes unreliable and even infeasible for test samples in multi-view scenarios. In this study, we prefer the philosophy of the k nearest neighbor method (KNN) to circumvent the second challenge. Without an explicit prediction model trained directly from the above multi-view data, a new multi-view local linear k nearest neighbor method (MV-LLKNN) is then developed to circumvent the two challenges so as to predict the label of each test sample. MV-LLKNN has its two reliable assumptions. One is the theoretically and experimentally provable assumption that any test sample can be well approximated by a linear combination of its neighbors in the multi-view training dataset. The other assumes that these neighbors should demonstrate their clustering property according to certain commonality-based similarity measure between the multi-view test sample and these multi-view neighbors so as to avoid the first challenge. MV-LLKNN can realize its effective prediction for a test multi-view sample by cheaply using both on-hand fast iterative shrinkage thresholding algorithm (FISTA) and KNN. Our theoretical analysis and experimental results about real multi-view face datasets indicate the effectiveness of MV-LLKNN.

3D Aided Duet GANs for Multi-View Face Image Synthesis

Multi-view face synthesis from a single image is an ill-posed computer vision problem. It often suffers from appearance distortions if it is not well-defined. Producing photo-realistic and identity preserving multi-view results is still a not well-defined synthesis problem. This paper proposes 3D aided duet generative adversarial networks (AD-GAN) to precisely rotate the yaw angle of an input face image to any specified angle. AD-GAN decomposes the challenging synthesis problem into two well-constrained subtasks that correspond to a face normalizer and a face editor. The normalizer first frontalizes an input image, and then the editor rotates the frontalized image to a desired pose guided by a remote code. In the meantime, the face normalizer is designed to estimate a novel dense UV correspondence field, making our model aware of 3D face geometry information. In order to generate photo-realistic local details and accelerate convergence process, the normalizer and the editor are trained in a two-stage manner and regulated by a conditional self-cycle loss and a perceptual loss. Exhaustive experiments on both controlled and uncontrolled environments demonstrate that the proposed method not only improves the visual realism of multi-view synthetic images but also preserves identity information well.

Robust Multiview Subspace Learning With Nonindependently and Nonidentically Distributed Complex Noise.

Multiview Subspace Learning (MSL), which aims at obtaining a low-dimensional latent subspace from multiview data, has been widely used in practical applications. Most recent MSL approaches, however, only assume a simple independent identically distributed (i.i.d.) Gaussian or Laplacian noise for all views of data, which largely underestimates the noise complexity in practical multiview data. Actually, in real cases, noises among different views generally have three specific characteristics. First, in each view, the data noise always has a complex configuration beyond a simple Gaussian or Laplacian distribution. Second, the noise distributions of different views of data are generally nonidentical and with evident distinctiveness. Third, noises among all views are nonindependent but obviously correlated. Based on such understandings, we elaborately construct a new MSL model by more faithfully and comprehensively considering all these noise characteristics. First, the noise in each view is modeled as a Dirichlet process (DP) Gaussian mixture model (DPGMM), which can fit a wider range of complex noise types than conventional Gaussian or Laplacian. Second, the DPGMM parameters in each view are different from one another, which encodes the "nonidentical" noise property. Third, the DPGMMs on all views share the same high-level priors by using the technique of hierarchical DP, which encodes the "nonindependent" noise property. All the aforementioned ideas are incorporated into an integrated graphics model which can be appropriately solved by the variational Bayes algorithm. The superiority of the proposed method is verified by experiments on 3-D reconstruction simulations, multiview face modeling, and background subtraction, as compared with the current state-of-the-art MSL methods.

Multi-view face hallucination using SVD and a mapping model

Multi-view face hallucination (MFH) presents a challenge issue in face recognition domain. In this paper, an efficient method based on singular value decomposition (SVD) and a mapping model is proposed for multi-view face hallucination. Based on an approximately same linear mapping relationship across different views, two corresponding matrices obtained from the SVD of the low resolution (LR) image for the high-resolution (HR) multi-view face images can be constructed via the mapping model using global reconstruction. Experiments show that our proposed multi-view face-hallucination scheme is effective and produces promising super-resolved results.

Joint Multi-view Face Alignment in the Wild.

The de facto algorithm for facial landmark estimation involves running a face detector with a subsequent deformable model fitting on the bounding box. This encompasses two basic problems: i) the detection and deformable fitting steps are performed independently, while the detector might not provide best-suited initialization for the fitting step, ii) the face appearance varies hugely across different poses, which makes the deformable face fitting very challenging and thus distinct models have to be used (e.g., one for profile and one for frontal faces). In this work, we propose the first, to the best of our knowledge, joint multi-view convolutional network to handle large pose variations across faces in-the-wild, and elegantly bridge face detection and facial landmark localization tasks. Existing joint face detection and landmark localization methods focus only on a very small set of landmarks. By contrast, our method can detect and align a large number of landmarks for semi-frontal (68 landmarks) and profile (39 landmarks) faces. We evaluate our model on a plethora of datasets including standard static image datasets such as IBUG, 300W, COFW, and the latest Menpo Benchmark for both semi-frontal and profile faces. Significant improvement over state-of-the-art methods on deformable face tracking is witnessed on 300VW benchmark. We also demonstrate state-ofthe- art results for face detection on FDDB and MALF datasets.

Deep learning for pose-invariant face detection in unconstrained environment

<span lang="EN-US">In the recent past, convolutional neural networks (CNNs) have seen resurgence and have performed extremely well on vision tasks. Visually the model resembles a series of layers each of which is processed by a function to form a next layer. It is argued that CNN first models the low level features such as edges and joints and then expresses higher level features as a composition of these low level features. The aim of this paper is to detect multi-view faces using deep convolutional neural network (DCNN). Implementation, detection and retrieval of faces will be obtained with the help of direct visual matching technology. Further, the probabilistic measure of the similarity of the face images will be done using Bayesian analysis. Experiment detects faces with ±90 degree out of plane rotations. Fine tuned AlexNet is used to detect pose invariant faces. For this work, we extracted examples of training from AFLW (Annotated Facial Landmarks in the Wild) dataset that involve 21K images with 24K annotations of the face.</span>

Robust in-plane and out-of-plane face detection algorithm using frontal face detector and symmetry extension

Face detection plays an important role in many computer vision applications. In recent years, much research has focused on extending the well-established Adaboost face detector algorithm for multi-view face detection. However, detecting in-plane and out-of-plane rotated faces simultaneously is still a challenging task today. In this paper, a very robust multi-view face detection algorithm, with its core functionality based on frontal face detection, is proposed to simultaneously detect in-plane and out-of-plane rotated faces. Moreover, only the training data in the frontal face is needed and we do not require the training data from different in-plane or out-of-plane rotation angles. In the proposed algorithm, first, techniques such as the skin filter and entropy rate superpixels (ERSs) are applied to obtain face candidates. Then, angle compensation and refinement are applied to improve the accuracy of face detection in the in-plane case. Moreover, the symmetry extension technique, i.e., extending the face candidate with its flipped version to create a face similar to the frontal one, is applied to detect out-of-plane faces without the need of training data. Simulations on the FEI, Pointing'04, BaoFace, Group, Utrecht, and WWW datasets demonstrate the proposed algorithm's effectiveness and superior performance as compared to state-of-the-art face detection methods.

Deep-learning-based face detection using iterative bounding-box regression

Multi-view face detection in open environments is a challenging task due to the diverse variations of face appearances and occlusion. In the task of face detection, localization accuracy is one of the key factors. However, many of the existing methods do not pay enough attention to localization. Some of the current methods have applied localization techniques, but they have not fully realized its potential and realized more accurate localization. In this paper, we propose a deep cascaded detection method that iteratively exploits bounding-box regression, a localization technique, to approach the detection of potential faces in images. In addition, we consider the inherent correlation of classification and bounding-box regression and exploit it to further increase overall performance. In particular, our method leverages a cascaded architecture with three stages of carefully designed deep convolutional networks to predict the existence of faces. Extensive experiments demonstrate the efficiency of our algorithm by comparing it with several popular face-detection algorithms on the widely used AFW and FDDB datasets.

BAdaCost: Multi-class Boosting with Costs

We present BAdaCost, a multi-class cost-sensitive classification algorithm. It combines a set of cost-sensitive multi-class weak learners to obtain a strong classification rule within the Boosting framework. To derive the algorithm we introduce CMEL, a Cost-sensitive Multi-class Exponential Loss that generalizes the losses optimized in various classification algorithms such as AdaBoost, SAMME, Cost-sensitive AdaBoost and PIBoost. Hence unifying them under a common theoretical framework. In the experiments performed we prove that BAdaCost achieves significant gains in performance when compared to previous multi-class cost-sensitive approaches. The advantages of the proposed algorithm in asymmetric multi-class classification are also evaluated in practical multi-view face and car detection problems.