Large Pose Variations Research Articles

Fully Deformable Network for Multiview Face Image Synthesis.

Photorealistic multiview face synthesis from a single image is a challenging problem. Existing works mainly learn a texture mapping model from the source to the target faces. However, they rarely consider the geometric constraints on the internal deformation arising from pose variations, which causes a high level of uncertainty in face pose modeling, and hence, produces inferior results for large pose variations. Moreover, current methods typically suffer from undesired facial details loss due to the adoption of the de-facto standard encoder-decoder architecture without any skip connections (SCs). In this article, we directly learn and exploit geometric constraints and propose a fully deformable network to simultaneously model the deformations of both landmarks and faces for face synthesis. Specifically, our model consists of two parts: a deformable landmark learning network (DLLN) and a gated deformable face synthesis network (GDFSN). The DLLN converts an initial reference landmark to an individual-specific target landmark as delicate pose guidance for face rotation. The GDFSN adopts a dual-stream structure, with one stream estimating the deformation of two views in the form of convolution offsets according to the source pose and the converted target pose, and the other leveraging the predicted deformation offsets to create the target face. In this way, individual-aware pose changes are explicitly modeled in the face generator to cope with geometric transformation, by adaptively focusing on pertinent regions of the source face. To compensate for offset estimation errors, we introduce a soft-gating mechanism for adaptive fusion between deformable features and primitive features. Additionally, a pose-aligned SC (PASC) is tailored to propagate low-level input features to the appropriate positions in the output features for further enhancing the facial details and identity preservation. Extensive experiments on six benchmarks show that our approach performs favorably against the state-of-the-arts, especially with large pose changes. Code is available at https://github.com/cschengxu/FDFace.

FP-Age: Leveraging Face Parsing Attention for Facial Age Estimation in the Wild.

Image-based age estimation aims to predict a person's age from facial images. It is used in a variety of real-world applications. Although end-to-end deep models have achieved impressive results for age estimation on benchmark datasets, their performance in-the-wild still leaves much room for improvement due to the challenges caused by large variations in head pose, facial expressions, and occlusions. To address this issue, we propose a simple yet effective method to explicitly incorporate facial semantics into age estimation, so that the model would learn to correctly focus on the most informative facial components from unaligned facial images regardless of head pose and non-rigid deformation. To this end, we design a face parsing-based network to learn semantic information at different scales and a novel face parsing attention module to leverage these semantic features for age estimation. To evaluate our method on in-the-wild data, we also introduce a new challenging large-scale benchmark called IMDB-Clean. This dataset is created by semi-automatically cleaning the noisy IMDB-WIKI dataset using a constrained clustering method. Through comprehensive experiment on IMDB-Clean and other benchmark datasets, under both intra-dataset and cross-dataset evaluation protocols, we show that our method consistently outperforms all existing age estimation methods and achieves a new state-of-the-art performance. To the best of our knowledge, our work presents the first attempt of leveraging face parsing attention to achieve semantic-aware age estimation, which may be inspiring to other high level facial analysis tasks.

Integrate memory mechanism in multi-granularity deep framework for driver drowsiness detection

Driver drowsiness detection is a critical task for early warning of safe driving, while existing spatial feature-based methods face the challenges of large variations of head pose. This paper proposes a novel approach to integrate the memory mechanism in a multi-granularity deep framework to detect driver drowsiness, and the temporal dependencies over sequential frames are well integrated with the spatial deep learning framework on the frontal faces. The proposed approach includes two steps. First, the spatial Multi-granularity Convolutional Neural Network is designed to utilize a group of parallel Convolutional Neural Network extractors on well-aligned facial patches of different granularities and extract facial representations effectively for large variations of head pose. Furthermore, it can flexibly fuse detailed appearance clues of the main parts and local-to-global spatial constraints. Second, the memory mechanism is set up using a deep long short-term memory network of facial representations to explore long-term relationships with variable length over sequential frames, which is capable of distinguishing the states with temporal dependencies, such as blinking and closing eyes. The proposed approach achieves 90.05% accuracy and about 37 frames per second (FPS) speed on the evaluation set of the National Tsing Hua University Driver Drowsiness Detection dataset, which is applied to the intelligent vehicle for driver drowsiness detection. A dataset named Forward Instant Driver Drowsiness Detection is also built and will be publicly accessible to speed up the study of driver drowsiness detection.

Person Re-Identification With Visual Semantic Representation Mining and Reasoning

Person re-identification aims to match images of the same person in different scenarios. The challenge of this task is how to extract discriminative features from person images with complex background noise, severe occlusions and large pose variations. Recently, some studies have attempted to apply human semantic parsing or attention mechanisms to help capture human parts or important object regions. Despite the performance improvements, these methods either require the introduction of additional prior knowledge or ignore the connections between the human body parts. To solve the above problem, we propose a novel Visual Semantic Representation Mining and Reasoning (SRMR) block to capture more discriminative semantic features of person images. Specifically, to get rid of the restriction of the external model, we propose to directly mine the clustering relationship between each local feature in the global structure and obtain the discriminative region in the person image by voting. Then, to establish the relationship between person body parts, we utilize Graph Convolutional Networks (GCN) to effectively construct the correlation between body parts. Extensive ablation studies demonstrate that our SRMR block can significantly improve the feature representation power and achieve state-of-the-art performance on several popular benchmarks.

POC-net: pelican optimization-based convolutional neural network for recognizing large pose variation from video

POC-net: pelican optimization-based convolutional neural network for recognizing large pose variation from video

Mask-FPAN: Semi-supervised face parsing in the wild with de-occlusion and UV GAN

The field of fine-grained semantic segmentation for a person’s face and head, which includes identifying facial parts and head components, has made significant progress in recent years. However, this task remains challenging due to the difficulty of considering ambiguous occlusions and large pose variations. To address these difficulties, we propose a new framework called Mask-FPAN. Our framework includes a de-occlusion module that learns to parse occluded faces in a semi-supervised manner, taking into account face landmark localization, face occlusion estimations, and detected head poses. Additionally, we improve the robustness of 2D face parsing by combining a 3D morphable face model with the UV GAN. We also introduce two new datasets, named FaceOccMask-HQ and CelebAMaskOcc-HQ, to aid in face parsing work. Our proposed Mask-FPAN framework successfully addresses the challenge of face parsing in the wild and achieves significant performance improvements, with a mIoU increase from 0.7353 to 0.9013 compared to the current state-of-the-art on challenging face datasets.

Pose- and Attribute-consistent Person Image Synthesis

Person Image Synthesis aims at transferring the appearance of the source person image into a target pose. Existing methods cannot handle large pose variations and therefore suffer from two critical problems: (1) synthesis distortion due to the entanglement of pose and appearance information among different body components and (2) failure in preserving original semantics (e.g., the same outfit). In this article, we explicitly address these two problems by proposing a Pose- and Attribute-consistent Person Image Synthesis Network (PAC-GAN). To reduce pose and appearance matching ambiguity, we propose a component-wise transferring model consisting of two stages. The former stage focuses only on synthesizing target poses, while the latter renders target appearances by explicitly transferring the appearance information from the source image to the target image in a component-wise manner. In this way, source-target matching ambiguity is eliminated due to the component-wise disentanglement of pose and appearance synthesis. Second, to maintain attribute consistency, we represent the input image as an attribute vector and impose a high-level semantic constraint using this vector to regularize the target synthesis. Extensive experimental results on the DeepFashion dataset demonstrate the superiority of our method over the state of the art, especially for maintaining pose and attribute consistencies under large pose variations.

Noise-related face image recognition based on double dictionary transform learning

The existing single dictionary learning algorithms are applied to face recognition and achieve satisfactory results. However, their performance is poor when dealing with noisy images and images involving complex variations such as large pose variations and occlusions. In this paper, a novel noise-related face image recognition method based on double dictionary transform learning (DDTL) is proposed. On the one hand, DDTL introduces the L2,p-norm to remove the redundant information in the dictionary and the noise involved in the training images, which makes the learned dictionary more discriminative. On the other hand, DDTL introduces the analysis dictionary and performs double dictionary transform learning with the synthetic dictionary. This can better reveal the relationship between the samples and the representation coefficients, and improve the accuracy of the learned dictionary and representation coefficients. Besides, DDTL introduces a linear regression term in the model learning process, which can distinguish and expand the differences between classes. Experimental results on six databases show that DDTL is superior to existing methods.

Face Patches Designed Through Neuroevolution for Face Recognition With Large Pose Variation

Face Patches Designed Through Neuroevolution for Face Recognition With Large Pose Variation

A Symmetrical Siamese Network Framework With Contrastive Learning for Pose-Robust Face Recognition.

Face recognition has achieved remarkable success owing to the development of deep learning. However, most of existing face recognition models perform poorly against pose variations. We argue that, it is primarily caused by pose-based long-tailed data - imbalanced distribution of training samples between profile faces and near-frontal faces. Additionally, self-occlusion and nonlinear warping of facial textures caused by large pose variations also increase the difficulty in learning discriminative features of profile faces. In this study, we propose a novel framework called Symmetrical Siamese Network (SSN), which can simultaneously overcome the limitation of pose-based long-tailed data and pose-invariant features learning. Specifically, two sub-modules are proposed in the SSN, i.e., Feature-Consistence Learning sub-Net (FCLN) and Identity-Consistence Learning sub-Net (ICLN). For FCLN, the inputs are all face images on training dataset. Inspired by the contrastive learning, we simulate pose variations of faces and constrain the model to focus on the consistent areas between the original face image and its corresponding virtual pose face images. For ICLN, only profile images are used as inputs, and we propose to adopt Identity Consistence Loss to minimize the intra-class feature variation across different poses. The collaborative learning of two sub-modules guarantees that the parameters of network are updated in a relatively equal probability between near-frontal face images and profile images, so that the pose-based long-tailed problem can be effectively addressed. The proposed SSN shows comparable results over the state-of-the-art methods on several public datasets. In this study, LightCNN is selected as the backbone of SSN, and existing popular networks also can be used into our framework for pose-robust face recognition.

LDA-CNN: Linear Discriminant Analysis Convolution Neural Network for Periocular Recognition in the Wild

Due to the COVID-19 pandemic, the necessity for a contactless biometric system able to recognize masked faces drew attention to the periocular region as a valuable biometric trait. However, periocular recognition remains challenging for deployments in the wild or in unconstrained environments where images are captured under non-ideal conditions with large variations in illumination, occlusion, pose, and resolution. These variations increase within-class variability and between-class similarity, which degrades the discriminative power of the features extracted from the periocular trait. Despite the remarkable success of convolutional neural network (CNN) training, CNN requires a huge volume of data, which is not available for periocular recognition. In addition, the focus is on reducing the loss between the actual class and the predicted class but not on learning the discriminative features. To address these problems, in this paper we used a pre-trained CNN model as a backbone and introduced an effective deep CNN periocular recognition model, called linear discriminant analysis CNN (LDA-CNN), where an LDA layer was incorporated after the last convolution layer of the backbone model. The LDA layer enforced the model to learn features so that the within-class variation was small, and the between-class separation was large. Finally, a new fully connected (FC) layer with softmax activation was added after the LDA layer, and it was fine-tuned in an end-to-end manner. Our proposed model was extensively evaluated using the following four benchmark unconstrained periocular datasets: UFPR, UBIRIS.v2, VISOB, and UBIPr. The experimental results indicated that LDA-CNN outperformed the state-of-the-art methods for periocular recognition in unconstrained environments. To interpret the performance, we visualized the discriminative power of the features extracted from different layers of the LDA-CNN model using the t-distributed Stochastic Neighboring Embedding (t-SNE) visualization technique. Moreover, we conducted cross-condition experiments (cross-light, cross-sensor, cross-eye, cross-pose, and cross-database) that proved the ability of the proposed model to generalize well to different unconstrained conditions.

Heatmap Regression via Randomized Rounding.

Heatmap regression has become the mainstream methodology for deep learning-based semantic landmark localization, including in facial landmark localization and human pose estimation. Though heatmap regression is robust to large variations in pose, illumination, and occlusion in unconstrained settings, it usually suffers from a sub-pixel localization problem. Specifically, considering that the activation point indices in heatmaps are always integers, quantization error thus appears when using heatmaps as the representation of numerical coordinates. Previous methods to overcome the sub-pixel localization problem usually rely on high-resolution heatmaps. As a result, there is always a trade-off between achieving localization accuracy and computational cost, where the computational complexity of heatmap regression depends on the heatmap resolution in a quadratic manner. In this paper, we formally analyze the quantization error of vanilla heatmap regression and propose a simple yet effective quantization system to address the sub-pixel localization problem. The proposed quantization system induced by the randomized rounding operation 1) encodes the fractional part of numerical coordinates into the ground truth heatmap using a probabilistic approach during training; and 2) decodes the predicted numerical coordinates from a set of activation points during testing. We prove that the proposed quantization system for heatmap regression is unbiased and lossless. Experimental results on popular facial landmark localization datasets (WFLW, 300W, COFW, and AFLW) and human pose estimation datasets (MPII and COCO) demonstrate the effectiveness of the proposed method for efficient and accurate semantic landmark localization. Code is available at http://github.com/baoshengyu/H3R.

Pose-Invariant Face Recognition via Adaptive Angular Distillation

Pose-invariant face recognition is a practically useful but challenging task. This paper introduces a novel method to learn pose-invariant feature representation without normalizing profile faces to frontal ones or learning disentangled features. We first design a novel strategy to learn pose-invariant feature embeddings by distilling the angular knowledge of frontal faces extracted by teacher network to student network, which enables the handling of faces with large pose variations. In this way, the features of faces across variant poses can cluster compactly for the same person to create a pose-invariant face representation. Secondly, we propose a Pose-Adaptive Angular Distillation loss to mitigate the negative effect of uneven distribution of face poses in the training dataset to pay more attention to the samples with large pose variations. Extensive experiments on two challenging benchmarks (IJB-A and CFP-FP) show that our approach consistently outperforms the existing methods.

CQA-Face: Contrastive Quality-Aware Attentions for Face Recognition

Few existing face recognition (FR) models take local representations into account. Although some works achieved this by extracting features on cropped parts around face landmarks, landmark detection may be inaccurate or even fail in some extreme cases. Recently, without relying on landmarks, attention-based networks can focus on useful parts automatically. However, there are two issues: 1) It is noticed that these approaches focus on few facial parts, while missing other potentially discriminative regions. This can cause performance drops when emphasized facial parts are invisible under heavy occlusions (e.g. face masks) or large pose variations; 2) Different facial parts may appear at various quality caused by occlusion, blur, or illumination changes. In this paper, we propose contrastive quality-aware attentions, called CQA-Face, to address these two issues. First, a Contrastive Attention Learning (CAL) module is proposed, pushing models to explore comprehensive facial parts. Consequently, more useful parts can help identification if some facial parts are invisible. Second, a Quality-Aware Network (QAN) is developed to emphasize important regions and suppress noisy parts in a global scope. Thus, our CQA-Face model is developed by integrating the CAL with QAN, which extracts diverse quality-aware local representations. It outperforms the state-of-the-art methods on several benchmarks, demonstrating its effectiveness and usefulness.

Robust Facial Landmark Detection by Multiorder Multiconstraint Deep Networks.

Recently, heatmap regression has been widely explored in facial landmark detection and obtained remarkable performance. However, most of the existing heatmap regression-based facial landmark detection methods neglect to explore the high-order feature correlations, which is very important to learn more representative features and enhance shape constraints. Moreover, no explicit global shape constraints have been added to the final predicted landmarks, which leads to a reduction in accuracy. To address these issues, in this article, we propose a multiorder multiconstraint deep network (MMDN) for more powerful feature correlations and shape constraints' learning. Especially, an implicit multiorder correlating geometry-aware (IMCG) model is proposed to introduce the multiorder spatial correlations and multiorder channel correlations for more discriminative representations. Furthermore, an explicit probability-based boundary-adaptive regression (EPBR) method is developed to enhance the global shape constraints and further search the semantically consistent landmarks in the predicted boundary for robust facial landmark detection. It is interesting to show that the proposed MMDN can generate more accurate boundary-adaptive landmark heatmaps and effectively enhance shape constraints to the predicted landmarks for faces with large pose variations and heavy occlusions. Experimental results on challenging benchmark data sets demonstrate the superiority of our MMDN over state-of-the-art facial landmark detection methods.

Unconstrained face sketch synthesis via perception-adaptive network and a new benchmark

Face sketch generation has attracted much attention in the field of visual computing. However, existing methods either are limited to constrained conditions or heavily rely on various preprocessing steps to deal with in-the-wild cases. In this paper, we argue that accurately perceiving facial region and facial components is crucial for unconstrained sketch synthesis. To this end, we propose a novel Perception-Adaptive Network (PANet), which can generate high-quality face sketches under unconstrained conditions in an end-to-end scheme. Specifically, our PANet is composed of: i) a Fully Convolutional Encoder for hierarchical feature extraction, ii) a Face-Adaptive Perceiving Decoder for extracting potential facial region and handling face variations, and iii) a Component-Adaptive Perceiving Module for facial component aware feature representation learning. To facilitate further researches of unconstrained face sketch synthesis, we introduce a new benchmark termed WildSketch, which contains 800 pairs of face photo-sketch with large variations in pose, expression, ethnic origin, background, and illumination. Extensive experiments demonstrate that the proposed method is capable of achieving state-of-the-art performance under both constrained and unconstrained conditions. Our source codes and the WildSketch benchmark are resealed on the project pagehttp://lingboliu.com/unconstrained_face_sketch.html.

Learning Contrastive Representation for Semantic Correspondence

Dense correspondence across semantically related images has been extensively studied, but still faces two challenges: 1) large variations in appearance, scale and pose exist even for objects from the same category, and 2) labeling pixel-level dense correspondences is labor intensive and infeasible to scale. Most existing methods focus on designing various matching modules using fully-supervised ImageNet pretrained networks. On the other hand, while a variety of self-supervised approaches are proposed to explicitly measure image-level similarities, correspondence matching the pixel level remains under-explored. In this work, we propose a multi-level contrastive learning approach for semantic matching, which does not rely on any ImageNet pretrained model. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects, while the performance can be further enhanced by regularizing cross-instance cycle-consistency at intermediate feature levels. Experimental results on the PF-PASCAL, PF-WILLOW, and SPair-71k benchmark datasets demonstrate that our method performs favorably against the state-of-the-art approaches.

Hybrid Modality Metric Learning for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (Re-ID) has received increasing research attention for its great practical value in night-time surveillance scenarios. Due to the large variations in person pose, viewpoint, and occlusion in the same modality, as well as the domain gap brought by heterogeneous modality, this hybrid modality person matching task is quite challenging. Different from the metric learning methods for visible person re-ID, which only pose similarity constraints on class level, an efficient metric learning approach for visible-infrared person Re-ID should take both the class-level and modality-level similarity constraints into full consideration to learn sufficiently discriminative and robust features. In this article, the hybrid modality is divided into two types, within modality and cross modality. We first fully explore the variations that hinder the ranking results of visible-infrared person re-ID and roughly summarize them into three types: within-modality variation, cross-modality modality-related variation, and cross-modality modality-unrelated variation. Then, we propose a comprehensive metric learning framework based on four kinds of paired-based similarity constraints to address all the variations within and cross modality. This framework focuses on both class-level and modality-level similarity relationships between person images. Furthermore, we demonstrate the compatibility of our framework with any paired-based loss functions by giving detailed implementation of combing it with triplet loss and contrastive loss separately. Finally, extensive experiments of our approach on SYSU-MM01 and RegDB demonstrate the effectiveness and superiority of our proposed metric learning framework for visible-infrared person Re-ID.

Cascaded Split-and-Aggregate Learning with Feature Recombination for Pedestrian Attribute Recognition

Multi-label pedestrian attribute recognition in surveillance is inherently a challenging task due to poor imaging quality, large pose variations, and so on. In this paper, we improve its performance from the following two aspects: (1) We propose a cascaded Split-and-Aggregate Learning (SAL) to capture both the individuality and commonality for all attributes, with one at the feature map level and the other at the feature vector level. For the former, we split the features of each attribute by using a designed attribute-specific attention module (ASAM). For the later, the split features for each attribute are learned by using constrained losses. In both modules, the split features are aggregated by using several convolutional or fully connected layers. (2) We propose a Feature Recombination (FR) that conducts a random shuffle based on the split features over a batch of samples to synthesize more training samples, which spans the potential samples’ variability. To the end, we formulate a unified framework, named CAScaded Split-and-Aggregate Learning with Feature Recombination (CAS-SAL-FR), to learn the above modules jointly and concurrently. Experiments on five popular benchmarks, including RAP, PA-100K, PETA, Market-1501 and Duke attribute datasets, show the proposed CAS-SAL-FR achieves new state-of-the-art performance.

Robust Face Recognition Under Partial Occlusion Based on Local Generic Features

Face recognition has drawn significant attention due to its potential use in biometric authentication, surveillance, security, robotics, and so on. It is a challenging task in the field of computer vision. Although the various state-of-the-art methods of face recognition in constrained environments have achieved satisfactory results, there are still many issues which are untouched in unconstrained environments, such as partial occlusions, large pose variations, etc. In this paper, the authors have proposed an approach which utilized the local generic feature (LGF) to recognize the face in the partial occlusion by fusing features scale invariant feature transform (SIFT) and multi-block local binary pattern (MB-LBP). It also utilizes robust kernel method for classification of the query image. They have validated the effectiveness of the proposed approach on the benchmark AR face database. The experimental outcomes illustrate that the proposed approach outperformed the state-of-art methods for robust face recognition.