Pose Invariant Features Research Articles

Multitask Learning Strategy with Pseudo-Labeling: Face Recognition, Facial Landmark Detection, and Head Pose Estimation

Most facial analysis methods perform well in standardized testing but not in real-world testing. The main reason is that training models cannot easily learn various human features and background noise, especially for facial landmark detection and head pose estimation tasks with limited and noisy training datasets. To alleviate the gap between standardized and real-world testing, we propose a pseudo-labeling technique using a face recognition dataset consisting of various people and background noise. The use of our pseudo-labeled training dataset can help to overcome the lack of diversity among the people in the dataset. Our integrated framework is constructed using complementary multitask learning methods to extract robust features for each task. Furthermore, introducing pseudo-labeling and multitask learning improves the face recognition performance by enabling the learning of pose-invariant features. Our method achieves state-of-the-art (SOTA) or near-SOTA performance on the AFLW2000-3D and BIWI datasets for facial landmark detection and head pose estimation, with competitive face verification performance on the IJB-C test dataset for face recognition. We demonstrate this through a novel testing methodology that categorizes cases as soft, medium, and hard based on the pose values of IJB-C. The proposed method achieves stable performance even when the dataset lacks diverse face identifications.

Joint spatial and scale attention network for multi-view facial expression recognition

Multi-view facial expression recognition (FER) is a challenging task because the appearance of an expression varies greatly due to poses. To alleviate the influences of poses, recently developed methods perform pose normalization, learn pose-invariant features, or learn pose-specific FER classifiers. However, these methods usually rely on a prerequisite pose estimator or expressive region detector that is independent of the subsequent expression analysis. Different from existing methods, we propose a joint spatial and scale attention network (SSA-Net) to localize proper regions for simultaneous head pose estimation (HPE) and FER. Specifically, SSA-Net discovers the regions most relevant to the facial expression at hierarchical scales by a spatial attention mechanism, and the most informative scales are selected in a scale attention learning manner to learn the joint pose-invariant and expression-discriminative representations. Then, we employ a dynamically constrained multi-task learning mechanism with a delicately designed constrain regulation to properly and adaptively train the network to optimize the representations, thus achieving accurate multi-view FER. The effectiveness of the proposed SSA-Net is validated on three multi-view datasets (BU-3DFE, Multi-PIE, and KDEF) and three in-the-wild FER datasets (AffectNet, SFEW, and FER2013). Extensive experiments demonstrate that the proposed framework outperforms existing state-of-the-art methods under both within-dataset and cross-dataset settings, with relative accuracy gains of 2.36%, 1.33%, 3.11%, 2.84%, 15.7%, and 7.57%, respectively.

A Symmetrical Siamese Network Framework With Contrastive Learning for Pose-Robust Face Recognition.

Face recognition has achieved remarkable success owing to the development of deep learning. However, most of existing face recognition models perform poorly against pose variations. We argue that, it is primarily caused by pose-based long-tailed data - imbalanced distribution of training samples between profile faces and near-frontal faces. Additionally, self-occlusion and nonlinear warping of facial textures caused by large pose variations also increase the difficulty in learning discriminative features of profile faces. In this study, we propose a novel framework called Symmetrical Siamese Network (SSN), which can simultaneously overcome the limitation of pose-based long-tailed data and pose-invariant features learning. Specifically, two sub-modules are proposed in the SSN, i.e., Feature-Consistence Learning sub-Net (FCLN) and Identity-Consistence Learning sub-Net (ICLN). For FCLN, the inputs are all face images on training dataset. Inspired by the contrastive learning, we simulate pose variations of faces and constrain the model to focus on the consistent areas between the original face image and its corresponding virtual pose face images. For ICLN, only profile images are used as inputs, and we propose to adopt Identity Consistence Loss to minimize the intra-class feature variation across different poses. The collaborative learning of two sub-modules guarantees that the parameters of network are updated in a relatively equal probability between near-frontal face images and profile images, so that the pose-based long-tailed problem can be effectively addressed. The proposed SSN shows comparable results over the state-of-the-art methods on several public datasets. In this study, LightCNN is selected as the backbone of SSN, and existing popular networks also can be used into our framework for pose-robust face recognition.

Pose-Invariant Face Recognition via Adaptive Angular Distillation

Pose-invariant face recognition is a practically useful but challenging task. This paper introduces a novel method to learn pose-invariant feature representation without normalizing profile faces to frontal ones or learning disentangled features. We first design a novel strategy to learn pose-invariant feature embeddings by distilling the angular knowledge of frontal faces extracted by teacher network to student network, which enables the handling of faces with large pose variations. In this way, the features of faces across variant poses can cluster compactly for the same person to create a pose-invariant face representation. Secondly, we propose a Pose-Adaptive Angular Distillation loss to mitigate the negative effect of uneven distribution of face poses in the training dataset to pay more attention to the samples with large pose variations. Extensive experiments on two challenging benchmarks (IJB-A and CFP-FP) show that our approach consistently outperforms the existing methods.

PIFS Scheme for HEad Pose Estimation Aimed at Faster Face Recognition

In the context of 2D face biometrics, pose represents one challenging intra-class variation that, over the years, has been approached in a number of different ways, such as through inherently pose-invariant features, or by means of specifically trained deep networks, or even by frontalization techniques aimed to restore a canonical pose. To this regard, the idea behind this paper is to perform head pose estimation (HPE) previous to the face recognition step, so that the probe’s resulting yaw, pitch and roll values can be used to possibly find the pose-wise closest elements in the gallery, thus reducing one-to-many matching time. We exploit a common approach to both pose estimation and feature matching, where fractal encoding is used to extract a feature vector and to compare it to a reference template through a metric distance for both HPE and the facial recognition. Experiments conducted on the Biwi Kinect Head Pose database and the AFLW2000 dataset show state-of-the art level of precision for head pose estimation along with a 10x efficiency boost in the overall face recognition task, compared to popular deep learning approaches, whilst achieving also an edge in terms of recognition accuracy.

Pose-invariant matching for non-rigid 3D models using Isomap.

The wide usage of 3D mesh models greatly increases the importance of an effective matching algorithm for them. In this paper, we propose a novel 3D model matching algorithm. Firstly, vertices on the input 3D mesh models are mapped to 1D space by employing Isomap. A pose-invariant feature set is then constructed from the vertices in 1D space. Finally, the similarity between any two 3D models can be computed by comparing their feature sets. Experimental results show that the algorithm is not only invariant to translation, rotation, scaling, but also invariant to different poses of 3D models. Additionally, the algorithm is robust to noise.

Frontal-Centers Guided Face: Boosting Face Recognition by Learning Pose-Invariant Features

In recent years, face recognition has made a remarkable breakthrough due to the emergence of deep learning. However, compared with frontal face recognition, plenty of deep face recognition models still suffer serious performance degradation when handling profile faces. To address this issue, we propose a novel Frontal-Centers Guided Loss (FCGFace) to obtain highly discriminative features for face recognition. Most existing discriminative feature learning approaches project features from the same class into a separated latent subspace. These methods only model the distribution at the identity-level but ignore the latent relationship between frontal and profile viewpoints. Different from these methods, FCGFace takes viewpoints into consideration by modeling the distribution at both the identity-level and the viewpoint-level. At the identity-level, a softmax-based loss is employed for a relatively rough classification. At the viewpoint-level, centers of frontal face features are defined to guide the optimization conducted in a more refined way. Specifically, our FCGFace is capable of adaptively adjusting the distribution of profile face features and narrowing the gap between them and frontal face features during different training stages to form compact identity clusters. Extensive experimental results on popular benchmarks, including cross-pose datasets (CFP-FP, CPLFW, VGGFace2-FP, and Multi-PIE) and non-cross-pose datasets (YTF, LFW, AgeDB-30, CALFW, IJB-B, IJB-C, and RFW), have demonstrated the superiority of our FCGFace over the SOTA competitors.

Cross-domain person re-identification with pose-invariant feature decomposition and hypergraph structure alignment

Person Re-identification (Re-ID) has attracted more and more attention thanks to its great practical value in the field of video surveillance. Most works have focused on solving the problem of supervised Re-ID on a single domain and made significant progress. However, the cross-domain Re-ID is still challenging due to the domain bias between the source and target domains. To this end, we propose a dictionary learning algorithm based on matrix factorization to eliminate the influence of style and pedestrian pose information on the cross-domain Re-ID. Specifically, the proposed approach includes two novel parts: (1) the original visual feature is decomposed into pose-invariant feature space, camera-style feature space and residual feature space to extract discriminant pose-invariant feature that is not affected by style and pedestrian pose information, such that the influence of interference information between pedestrians on recognition can be eliminated; (2) considering the domain-invariance of attribute, a hypergraph structure alignment is introduced to integrate pose-invariant feature, attribute and pedestrian identity into a dictionary learning framework. The relationship between pose-invariant feature and attribute is built so that the pedestrian attribute of the target dataset can be accurately predicted during testing. Finally, the pedestrian similarity measurement can be carried out by combining the pose-invariant feature and attribute of pedestrians. The effectiveness of the proposed algorithm is verified with the experiments on several benchmark Re-ID datasets.

Pose and Illumination Invariant Hybrid Feature Extraction for Newborn

Background: Abduction, swapping and mix-ups are the unfortunate events that could happen to Newborn while in hospital premises and medical personnel are finding it difficult to curb this unfortunate incident. Accurate patient identification (ID) is essential for patient safety, especially with our smallest and most vulnerable paediatric patients. The level of security is very crucial issue in maternity ward and the problem of missing and swapping of Newborn is of prime concern to the persons involved and affected. There is a common perception in the society that nothing can be done to prevent this unfortunate tragedy. In comparison to developed nations the developing countries are facing more challenges because of overcrowding and scarcity of medical facilities in the hospital. The face of a newborn baby changes appearance in different lighting conditions in spite of various contrast enhancement techniques. The class of images which are under similar environment conditions (inter-class) may result in more differences than that of the intra-class images under different illumination conditions. This gives rise to the misclassification of images. Objective: The main objective of this paper is perform newborn face recognition in a hybrid approach. Speeded Up Robust Features (SURF) and Local Binary Pattern (LBP). Methods: The scientific contributions of the proposed work are stated below: For face recognition of newborns in a semi controlled environment. Overcoming the pose and illumination challenge. Uses single gallery image. Uses a hybrid approach to improve the results. Conclusion: In this paper new semi supervised technique is used for demonstrate the improve the performance of the newborn face recognition system in different illumination and pose conditions. In our. Results: The average Rank 1 accuracy of the proposed method is 93.65% whereas that of the existing algorithms is quite low. The proposed technique is 12.8% more accurate than LBP, 10.5% more accurate than SURF, 12.8% more accurate than LDA and 18.1% more accurate than PCA for rank1.

Person Reidentification Based on Pose-Invariant Feature and B-KNN Reranking

Person reidentification, as one of the most important areas in the video surveillance field, is a crucial task in computer vision. It has attracted more and more attention in both academic research and industry due to its extremely high application value. However, the recognition accuracy of person reidentification is subjected to factors, such as illumination, pose, occlusion, and viewpoint. To alleviate the effect of such factors, the multiscale Retinex with color restoration (MSRSC) algorithm is adopted to preprocess the original images so that the color information can be restored and the illumination condition can be improved. To obtain pose-invariant features (PIFs), the convolution pose machine that can generate the body joint points of pedestrians is applied to divide the body into seven parts, the pose transform network is then used to align the body parts, and finally, the PIFs can be obtained from a designed pseudo-Siamese network by using the original and aligned images as the training samples. To further improve recognition accuracy, a reranking method based on bidirectional k-nearest neighbors (KNN) is presented to optimize the ranking list. Experimentally, the proposed method is implemented on three data sets: viewpoint invariant pedestrian recognition (VIPeR), CUHK03, and Market1501. The results demonstrate our method outperforms the other methods, as a result of both the representation of a more discriminative feature descriptor and the introduction of a reranking method.

Transformation guided representation GAN for pose invariant face recognition

Face recognition is an important topic in the field of computer vision and has been a vital biometric technique for identity authentication. It is widely used in areas such as public security, military, and daily life. However, face recognition is inherently a challenging problem due to variations in poses, facial expressions, age, and occlusion. In this work, we propose a generative adversarial network (GAN) architecture that disentangles identity and pose variations to learn generative and discriminative representations for pose-invariant face recognition. We use an iterative warping scheme that achieves better results than with the use of a single generator. The features from the encoder are considered pose-invariant features for face recognition, and evaluations on databases demonstrate the usefulness of this approach over prior methods. For example, we report 97.0% (+ 12.7%) and 90.5% (+ 8.4%) accuracy on the Feret and Caspeal datasets compared to 78.2% achieved by the best method without warping. In particular, there are two notable novelties. First, the disentangled architecture GAN (D-GAN) performs frontal face synthesis via an encoder-decoder structure in the generator with the pose variations provided to the decoder and discriminator. Second, we utilize the generator encoder as a spatial transformer network that seeks realistic image synthesis in the geometric warp parameter space.

Boosting Face Recognition under Drastic Views Using a Pose AutoAugment Manner

Face recognition under drastic pose drops rapidly due to the limited samples during the model training. In this paper, we propose a pose-autoaugment face recognition framework (PAFR) based on the training of a Convolutional Neural Network (CNN) with multi-view face augmentation. The proposed framework consists of three parts: face augmentation, CNN training, and face matching. The face augmentation part is composed of pose autoaugment and background appending for increasing the pose variations of each subject. In the second part, we train a CNN model with the generated facial images to enhance the pose-invariant feature extraction. In the third part, we concatenate the feature vectors of each face and its horizontally flipped face from the trained CNN model to obtain a robust feature. The correlation score between the two faces is computed by the cosine similarity of their robust features. Comparable experiments are demonstrated on Bosphorus and CASIA-3D databases.

Patch-based pose invariant features for single sample face recognition

Pose variation is considered as one of the major challenges that degrade the performance of face recognition systems. Existing techniques address this problem from different attitudes. However, these methods may be inefficient or impractical in the case of single sample face recognition. This article presents an automatic patch-based pose invariant feature extraction method that can handle pose variations for the aforementioned case. The proposed method extracts Gabor and histograms of oriented gradients features from landmark-based patches. The features are then concatenated, dimensionally reduced using principal component analysis, fused using canonical correlation analysis, and normalized using min-max normalization. Experimental results carried out on the FERET database have shown the outstanding performance of the proposed method compared to that of the state-of-the-art approaches. The proposed approach achieved $$100\%$$ and $$96\%$$ and $$94.5\%$$ recognition rates for moderate and wide pose variations, respectively.

Combining Pose Invariant and Discriminative Features for Vehicle Reidentification

Vehicle reidentification, aiming at identifying vehicles across images, has drawn a lot of attention and has made significant achievements in recent years. However, vehicle reidentification remains a challenging task caused by severe appearance changes due to different orientations. In practice, the result of reidentification is greatly influenced by the pose of vehicles, and we call this influence as a pose barrier problem. One way to address the pose barrier problem is to train a feature representation that is invariant for various vehicle poses. To this end, we present pose robust features (PRFs) that contains two components: 1) pose-invariant features (PIFs) and 2) pose discriminative features (PDFs). On the one hand, PIF is the expert in exploring the overall characteristic of vehicles. When training PIF, we adopt an identity classifier as well as an orientation classifier. In addition, an adversarial loss is deployed in the PIF network. On the other hand, we design a PDF network, which has a similar architecture to the PIF network but can distinguish the difference between local details. The difference between PDF and PIF is that the network of training PDF does not apply the adversarial loss. Finally, by combining PIF and PDF, PRF has the advantages of the two features and can alleviate the influence of the pose barrier problem. Experiments are conducted on the VeRi-776 and VehicleID data sets. We show that PIF and PDF are complementary and that PRF produces competitive performance compared with state-of-the-art approaches.

Look More Into Occlusion: Realistic Face Frontalization and Recognition With BoostGAN.

Many factors can affect face recognition, such as occlusion, pose, aging, and illumination. First and foremost are occlusion and large-pose problems, which may even lead to more than 10% accuracy degradation. Recently, generative adversarial net (GAN) and its variants have been proved to be effective in processing pose and occlusion. For the former, pose-invariant feature representation and face frontalization based on GAN models have been studied to solve the pose variation problem. For the latter, frontal face completion on occlusions based on GAN models have also been presented, which is much concerned with facial structure and realistic pixel details rather than identity preservation. However, synthesizing and recognizing the occluded but profile faces is still an understudied problem. Therefore, in this article, to address this problem, we contribute an efficient but effective solution on how to synthesize and recognize faces with large-pose variations and simultaneously corrupted regions (e.g., nose and eyes). Specifically, we propose a boosting GAN (BoostGAN) for occluded but profile face frontalization, deocclusion, and recognition, which has two aspects: 1) with the assumption that face occlusion is incomplete and partial, multiple images with patch occlusion are fed into our model for knowledge boosting, i.e., identity and texture information and 2) a new aggregation structure integrated with a deep encoder-decoder network for coarse face synthesis and a boosting network for fine face generation is carefully designed. Exhaustive experiments on benchmark data sets with regular and irregular occlusions demonstrate that the proposed model not only shows clear photorealistic images but also presents powerful recognition performance over state-of-the-art GAN models for occlusive but profile face recognition in both the controlled and uncontrolled environments. To the best of our knowledge, this article proposes to solve face synthesis and recognition under poses and occlusions for the first time.

Scale and Pose-Invariant Feature Quality Inspection for Freeform Geometries in Additive Manufacturing

Abstract Additive manufacturing (AM) has the unprecedented ability to create customized, complex, and nonparametric geometry, and it has made this ability accessible to individuals outside of traditional production environments. Geometric inspection technology, however, has yet to adapt to take full advantage of AM’s abilities. Coordinate measuring machines are accurate, but they are also slow, expensive to operate, and inaccessible to many AM users. On the other hand, 3D-scanners provide fast, high-density measurements, but there is a lack of feature-based analysis techniques for point cloud data. There exists a need for developing fast, feature-based geometric inspection techniques that can be implemented by users without specialized training in inspection according to geometric dimensioning and tolerancing conventions. This research proposes a new scale- and pose-invariant quality inspection method based on a novel location-orientation-shape (LOS) distribution derived from point cloud data. The key technique of the new method is to describe the shape and pose of key features via kernel density estimation and detect nonconformities based on statistical divergence. Numerical examples are provided and tests on physical AM builds are conducted to validate the method. The results show that the proposed inspection scheme is able to identify form, position, and orientation defects. The results also demonstrate how datum features can be incorporated into point cloud inspection, that datum features can be complex, nonparametric surfaces, and how the specification of datums can be more intuitive and meaningful, particularly for users without special training.

Pose-Guided Representation Learning for Person Re-Identification.

The large pose variations and misalignment errors exhibited by person images significantly increase the difficulty of person Re-Identification (ReID). Existing works commonly apply extra operations like pose estimation, part segmentation, etc., to alleviate those issues and improve the robustness of pedestrian representations. While boosting the ReID accuracy, those operations introduce considerable computational overheads and make the deep models complex and hard to tune. To chase a more efficient solution, we propose a Part-Guided Representation (PGR) composed of Pose Invariant Feature (PIF) and Local Descriptive Feature (LDF), respectively. We call PGR "Part-Guided" because it is trained and supervised by local part cues. Specifically, PIF approximates a pose invariant representation inferred by pose estimation and pose normalization. LDF focuses on discriminative body parts by approximating a representation learned with body region segmentation. In this way, extra pose extraction is only introduced during the training stage to supervise the learning of PGR, but is not required during the testing stage for feature extraction. Extensive comparisons with recent works on five widely used datasets demonstrate the competitive accuracy and efficiency of PGR.

Learning Identity-Aware Face Features Across Poses Based on Deep Siamese Networks

Face recognition is an important biometric due to its non-intrusive collection of data that can be applied to surveillance systems. However, the human pose is unconstrained under surveillance, and there is only one frontal face available in the gallery in most scenarios, and these two factors challenge face recognition performance. A goodly amount of research has been published to solve this problem. Specially, deep learning-based methods learn generic models between poses to synthesize different poses of a given face, however, generic synthesis models can lose the face identity while warping the face, which deteriorates the discriminative capability of learned features. In this paper, we proposed the deep Siamese networks to learn identity-aware and pose-invariant features, adding contrastive loss to the face synthesis model to preserve the face identity while synthesizing the face. In addition, we trained various face synthesis models with different target poses as supervisory signals, the learned pose-invariant features were incorporated by another Siamese network, resulting in deeper pose-invariant and identity-aware features. The proposed network is free of landmark estimation and face pose, and it is in real time. We tested the proposed algorithm in the FERET and CAS-PEAL datasets, and experimental results demonstrated that our network achieved superior performance to that of recently published algorithms for cross-pose face recognition, especially the 2D deep learning-based algorithms.

Mutually reinforcing motion-pose framework for pose invariant action recognition

Action recognition from videos has many potential applications. However, there are many unresolved challenges, such as pose-invariant recognition, robustness to occlusion and others. In this paper, we propose to combine motion of body parts and pose hypothesis generation validated with specific canonical poses observed in a novel mutually reinforcing framework to achieve pose-invariant action recognition. To capture the temporal dynamics of an action, we introduce temporal stick features computed using the stick poses obtained. The combination of pose-invariant kinematic features from motion, pose hypothesis and temporal stick features are used for action recognition, thus forming a mutually reinforcing framework that repeats until the action recognition result converges. The proposed mutual reinforcement framework is capable of handling changes in posture of the person, occlusion and partial view-invariance. We perform experiments on several benchmark datasets which showed the performance of the proposed algorithm and its ability to handle pose variation and occlusion.

A spatial self-similarity based feature learning method for face recognition under varying poses

In this paper, we propose a low-complexity method to learn pose-invariant features for face recognition with no need for pose information. In contrast to the commonly used approaches of recovering frontal face images from profile views, the proposed method extracts the subject related part from a local feature by removing its pose related part. First, the method generates a self-similarity feature by computing the distances between local feature descriptors of different non-overlapping blocks in a face image. Secondly, it subtracts from the local feature a linear transformation of the self-similarity feature and the transformation matrix is learned through minimizing the feature distance between face images from the same person but under different poses while retaining the discriminative information across different persons. In order to evaluate our method, extensive experiments on face recognition across poses are conducted using FERET and Multi-PIE, in addition, experiments on face recognition under unconstrained situations are conducted using LFW-a. Results on these three public databases show that the proposed method is able to significantly improve the recognition performance as compared with using the original local features and outperforms or is comparable to related, state-of-the-art pose-invariant face recognition approaches.