Significant Pose Variations Research Articles

DSNet: dual-stream network for fine-grained ship classification in optical remote sensing images

ABSTRACT The fine-grained ship classification (FGSC) task in remote sensing aims to distinguish subordinate ship categories. Due to the high similarity among different subcategories and the significant variations in pose, scale, rotation, and color within the same subcategory in the fine-grained ship datasets, the FGSC task is extremely challenging. To alleviate these issues, a novel approach, named Dual-Stream Network (DSNet), is proposed in this paper for FGSC. DSNet consists of two sub-branches: the Local Residual Branch (LRB) and the Global Representation Branch (GRB). Specifically, the LRB effectively extracts the most discriminative local fine-grained features of the target using Full-scale Skip Connections (FSC) and Stripe Pooling Add (SPA) modules. The GRB captures global contextual information through Swin-Transformer (Swin-T), establishing remote dependencies of the target and extracting global coarse-grained features. Furthermore, a Lightweight Fusion (LF) block is designed in the feature fusion module to combine the feature information from the two branches. This ensures that the discriminative global and local information complement each other without overwhelming one another, thereby effectively improving the performance of the model. Extensive experiments and analyses demonstrate that our proposed method achieves state-of-the-art performance on two mainstream FGSC datasets in remote sensing.

STAR-TM: STructure Aware Reconstruction of Textured Mesh From Single Image.

We present a novel method for single-view 3D reconstruction of textured meshes, with a focus to address the primary challenge surrounding texture inference and transfer. Our key observation is that learning textured reconstruction in a structure-aware and globally consistent manner is effective in handling the severe ill-posedness of the texturing problem and significant variations in object pose and texture details. Specifically, we perform structured mesh reconstruction, via a retrieval-and-assembly approach, to produce a set of genus-zero parts parameterized by deformable boxes and endowed with semantic information. For texturing, we first transfer visible colors from the input image onto the unified UV texture space of the deformable boxes. Then we combine a learned transformer model for per-part texture completion with a global consistency loss to optimize inter-part texture consistency. Our texture completion model operates in a VQ-VAE embedding space and is trained end-to-end, with the transformer training enhanced with retrieved texture instances to improve texture completion performance amid significant occlusion. Extensive experiments demonstrate higher-quality textured mesh reconstruction obtained by our method over state-of-the-art alternatives, both quantitatively and qualitatively, as reflected by a better recovery of texture coherence and details.

Cross-domain point cloud completion for multi-class indoor incomplete objects via class-conditional GAN inversion

Cross-domain point cloud completion for multi-class indoor incomplete objects via class-conditional GAN inversion

Attention Span Prediction Using Head-Pose Estimation With Deep Neural Networks

Automated human pose estimation is evolving as an exciting research area in human activity detection. It includes sophisticated applications such as malpractice detection in the examination, distracted driving, gesture detection, etc., and requires robust and reliable pose estimation techniques. These applications help to map the attention of the user with head pose estimation (HPE) metrics supported by emotion and gaze analysis. This paper solves the problem of attention score estimation with HPE. The proposed method ensures ease of implementation while addressing head pose estimation using 68 facial features. Further, to attain reliability and precision, head pose estimation has been implemented as a regression task. The coordinate pair angle method (CPAM) with deep neural network (DNN) regression and elastic net regression is carried out. The use of DNN ensures precision on low lighting, distorted or occluded images. CPAM methodology leverages facial landmark detection and angular difference to estimate head pose. Experimentation results showed that the proposed model could handle large datasets, real-time data processing, significant pose variations, partial occlusions, and diverse facial expressions with a mean absolute error (MAE) of 3° and less. The proposed system was evaluated on three standard databases: the 300W across large poses (300W-LP) dataset, annotated facial landmarks in the wild (AFLW2000) dataset, and the national institute of mental health child emotional faces picture set (NIMH - ChEFS ) dataset. The results achieved are on par with recent state-of-the-art methodologies such as anisotropic angle distribution learning (AADL), joint head pose estimation and face alignment algorithm (JFA), rotation axis focused attention network (RAFA-Net), and propose an MAE ranging up to 6°. The paper could achieve remarkable results for attention span prediction using head pose estimation and for many possible future applications.

Simultaneous multi-descent regression and feature learning for facial landmarking in depth images

Face alignment (or facial landmarking) is an important task in many face-related applications, ranging from registration, tracking, and animation to higher-level classification problems such as face, expression, or attribute recognition. While several solutions have been presented in the literature for this task so far, reliably locating salient facial features across a wide range of posses still remains challenging. To address this issue, we propose in this paper a novel method for automatic facial landmark localization in 3D face data designed specifically to address appearance variability caused by significant pose variations. Our method builds on recent cascaded regression-based methods to facial landmarking and uses a gating mechanism to incorporate multiple linear cascaded regression models each trained for a limited range of poses into a single powerful landmarking model capable of processing arbitrary-posed input data. We develop two distinct approaches around the proposed gating mechanism: (1) the first uses a gated multiple ridge descent mechanism in conjunction with established (hand-crafted) histogram of gradients features for face alignment and achieves state-of-the-art landmarking performance across a wide range of facial poses and (2) the second simultaneously learns multiple-descent directions as well as binary features that are optimal for the alignment tasks and in addition to competitive landmarking results also ensures extremely rapid processing. We evaluate both approaches in rigorous experiments on several popular datasets of 3D face images, i.e., the FRGCv2 and Bosphorus 3D face datasets and image collections F and G from the University of Notre Dame. The results of our evaluation show that both approaches compare favorably to the state-of-the-art, while exhibiting considerable robustness to pose variations.

Tracking Persons-of-Interest via Unsupervised Representation Adaptation

Multi-face tracking in unconstrained videos is a challenging problem as faces of one person often can appear drastically different in multiple shots due to significant variations in scale, pose, expression, illumination, and make-up. Existing multi-target tracking methods often use low-level features which are not sufficiently discriminative for identifying faces with such large appearance variations. In this paper, we tackle this problem by learning discriminative, video-specific face representations using convolutional neural networks (CNNs). Unlike existing CNN-based approaches which are only trained on large-scale face image datasets offline, we automatically generate a large number of training samples using the contextual constraints for a given video, and further adapt the pre-trained face CNN to the characters in the specific videos using discovered training samples. The embedding feature space is fine-tuned so that the Euclidean distance in the space corresponds to the semantic face similarity. To this end, we devise a symmetric triplet loss function which optimizes the network more effectively than the conventional triplet loss. With the learned discriminative features, we apply an EM clustering algorithm to link tracklets across multiple shots to generate the final trajectories. We extensively evaluate the proposed algorithm on two sets of TV sitcoms and YouTube music videos, analyze the contribution of each component, and demonstrate significant performance improvement over existing techniques.

Dictionary Alignment With Re-Ranking for Low-Resolution NIR-VIS Face Recognition

Recently, near-infrared (NIR) images are increasingly being captured for recognizing faces in low-light/night-time conditions. Matching these images against the controlled high-resolution visible facial images usually present in the database is a challenging task. In surveillance scenarios, the NIR images can have very low resolution and also non-frontal pose which makes the problem even more challenging. In this paper, we propose an orthogonal dictionary alignment approach for addressing this problem. We also propose a re-ranking approach to further improve the recognition performance for each probe by combining the rank list given by the proposed algorithm with that given by another complementary feature/algorithm. Finally, we have also collected our own database Heterogeneous face recognition across Pose and Resolution (HPR) that has facial images captured from two surveillance quality NIR cameras and one high-resolution visible camera, with significant variations in head pose and resolution. Extensive experiments on the modified CASIA NIR-VIS 2.0 database, the Surveillance Camera face database, and our HPR database show the effectiveness of the proposed approaches and the collected database.

Tracking objects using Grassmann manifold appearance modeling based on wireless multimedia sensor networks

Visual object tracking methods based on wireless multimedia sensor network is one of the research hotspots while the present linear method for processing feature vectors often lead to the tracking drift when tracking object with significant nonplanar pose variations through wireless sensor networks. In this article, we propose a novel nonlinear algorithm for tracking significant deformable objects. The proposed tracking scheme has two filters. On one hand, considering that Grassmann manifold is one of entropy manifold in Lie group manifold, which can describe and process the data of appearance feature more accurately, one filter is designed on it, to estimate the object appearance, by making full use of the transformation relationship between the point on manifold and its corresponding point on tangent space. On the other hand, considering that the process of objects imaging is essentially projection transformation process, the other filter is designed on projection transformation (SL(3)) group, describing the geometric deformation of the objects. The two filters execute alternatively to mitigate tracking drift. Extensive experiments prove that the proposed method can realize stable and accurate tracking for targets with significant geometric deformation, even obscured and illumination changes.

Pose-invariant 3D face recognition using half face

Abstract Pose variations are still challenging problems in 3D face recognition because large pose variations will cause self-occlusion and result in missing data. In this paper, a new method for pose-invariant 3D face recognition is proposed to handle significant pose variations. For pose estimation and registration, a coarse-to-fine strategy is proposed to detect landmarks under large yaw variations. At the coarse search step, candidate landmarks are detected using HK curvature analysis and subdivided according to a facial geometrical structure-based classification strategy. At the fine search step, candidate landmarks are identified and labeled by comparing with a Facial Landmark Model. By using the half face matching, we perform the matching step with respect to frontal scans and side scans. Experiments carried out on the Bosphorus and UND/FRGC v2.0 databases show that our method has high accuracy and robustness to pose variations.

PBC: Polygon-Based Classifier for Fine-Grained Categorization

Fine-grained categorization is a challenging task mainly due to two factors: first, objects share similar appearances between different categories; second, objects present significant pose variation within the same category. To address these challenges, we propose a method to automatically detect discriminative and pose-invariant regions, which is referred to as a polygon-based classifier (PBC). In the first stage, we generate a set of polygons that are composed of multiple parts. For each polygon, a classifier is trained based on deep features of a convolutional network. Then, a greedy algorithm is employed to select the discriminative and complementary polygon-based classifiers that deliver highest classification accuracy for fine-grained object categories. In the second stage, the confusing classes of the first stage are selected and employed to train the polygon-based classifiers. Then, a greedy algorithm is employed to select discriminative classifiers. For the test images, we use the classifiers trained in the first stage to obtain a coarse result. Then, the classifiers of the second stage are adopted to distinguish the confusing classes of the coarse result. In our experiments, the proposed approach is evaluated on three well-known fine-grained datasets. The experiments show that our approach outperforms the state-of-the-art methods.

Robust Statistical Frontalization of Human and Animal Faces

The unconstrained acquisition of facial data in real-world conditions may result in face images with significant pose variations, illumination changes, and occlusions, affecting the performance of facial landmark localization and recognition methods. In this paper, a novel method, robust to pose, illumination variations, and occlusions is proposed for joint face frontalization and landmark localization. Unlike the state-of-the-art methods for landmark localization and pose correction, where large amount of manually annotated images or 3D facial models are required, the proposed method relies on a small set of frontal images only. By observing that the frontal facial image of both humans and animals, is the one having the minimum rank of all different poses, a model which is able to jointly recover the frontalized version of the face as well as the facial landmarks is devised. To this end, a suitable optimization problem is solved, concerning minimization of the nuclear norm (convex surrogate of the rank function) and the matrix ell _1 norm accounting for occlusions. The proposed method is assessed in frontal view reconstruction of human and animal faces, landmark localization, pose-invariant face recognition, face verification in unconstrained conditions, and video inpainting by conducting experiment on 9 databases. The experimental results demonstrate the effectiveness of the proposed method in comparison to the state-of-the-art methods for the target problems.

Sparse representation matching for person re-identification

The need for recognizing people across distributed surveillance cameras leads to the growth of recent research interest in person re-identification. Person re-identification aims at matching people in non-overlapping cameras at different time and locations. It is a difficult pattern matching task due to significant appearance variations in pose, illumination, or occlusion in different camera views. To address this multi-view matching problem, we first learn a subspace using canonical correlation analysis (CCA) in which the goal is to maximize the correlation between data from different cameras but corresponding to the same people. Given a probe from one camera view, we represent it using a sparse representation from a jointly learned coupled dictionary in the CCA subspace. The ?1 induced sparse representation are regularized by an ?2 regularization term. The introduction of ?2 regularization allows learning a sparse representation while maintaining the stability of the sparse coefficients. To compute the matching scores between probe and gallery, their ?2 regularized sparse representations are matched using a modified cosine similarity measure. Experimental results with extensive comparisons on challenging datasets demonstrate that the proposed method outperforms the state-of-the-art methods and using ?2 regularized sparse representation (?1 + ?2) is more accurate compared to use a single ?1 or ?2 regularization term.

Automatic Ear Landmark Localization, Segmentation, and Pose Classification in Range Images

Multibiometric systems using face and ear features are increasingly adopted for forensic and civilian applications to address the challenges of facial expressions and occlusions. Although numerous ear and face recognition techniques have been proposed, not much work has been conducted in the field of 3-D fiducial points localization and 3-D ear detection. This paper presents an effective and efficient system of ear landmark localization, ear detection, and pose classification based on 3-D ears captured under large yaw variations. By utilizing the symmetrical property of human heads and classifying the ear with respect to its pose, all three tasks can be fulfilled given either left or right ears, without any prior pose information. A novel ear tree-structured graph (ETG) is proposed to represent the 3-D ear, after which a 3-D flexible mixture model is trained to locate the landmarks automatically. Then, the ear region is segmented based on them and the pose of the ear, i.e., whether it is a left or right ear, is classified based on the detected ETG. To the best of our knowledge, this paper is the first to present automatic landmark localization of 3-D ears extracted from facial scans with significant pose variations. Experiments were conducted at the University of Notre Dame collection F, G and J2, which contain large occlusion and pose variations, validating the effectiveness of the proposed methods.

Multi-task pose-invariant face recognition.

Face images captured in unconstrained environments usually contain significant pose variation, which dramatically degrades the performance of algorithms designed to recognize frontal faces. This paper proposes a novel face identification framework capable of handling the full range of pose variations within ±90° of yaw. The proposed framework first transforms the original pose-invariant face recognition problem into a partial frontal face recognition problem. A robust patch-based face representation scheme is then developed to represent the synthesized partial frontal faces. For each patch, a transformation dictionary is learnt under the proposed multi-task learning scheme. The transformation dictionary transforms the features of different poses into a discriminative subspace. Finally, face matching is performed at patch level rather than at the holistic level. Extensive and systematic experimentation on FERET, CMU-PIE, and Multi-PIE databases shows that the proposed method consistently outperforms single-task-based baselines as well as state-of-the-art methods for the pose problem. We further extend the proposed algorithm for the unconstrained face verification problem and achieve top-level performance on the challenging LFW data set.

Semantic Pyramids for Gender and Action Recognition

Person description is a challenging problem in computer vision. We investigated two major aspects of person description: 1) gender and 2) action recognition in still images. Most state-of-the-art approaches for gender and action recognition rely on the description of a single body part, such as face or full-body. However, relying on a single body part is suboptimal due to significant variations in scale, viewpoint, and pose in real-world images. This paper proposes a semantic pyramid approach for pose normalization. Our approach is fully automatic and based on combining information from full-body, upper-body, and face regions for gender and action recognition in still images. The proposed approach does not require any annotations for upper-body and face of a person. Instead, we rely on pretrained state-of-the-art upper-body and face detectors to automatically extract semantic information of a person. Given multiple bounding boxes from each body part detector, we then propose a simple method to select the best candidate bounding box, which is used for feature extraction. Finally, the extracted features from the full-body, upper-body, and face regions are combined into a single representation for classification. To validate the proposed approach for gender recognition, experiments are performed on three large data sets namely: 1) human attribute; 2) head-shoulder; and 3) proxemics. For action recognition, we perform experiments on four data sets most used for benchmarking action recognition in still images: 1) Sports; 2) Willow; 3) PASCAL VOC 2010; and 4) Stanford-40. Our experiments clearly demonstrate that the proposed approach, despite its simplicity, outperforms state-of-the-art methods for gender and action recognition.

Tracking and recognition face in videos with incremental local sparse representation model

This paper addresses the problem of tracking and recognizing faces via incremental local sparse representation. First a robust face tracking algorithm is proposed via employing local sparse appearance and covariance pooling method. In the following face recognition stage, with the employment of a novel template update strategy, which combines incremental subspace learning, our recognition algorithm adapts the template to appearance changes and reduces the influence of occlusion and illumination variation. This leads to a robust video-based face tracking and recognition with desirable performance. In the experiments, we test the quality of face recognition in real-world noisy videos on YouTube database, which includes 47 celebrities. Our proposed method produces a high face recognition rate at 95% of all videos. The proposed face tracking and recognition algorithms are also tested on a set of noisy videos under heavy occlusion and illumination variation. The tracking results on challenging benchmark videos demonstrate that the proposed tracking algorithm performs favorably against several state-of-the-art methods. In the case of the challenging dataset in which faces undergo occlusion and illumination variation, and tracking and recognition experiments under significant pose variation on the University of California, San Diego (Honda/UCSD) database, our proposed method also consistently demonstrates a high recognition rate.

Acquiring 3D indoor environments with variability and repetition

Large-scale acquisition of exterior urban environments is by now a well-established technology, supporting many applications in search, navigation, and commerce. The same is, however, not the case for indoor environments, where access is often restricted and the spaces are cluttered. Further, such environments typically contain a high density of repeated objects (e.g., tables, chairs, monitors, etc.) in regular or non-regular arrangements with significant pose variations and articulations. In this paper, we exploit the special structure of indoor environments to accelerate their 3D acquisition and recognition with a low-end handheld scanner. Our approach runs in two phases: (i) a learning phase wherein we acquire 3D models of frequently occurring objects and capture their variability modes from only a few scans, and (ii) a recognition phase wherein from a single scan of a new area, we identify previously seen objects but in different poses and locations at an average recognition time of 200ms/model. We evaluate the robustness and limits of the proposed recognition system using a range of synthetic and real world scans under challenging settings.

Contactless and Pose Invariant Biometric Identification Using Hand Surface

This paper presents a novel approach for hand matching that achieves significantly improved performance even in the presence of large hand pose variations. The proposed method utilizes a 3-D digitizer to simultaneously acquire intensity and range images of the user's hand presented to the system in an arbitrary pose. The approach involves determination of the orientation of the hand in 3-D space followed by pose normalization of the acquired 3-D and 2-D hand images. Multimodal (2-D as well as 3-D) palmprint and hand geometry features, which are simultaneously extracted from the user's pose normalized textured 3-D hand, are used for matching. Individual matching scores are then combined using a new dynamic fusion strategy. Our experimental results on the database of 114 subjects with significant pose variations yielded encouraging results. Consistent (across various hand features considered) performance improvement achieved with the pose correction demonstrates the usefulness of the proposed approach for hand based biometric systems with unconstrained and contact-free imaging. The experimental results also suggest that the dynamic fusion approach employed in this work helps to achieve performance improvement of 60% (in terms of EER) over the case when matching scores are combined using the weighted sum rule.

Using Facial Symmetry to Handle Pose Variations in Real-World 3D Face Recognition

The uncontrolled conditions of real-world biometric applications pose a great challenge to any face recognition approach. The unconstrained acquisition of data from uncooperative subjects may result in facial scans with significant pose variations along the yaw axis. Such pose variations can cause extensive occlusions, resulting in missing data. In this paper, a novel 3D face recognition method is proposed that uses facial symmetry to handle pose variations. It employs an automatic landmark detector that estimates pose and detects occluded areas for each facial scan. Subsequently, an Annotated Face Model is registered and fitted to the scan. During fitting, facial symmetry is used to overcome the challenges of missing data. The result is a pose invariant geometry image. Unlike existing methods that require frontal scans, the proposed method performs comparisons among interpose scans using a wavelet-based biometric signature. It is suitable for real-world applications as it only requires half of the face to be visible to the sensor. The proposed method was evaluated using databases from the University of Notre Dame and the University of Houston that, to the best of our knowledge, include the most challenging pose variations publicly available. The average rank-one recognition rate of the proposed method in these databases was 83.7 percent.

Thermal and reflectance based personal identification methodology under variable illumination

The objective of this work is to authenticate individuals based on the appearance of their faces. This is a difficult pattern recognition problem because facial appearance is generally greatly affected by the changes in the way a face is illuminated, by the camera viewpoint and partial occlusions, for example due to eye-wear. We describe a fully automatic algorithm that systematically addresses each of these challenges. The main novelty is an algorithm for decision-level fusion of two types of imagery: one acquired in the visual and one acquired in infrared electromagnetic spectrum. Specifically: we examine: (i) the effects of preprocessing of data in each domain, (ii) the fusion of holistic and local facial appearance, and (iii) propose an algorithm for combining the similarity scores in visual and thermal spectra in the presence of prescription glasses and significant pose variations, using a small number of training images (5-7). Our system achieved a high correct identification rate of 97% on a freely available data set containing extreme illumination changes.