Viewpoint Variations Research Articles

Locally Nonlinear Affine Verification for Multisensor Image Matching

Matching local features between two overlapped images is a fundamental task in photogrammetry and remote sensing. However, images acquired by multiple sensors often differ substantially in properties, thus posing a great challenge to the robustness and flexibility of feature matching methods. In this article, we propose a locally non-linear affine verification (LAV) method for robust multisensor image matching. The main idea of the LAV is the development of a nonlinear regression formulation that practically models the nonlinear deviation of a real surface around a point from its tangent plane during affine verification. Specifically, we start by selecting a restricted set of reliable and well-distributed putative matches as the matching seeds and assign them with neighbors to construct search spaces. In each search space, the regression seeks the smoothest affine model consistent with the latent correct matches, thereby deriving a set of affine parameters to verify correspondence hypotheses for true matches. The verification can be extended to all nearest neighbor matches to discover additional inlier matches. Evaluation on multisensor image datasets with different extents of variations in viewpoint, scale, illumination, and appearance shows that the proposed LAV consistently outperforms existing methods. LAV can achieve a considerable number of high-quality matches, in cases where existing methods provide few or no correct matches.

A brief survey for person re-identification based on deep learning

Person Re-Identification (Re-ID) has been paid more attention due to its wide application in intelligent surveillance systems. Finding the same person from other non-overlapping cameras when a specific image of the pedestrian is given, which is a challenging problem for the reason of viewpoint variation, clothes changing, low resolution, etc. In this paper, we motivate reviewing for deep learning-based methods of Person Re-ID. We present a detailed survey of the state-of-the-art in terms of the description and analysis of supervised-based and unsupervised-based networks and their performance evaluation in the commonly used data sets. Finally, we analyse the challenging problems and discuss future works in this area.

Comparative Study of Markerless Vision-Based Gait Analyses for Person Re-Identification

The model-based gait analysis of kinematic characteristics of the human body has been used to identify individuals. To extract gait features, spatiotemporal changes of anatomical landmarks of the human body in 3D were preferable. Without special lab settings, 2D images were easily acquired by monocular video cameras in real-world settings. The 2D and 3D locations of key joint positions were estimated by the 2D and 3D pose estimators. Then, the 3D joint positions can be estimated from the 2D image sequences in human gait. Yet, it has been challenging to have the exact gait features of a person due to viewpoint variance and occlusion of body parts in the 2D images. In the study, we conducted a comparative study of two different approaches: feature-based and spatiotemporal-based viewpoint invariant person re-identification using gait patterns. The first method is to use gait features extracted from time-series 3D joint positions to identify an individual. The second method uses a neural network, a Siamese Long Short Term Memory (LSTM) network with the 3D spatiotemporal changes of key joint positions in a gait cycle to classify an individual without extracting gait features. To validate and compare these two methods, we conducted experiments with two open datasets of the MARS and CASIA-A datasets. The results show that the Siamese LSTM outperforms the gait feature-based approaches on the MARS dataset by 20% and 55% on the CASIA-A dataset. The results show that feature-based gait analysis using 2D and 3D pose estimators is premature. As a future study, we suggest developing large-scale human gait datasets and designing accurate 2D and 3D joint position estimators specifically for gait patterns. We expect that the current comparative study and the future work could contribute to rehabilitation study, forensic gait analysis and early detection of neurological disorders.

Sequences consistency feature learning for video‐based person re‐identification

Video-based person re-identification aims to match pedestrians from video sequences across non-overlapping cameras. A major challenge of the person re-identification is the serious intra-class distance caused by cropped frames variation in video sequences. To address this issue, a novel sequences consistency feature learning (SCFL) framework for video-based person re-identification is proposed. Specifically, SCFL utilizes a deep neural network and the proposed sequences consistency loss to learn sequences-invariant features for each pedestrian, which decreases the intra-class distance across the partial occlusions, inaccurate detection and viewpoint variation. During the training process, SCFL slices the video-level features and computes the cosine similarity between disjoint sequences features pairs of the same pedestrian as the sequences consistency loss to minimize the intra-class distance. The experiments demonstrate that our approach improves the performance of the existing deep networks and achieves competitive performance on the large-scale benchmark datasets including MARS and DukeMTMC-VideoReID.

Validating a model of architectural hazard visibility with low-vision observers.

Pedestrians with low vision are at risk of injury when hazards, such as steps and posts, have low visibility. This study aims at validating the software implementation of a computational model that estimates hazard visibility. The model takes as input a photorealistic 3D rendering of an architectural space, and the acuity and contrast sensitivity of a low-vision observer, and outputs estimates of the visibility of hazards in the space. Our experiments explored whether the model could predict the likelihood of observers correctly identifying hazards. In Experiment 1, we tested fourteen normally sighted subjects with blur goggles that simulated moderate or severe acuity reduction. In Experiment 2, we tested ten low-vision subjects with moderate to severe acuity reduction. Subjects viewed computer-generated images of a walkway containing five possible targets ahead-big step-up, big step-down, small step-up, small step-down, or a flat continuation. Each subject saw these stimuli with variations of lighting and viewpoint in 250 trials and indicated which of the five targets was present. The model generated a score on each trial that estimated the visibility of the target. If the model is valid, the scores should be predictive of how accurately the subjects identified the targets. We used logistic regression to examine the correlation between the scores and the participants' responses. For twelve of the fourteen normally sighted subjects with artificial acuity reduction and all ten low-vision subjects, there was a significant relationship between the scores and the participant's probability of correct identification. These experiments provide evidence for the validity of a computational model that predicts the visibility of architectural hazards. It lays the foundation for future validation of this hazard evaluation tool, which may be useful for architects to assess the visibility of hazards in their designs, thereby enhancing the accessibility of spaces for people with low vision.

Social Context-aware Person Search in Videos via Multi-modal Cues

Person search has long been treated as a crucial and challenging task to support deeper insight in personalized summarization and personality discovery. Traditional methods, e.g., person re-identification and face recognition techniques, which profile video characters based on visual information, are often limited by relatively fixed poses or small variation of viewpoints and suffer from more realistic scenes with high motion complexity (e.g., movies). At the same time, long videos such as movies often have logical story lines and are composed of continuously developmental plots. In this situation, different persons usually meet on a specific occasion, in which informative social cues are performed. We notice that these social cues could semantically profile their personality and benefit person search task in two aspects. First, persons with certain relationships usually co-occur in short intervals; in case one of them is easier to be identified, the social relation cues extracted from their co-occurrences could further benefit the identification for the harder ones. Second, social relations could reveal the association between certain scenes and characters (e.g., classmate relationship may only exist among students), which could narrow down candidates into certain persons with a specific relationship. In this way, high-level social relation cues could improve the effectiveness of person search. Along this line, in this article, we propose a social context-aware framework, which fuses visual and social contexts to profile persons in more semantic perspectives and better deal with person search task in complex scenarios. Specifically, we first segment videos into several independent scene units and abstract out social contexts within these scene units. Then, we construct inner-personal links through a graph formulation operation for each scene unit, in which both visual cues and relation cues are considered. Finally, we perform a relation-aware label propagation to identify characters’ occurrences, combining low-level semantic cues (i.e., visual cues) and high-level semantic cues (i.e., relation cues) to further enhance the accuracy. Experiments on real-world datasets validate that our solution outperforms several competitive baselines.

Refined Color Texture Classification Using CNN and Local Binary Pattern

Representation and classification of color texture generate considerable interest within the field of computer vision. Texture classification is a difficult task that assigns unlabeled images or textures to the correct labeled class. Some key factors such as scaling and viewpoint variations and illumination changes make this task challenging. In this paper, we present a new feature extraction technique for color texture classification and recognition. The presented approach aggregates the features extracted from local binary patterns (LBP) and convolution neural network (CNN) to provide discriminatory information, leading to better texture classification results. Almost all of the CNN model cases classify images based on global features that describe the image as a whole to generalize the entire object. LBP classifies images based on local features that describe the image’s key points (image patches). Our analysis shows that using LBP improves the classification task when compared to using CNN only. We test the proposed approach experimentally over three challenging color image datasets (ALOT, CBT, and Outex). The results demonstrated that our approach improved up to 25% in the classification accuracy over the traditional CNN models. We identify optimal combinations for each dataset and obtain high classification rates. The proposed approach is robust, stable, and discriminatory among the three datasets and has shown enhancement in classification and recognition compared to the state-of-the-art method.

Re-Identification in Urban Scenarios: A Review of Tools and Methods

With the widespread use of surveillance image cameras and enhanced awareness of public security, objects, and persons Re-Identification (ReID), the task of recognizing objects in non-overlapping camera networks has attracted particular attention in computer vision and pattern recognition communities. Given an image or video of an object-of-interest (query), object identification aims to identify the object from images or video feed taken from different cameras. After many years of great effort, object ReID remains a notably challenging task. The main reason is that an object’s appearance may dramatically change across camera views due to significant variations in illumination, poses or viewpoints, or even cluttered backgrounds. With the advent of Deep Neural Networks (DNN), there have been many proposals for different network architectures achieving high-performance levels. With the aim of identifying the most promising methods for ReID for future robust implementations, a review study is presented, mainly focusing on the person and multi-object ReID and auxiliary methods for image enhancement. Such methods are crucial for robust object ReID, while highlighting limitations of the identified methods. This is a very active field, evidenced by the dates of the publications found. However, most works use data from very different datasets and genres, which presents an obstacle to wide generalized DNN model training and usage. Although the model’s performance has achieved satisfactory results on particular datasets, a particular trend was observed in the use of 3D Convolutional Neural Networks (CNN), attention mechanisms to capture object-relevant features, and generative adversarial training to overcome data limitations. However, there is still room for improvement, namely in using images from urban scenarios among anonymized images to comply with public privacy legislation. The main challenges that remain in the ReID field, and prospects for future research directions towards ReID in dense urban scenarios, are also discussed.

Intelligent recognition system for viewpoint variations on gait and speech using CNN-CapsNet

PurposeThe paper aims to introduce an intelligent recognition system for viewpoint variations of gait and speech. It proposes a convolutional neural network-based capsule network (CNN-CapsNet) model and outlining the performance of the system in recognition of gait and speech variations. The proposed intelligent system mainly focuses on relative spatial hierarchies between gait features in the entities of the image due to translational invariances in sub-sampling and speech variations.Design/methodology/approachThis proposed work CNN-CapsNet is mainly used for automatic learning of feature representations based on CNN and used capsule vectors as neurons to encode all the spatial information of an image by adapting equal variances to change in viewpoint. The proposed study will resolve the discrepancies caused by cofactors and gait recognition between opinions based on a model of CNN-CapsNet.FindingsThis research work provides recognition of signal, biometric-based gait recognition and sound/speech analysis. Empirical evaluations are conducted on three aspects of scenarios, namely fixed-view, cross-view and multi-view conditions. The main parameters for recognition of gait are speed, change in clothes, subjects walking with carrying object and intensity of light.Research limitations/implicationsThe proposed CNN-CapsNet has some limitations when considering for detecting the walking targets from surveillance videos considering multimodal fusion approaches using hardware sensor devices. It can also act as a pre-requisite tool to analyze, identify, detect and verify the malware practices.Practical implicationsThis research work includes for detecting the walking targets from surveillance videos considering multimodal fusion approaches using hardware sensor devices. It can also act as a pre-requisite tool to analyze, identify, detect and verify the malware practices.Originality/valueThis proposed research work proves to be performing better for the recognition of gait and speech when compared with other techniques.

Semantic Histogram Based Graph Matching for Real-Time Multi-Robot Global Localization in Large Scale Environment

The core problem of visual multi-robot simultaneous localization and mapping (MR-SLAM) is how to efficiently and accurately perform multi-robot global localization (MR-GL). The difficulties are two-fold. The first is the difficulty of global localization for significant viewpoint difference. Appearance-based localization methods tend to fail under large viewpoint changes. Recently, semantic graphs have been utilized to overcome the viewpoint variation problem. However, the methods are highly time-consuming, especially in large-scale environments. This leads to the second difficulty, which is how to perform real-time global localization. In this paper, we propose a semantic histogram-based graph matching method that is robust to viewpoint variation and can achieve real-time global localization. Based on that, we develop a system that can accurately and efficiently perform MR-GL for both homogeneous and heterogeneous robots. The experimental results show that our approach is about 30 times faster than Random Walk based semantic descriptors. Moreover, it achieves an accuracy of 95% for global localization, while the accuracy of the state-of-the-art method is 85%.

Mask-guided contrastive attention and two-stream metric co-learning for person Re-identification

Person Re-identification (ReID) is an important yet challenging task in computer vision. Due to diverse background clutters, variations of viewpoints and body poses, it is far from being solved. How to extract discriminative and robust features invariant to background clutters is one of the core problems. In this paper, we first introduce a set of binary segmentation masks to construct synthetic RGB-Mask pairs as inputs, and then design a mask-guided contrastive attention model (MGCAM) to learn features separately from the body and background regions. Moreover, we propose a novel region-level triplet loss to guide the features learning, i.e., pulling the features from the full image and body region close, whereas pushing the features from backgrounds away. To learn the similarities from multiple features of the proposed MGCAM, we further introduce the instance-level two-stream metric co-learning (TSMCL) to help learn pair-wise relations between features from not only different regions but also different instances. TSMCL could help learn more compact features across full and body streams, enhancing the performance of MGCAM. We evaluate the proposed method on four public datasets, including MARS, Market-1501, CUHK03, and DukeMTMC-reID. Extensive experiments show that the proposed method is effective and achieves satisfying results.

Single Depth View Based Real-Time Reconstruction of Hand-Object Interactions

Reconstructing hand-object interactions is a challenging task due to strong occlusions and complex motions. This article proposes a real-time system that uses a single depth stream to simultaneously reconstruct hand poses, object shape, and rigid/non-rigid motions. To achieve this, we first train a joint learning network to segment the hand and object in a depth image, and to predict the 3D keypoints of the hand. With most layers shared by the two tasks, computation cost is saved for the real-time performance. A hybrid dataset is constructed here to train the network with real data (to learn real-world distributions) and synthetic data (to cover variations of objects, motions, and viewpoints). Next, the depth of the two targets and the keypoints are used in a uniform optimization to reconstruct the interacting motions. Benefitting from a novel tangential contact constraint, the system not only solves the remaining ambiguities but also keeps the real-time performance. Experiments show that our system handles different hand and object shapes, various interactive motions, and moving cameras.

Separated and overlapping neural coding of face and body identity.

Recognising a person's identity often relies on face and body information, and is tolerant to changes in low‐level visual input (e.g., viewpoint changes). Previous studies have suggested that face identity is disentangled from low‐level visual input in the anterior face‐responsive regions. It remains unclear which regions disentangle body identity from variations in viewpoint, and whether face and body identity are encoded separately or combined into a coherent person identity representation. We trained participants to recognise three identities, and then recorded their brain activity using fMRI while they viewed face and body images of these three identities from different viewpoints. Participants' task was to respond to either the stimulus identity or viewpoint. We found consistent decoding of body identity across viewpoint in the fusiform body area, right anterior temporal cortex, middle frontal gyrus and right insula. This finding demonstrates a similar function of fusiform and anterior temporal cortex for bodies as has previously been shown for faces, suggesting these regions may play a general role in extracting high‐level identity information. Moreover, we could decode identity across fMRI activity evoked by faces and bodies in the early visual cortex, right inferior occipital cortex, right parahippocampal cortex and right superior parietal cortex, revealing a distributed network that encodes person identity abstractly. Lastly, identity decoding was consistently better when participants attended to identity, indicating that attention to identity enhances its neural representation. These results offer new insights into how the brain develops an abstract neural coding of person identity, shared by faces and bodies.

Lack of standardisation in interpretation and reporting of autoantibody assays: a survey analysis of Australasian laboratories with focus on line immunoassays.

Autoantibody assays are reported in a variety of formats. Results only slightly above established cut-offs provide lower likelihood ratios; therefore, their clinical significance may be more uncertain, which is not readily communicated with dichotomous qualitative reporting. Line immunoassays (LIA) are a common method for detecting antibodies to extractable nuclear antigens (ENA) and myositis-associated antibodies. However, recommended positive cut-offs are contentious. We distributed a survey via e-mail to participants in the Royal College of Pathologists of Australasia Quality Assurance Program (RCPAQAP) Immunology modules and to a dedicated immunology mailing list in Australasia. Questions explored general viewpoints surrounding autoantibody reporting, as well as current laboratory practices, with particular focus on interpretation and reporting of the most commonly used ENA LIA manufactured by Euroimmun. There were 31 responders, representative of at least 17 unique laboratories across Australia (8 public, 5 private) and New Zealand (4 laboratories). Responses suggest that autoantibody reporting is not standardised; there was variation in general viewpoints and reporting practices, particularly regarding the interpretation of and positive cut-offs used for the Euroimmun ENA LIA, which were contrary to the manufacturer's guidelines in a majority of the responses. Interpretative qualitative reporting based on results from other investigations and the clinical history was a common theme. There is large variation in the reporting of autoantibody assays within Australasia, especially by LIA. A majority of respondents report the most commonly used ENA LIA contrary to manufacturer's guidelines; alternative positive cut-offs are commonly utilised. LIA reports should indicate the level of positivity to enhance their relevance in the clinical decision-making process.

VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change

Visual place recognition (VPR) is the process of recognising a previously visited place using visual information, often under varying appearance conditions and viewpoint changes and with computational constraints. VPR is related to the concepts of localisation, loop closure, image retrieval and is a critical component of many autonomous navigation systems ranging from autonomous vehicles to drones and computer vision systems. While the concept of place recognition has been around for many years, VPR research has grown rapidly as a field over the past decade due to improving camera hardware and its potential for deep learning-based techniques, and has become a widely studied topic in both the computer vision and robotics communities. This growth however has led to fragmentation and a lack of standardisation in the field, especially concerning performance evaluation. Moreover, the notion of viewpoint and illumination invariance of VPR techniques has largely been assessed qualitatively and hence ambiguously in the past. In this paper, we address these gaps through a new comprehensive open-source framework for assessing the performance of VPR techniques, dubbed “VPR-Bench”. VPR-Bench (Open-sourced at: https://github.com/MubarizZaffar/VPR-Bench) introduces two much-needed capabilities for VPR researchers: firstly, it contains a benchmark of 12 fully-integrated datasets and 10 VPR techniques, and secondly, it integrates a comprehensive variation-quantified dataset for quantifying viewpoint and illumination invariance. We apply and analyse popular evaluation metrics for VPR from both the computer vision and robotics communities, and discuss how these different metrics complement and/or replace each other, depending upon the underlying applications and system requirements. Our analysis reveals that no universal SOTA VPR technique exists, since: (a) state-of-the-art (SOTA) performance is achieved by 8 out of the 10 techniques on at least one dataset, (b) SOTA technique in one community does not necessarily yield SOTA performance in the other given the differences in datasets and metrics. Furthermore, we identify key open challenges since: (c) all 10 techniques suffer greatly in perceptually-aliased and less-structured environments, (d) all techniques suffer from viewpoint variance where lateral change has less effect than 3D change, and (e) directional illumination change has more adverse effects on matching confidence than uniform illumination change. We also present detailed meta-analyses regarding the roles of varying ground-truths, platforms, application requirements and technique parameters. Finally, VPR-Bench provides a unified implementation to deploy these VPR techniques, metrics and datasets, and is extensible through templates.

Cross-view geo-localization via Salient Feature Partition Network

Cross-view geo-localization aims to find images containing the same geographic target from images obtained from different platforms. The extreme viewpoint variations bring challenges to this task. Existing methods usually focus on mining the fine-grained features of geographic targets in images, ignoring the potential contextual information around them. In this paper, we consider that the background regions can be used as auxiliary information, which can make the image representation for geo-localization more discriminative. Specifically, we designed a classification network that divides regional features based on saliency, called Salient Feature Partition Network (SFPN), which utilizes background information in an end-to-end manner. Without using additional part estimators, SFPN divides the features into foreground and background based on saliency. It simplifies part matching and realizes the region division learning. The method proposed in this paper has achieved competitive results on the university 1652 dataset.

Robust Video-Based Person Re-Identification by Hierarchical Mining

Video-based person re-identification (Re-ID) aims at retrieving the person through the video sequences across non-overlapping cameras. Some characteristics of pedestrians are not consecutive across frames due to the variations of viewpoints, postures, and occlusions over time. However, existing methods ignore such data peculiarity and the networks tend to only learn those salient consecutive characteristics among frames in video sequences. As a result, the learned representations fail to cover all the characteristics of pedestrians, thus lacking integrity and discrimination. To tackle this problem, we present a novel deep architecture termed Hierarchical Mining Network (HMN), which mines as many pedestrians’ characteristics by referring to the temporal and intra-class knowledge. It consists of a novel Attentive Temporal Module (ATM) and a Dynamic Supervising Branch (DSB), with a Balancing Triplet Loss (BTL) assisting the training. The proposed ATM, with pedestrian perceiving capacity, is capable of evaluating each activation of features through temporal analysis, so that the temporally scattered characteristics of pedestrians can be better aggregated and the contaminated ones can be eliminated. Then, the DSB along with the BTL further enhances the integrity of representations by multiple supervision. Specifically, the DSB perceives the diversities of intra-class samples in each mini-batch and generates targeted supervising signals for them, in which process the BTL guarantees the signals with smaller intra-class variations and larger inter-class variations. Comprehensive experiments on two video-based datasets, i.e., MARS, and DukeMTMC-VideoReID, demonstrate the contribution of each component and the superiority of the proposed HMN over the state-of-the-arts. Benchmarking our model on three popular image-based datasets, i.e., Market1501, DukeMTMC-Reid, and MSMT17 additionally verifies the promising generalizability of the proposed DSB and BTL.

MS-Faster R-CNN: Multi-Stream Backbone for Improved Faster R-CNN Object Detection and Aerial Tracking from UAV Images

Tracking objects across multiple video frames is a challenging task due to several difficult issues such as occlusions, background clutter, lighting as well as object and camera view-point variations, which directly affect the object detection. These aspects are even more emphasized when analyzing unmanned aerial vehicles (UAV) based images, where the vehicle movement can also impact the image quality. A common strategy employed to address these issues is to analyze the input images at different scales to obtain as much information as possible to correctly detect and track the objects across video sequences. Following this rationale, in this paper, we introduce a simple yet effective novel multi-stream (MS) architecture, where different kernel sizes are applied to each stream to simulate a multi-scale image analysis. The proposed architecture is then used as backbone for the well-known Faster-R-CNN pipeline, defining a MS-Faster R-CNN object detector that consistently detects objects in video sequences. Subsequently, this detector is jointly used with the Simple Online and Real-time Tracking with a Deep Association Metric (Deep SORT) algorithm to achieve real-time tracking capabilities on UAV images. To assess the presented architecture, extensive experiments were performed on the UMCD, UAVDT, UAV20L, and UAV123 datasets. The presented pipeline achieved state-of-the-art performance, confirming that the proposed multi-stream method can correctly emulate the robust multi-scale image analysis paradigm.

Segment attention‐guided part‐aligned network for person re‐identification

Part misalignment of the human body caused by complex variations in viewpoint and pose poses a fundamental challenge to person re-identification. This letter examines Res2Net as the backbone network to extract multi-scale appearance features. At the same time, it uses the human parsing model to extract part features, which can be used as an attention stream to guide part features re-calibration from the spatial dimension. Additionally, in order to ensure the diversity of features, SAG-PAN effectively integrates the global appearance features of person image with part fine-grained features. The experimental results on the Market-1501, DukeMTMC-reID and CUHK03 datasets show that the proposed SAG-PAN achieved superior performance against the existing state-of-the-art methods.

Multilabel CNN-Based Hybrid Learning Metric for Pedestrian Reidentification

Pedestrian reidentification has recently emerged as a hot topic that attains considerable attention since it can be applied to many potential applications in the surveillance system. However, high-accuracy pedestrian reidentification is a stimulating research problem because of variations in viewpoints, color, light, and other reasons. This work addresses the interferences and improves pedestrian reidentification accuracy by proposing two novel algorithms, pedestrian multilabel learning, and investigating hybrid learning metrics. First, unlike the existing models, we construct the identification framework using two subnetworks, namely, part detection subnetwork and feature extraction subnetwork, to obtain pedestrian attributes and low-level feature scores, respectively. Then, a hybrid learning metric that combines pedestrian attributes and low-level feature scores is proposed. Both low-level features and pedestrian attributes are utilized, thus enhancing the identification rate. Our simulation results on both datasets, i.e., CUHK03 and VIPeR, reveal that the identification rate is improved compared to the existing pedestrian reidentification methods.