Personal Retrieval Research Articles

A survey on person and vehicle re‐identification

AbstractPerson/vehicle re‐identification aims to use technologies such as cross‐camera retrieval to associate the same person (same vehicle) in the surveillance videos at different locations, different times, and images captured by different cameras so as to achieve cross‐surveillance image matching, person retrieval and trajectory tracking. It plays an extremely important role in the fields of intelligent security, criminal investigation etc. In recent years, the rapid development of deep learning technology has significantly propelled the advancement of re‐identification (Re‐ID) technology. An increasing number of technical methods have emerged, aiming to enhance Re‐ID performance. This paper summarises four popular research areas in the current field of re‐identification, focusing on the current research hotspots. These areas include the multi‐task learning domain, the generalisation learning domain, the cross‐modality domain, and the optimisation learning domain. Specifically, the paper analyses various challenges faced within these domains and elaborates on different deep learning frameworks and networks that address these challenges. A comparative analysis of re‐identification tasks from various classification perspectives is provided, introducing mainstream research directions and current achievements. Finally, insights into future development trends are presented.

Data compensation and feature fusion for sketch based person retrieval

Sketch re-identification (Re-ID) aims to retrieve pedestrian photo in the gallery dataset by a query sketch drawn by professionals. The sketch Re-ID task has not been adequately studied because collecting such sketches is difficult and expensive. In addition, the significant modality difference between sketches and images makes extracting the discriminative feature information difficult. To address above issues, we introduce a novel sketch-style pedestrian dataset named Pseudo-Sketch dataset. Our proposed dataset maximizes the utilization of the existing person dataset resources and is freely available, thus effectively reducing the expenses associated with the training and deployment phases. Furthermore, to mitigate the modality gap between sketches and visible images, a cross-modal feature fusion network is proposed that incorporates information from each modality. Experiment results show that the proposed Pseudo-Sketch dataset can effectively complement the real sketch dataset, and the proposed network obtains competitive results than SOTA methods. The dataset will be released later.

Cross-modal semantic aligning and neighbor-aware completing for robust text–image person retrieval

Most existing text–image person re-identification (TIReID) methods are performed in an ideal environment where both image and text instances are fully intact and identity annotated. However, in real-world open environments, these ideal assumptions often cannot be satisfied. In this study, we are the first to explore how to enhance the robustness of the TIReID model to better adapt to open environments, focusing on two aspects: 1) unlabeled multi-modal data, and 2) incomplete multi-modal data. Here, unlabeled multi-modal data faces two key issues: intra-class variations and cross-modal aligning problems, while incomplete data leads to data semantic inconsistency problems and severe performance degradation. In order to tackle the above issues, we propose a novel Cross-modal Semantic Aligning and Neighbor-aware Completing (CANC) method for robust text–image person re-identification (RTIReID). Specifically, to tackle the intra-class variation problem, we propose intra-view prototype contrastive matching for image and text modalities, which helps the model learn intra-modal discriminative representations while increasing inter-class distance and suppressing intra-class variation. Furthermore, to resolve the cross-modal alignment problem, we propose the cross-view instance projection matching to establish the relationship between different modal representations. Additionally, regarding how to effectively handle incomplete samples, we introduce nearest neighbor consistent completion, which can restore high-quality completion features. We adopt CLIP-ViT and CLIP-Xformer as image and text encoders respectively, and Rank-k metrics (k=1,5,10) as primary evaluation metrics. Extensive experiments on several public datasets demonstrate that our method can effectively address TIReID in open environments, consistently surpassing several SOTA TIReID methods.

BCRA: bidirectional cross-modal implicit relation reasoning and aligning for text-to-image person retrieval

BCRA: bidirectional cross-modal implicit relation reasoning and aligning for text-to-image person retrieval

Causality-Inspired Invariant Representation Learning for Text-Based Person Retrieval

Text-based Person Retrieval (TPR) aims to retrieve relevant images of specific pedestrians based on the given textual query. The mainstream approaches primarily leverage pretrained deep neural networks to learn the mapping of visual and textual modalities into a common latent space for cross-modality matching. Despite their remarkable achievements, existing efforts mainly focus on learning the statistical cross-modality correlation found in training data, other than the intrinsic causal correlation. As a result, they often struggle to retrieve accurately in the face of environmental changes such as illumination, pose, and occlusion, or when encountering images with similar attributes. In this regard, we pioneer the observation of TPR from a causal view. Specifically, we assume that each image is composed of a mixture of causal factors (which are semantically consistent with text descriptions) and non-causal factors (retrieval-irrelevant, e.g., background), and only the former can lead to reliable retrieval judgments. Our goal is to extract text-critical robust visual representation (i.e., causal factors) and establish domain invariant cross-modality correlations for accurate and reliable retrieval. However, causal/non-causal factors are unobserved, so we emphasize that ideal causal factors that can simulate causal scenes should satisfy two basic principles:1） Independence: being independent of non-causal factors, and 2）Sufficiency: being causally sufficient for TPR across different environments. Building on that, we propose an Invariant Representation Learning method for TPR (IRLT), that enforces the visual representations to satisfy the two aforementioned critical properties. Extensive experiments on three datasets clearly demonstrate the advantages of IRLT over leading baselines in terms of accuracy and generalization.

Adaptive Uncertainty-Based Learning for Text-Based Person Retrieval

Text-based person retrieval aims at retrieving a specific pedestrian image from a gallery based on textual descriptions. The primary challenge is how to overcome the inherent heterogeneous modality gap in the situation of significant intra-class variation and minimal inter-class variation. Existing approaches commonly employ vision-language pre-training or attention mechanisms to learn appropriate cross-modal alignments from noise inputs. Despite commendable progress, current methods inevitably suffer from two defects: 1) Matching ambiguity, which mainly derives from unreliable matching pairs; 2) One-sided cross-modal alignments, stemming from the absence of exploring one-to-many correspondence, i.e., coarse-grained semantic alignment. These critical issues significantly deteriorate retrieval performance. To this end, we propose a novel framework termed Adaptive Uncertainty-based Learning (AUL) for text-based person retrieval from the uncertainty perspective. Specifically, our AUL framework consists of three key components: 1) Uncertainty-aware Matching Filtration that leverages Subjective Logic to effectively mitigate the disturbance of unreliable matching pairs and select high-confidence cross-modal matches for training; 2) Uncertainty-based Alignment Refinement, which not only simulates coarse-grained alignments by constructing uncertainty representations but also performs progressive learning to incorporate coarse- and fine-grained alignments properly; 3) Cross-modal Masked Modeling that aims at exploring more comprehensive relations between vision and language. Extensive experiments demonstrate that our AUL method consistently achieves state-of-the-art performance on three benchmark datasets in supervised, weakly supervised, and domain generalization settings. Our code is available at https://github.com/CFM-MSG/Code-AUL.

EESSO: Exploiting Extreme and Smooth Signals via Omni-frequency learning for Text-based Person Retrieval

Most of the existing methods manage to tackle the problem of text-based person retrieval from the spatial-wise perspective. In this paper, we manage to address the problem of this task from a novel perspective, namely, the frequency-wise perspective. To this end, we propose to Exploit Extreme and Smooth Signals via Omni-frequency learning (EESSO) through a jointly optimized multi-stream architecture. It consists of a Spatial Information Stream (SIS), an Extreme Signal Stream (ESS) and a Smooth Signal Stream (SSS). EESSO aims to excavate the complementary effect between spatial-wise and frequency-wise features, so as to achieve a superior performance. A novel Uncertainty-Guided Mutual Learning Mechanism (UG-MLM) is utilized during training, which not only enables the three streams to communicate with and learn from each other, but also models the data-related heteroscedastic uncertainty as a weight for knowledge transference, and hence enables each stream to adaptively allocate knowledge from the other two streams. A large number of experiments are carried out on the widely-used CUHK-PEDES, RSTPReid and ICFG-PEDES datasets to verify the effectiveness of EESSO. Through the experimental results, it may not be hard to find that EESSO has achieved the state-of-the-art performance in supervised, weakly supervised and cross-domain text-based person retrieval settings.

Improving Text-Based Person Retrieval by Excavating All-Round Information Beyond Color.

Text-based person retrieval is the process of searching a massive visual resource library for images of a particular pedestrian, based on a textual query. Existing approaches often suffer from a problem of color (CLR) over-reliance, which can result in a suboptimal person retrieval performance by distracting the model from other important visual cues such as texture and structure information. To handle this problem, we propose a novel framework to Excavate All-round Information Beyond Color for the task of text-based person retrieval, which is therefore termed EAIBC. The EAIBC architecture includes four branches, namely an RGB branch, a grayscale (GRS) branch, a high-frequency (HFQ) branch, and a CLR branch. Furthermore, we introduce a mutual learning (ML) mechanism to facilitate communication and learning among the branches, enabling them to take full advantage of all-round information in an effective and balanced manner. We evaluate the proposed method on three benchmark datasets, including CUHK-PEDES, ICFG-PEDES, and RSTPReid. The experimental results demonstrate that EAIBC significantly outperforms existing methods and achieves state-of-the-art (SOTA) performance in supervised, weakly supervised, and cross-domain settings.

Discriminative Identity-Feature Exploring and Differential Aware Learning for Unsupervised Person Re-Identification

Unsupervised person re-identification (Re-ID) aims to learn discriminative representations for person retrieval from unlabeled data. Currently, state-of-the-art techniques accomplish this task by using instance contrastive learning, which contrasts the similarities of the instances in different views. However, existing contrastive methods only focus on the positive effects of inter-instance relationships, while neglecting the negative effects of intra-instance redundancy information. This redundancy information can generate invalid or spurious intra-class relationships during the instance contrasting process, which enlarges the intra-class gaps and increases the noisy pseudo-labels. To address this issue, we propose a discriminative identity-feature exploring and differential aware learning (DiDAL) framework to learn more discriminative intra-identity representations. Specifically, the DiDAL extracts intra-instance salient features by synthetic complementary attention, and further explores the discriminative identity features by modeling the relationship among these salient features based on graph neural networks. This strategy aims to reduce the intra-instance redundancy information. Moreover, DiDAL explores hard instances by leveraging the extracted intra-instance salient features, and matches an anchor with multiple hard positive instances to enhance the robustness of the model to noisy pseudo-labels. Extensive experiment results on two widely used person re-identification datasets and a vehicle re-identification dataset demonstrate the superiority of the proposed method compared with existing state-of-the-art methods.

A novel self-boosting dual-branch model for pedestrian attribute recognition

Pedestrian attributes carry broad information of abstract and detailed annotations. Pedestrian attribute recognition (PAR) can generate high-level semantic feature maps and provide auxiliary information in tasks such as person retrieval and re-identification. Since pedestrian attributes have broad categories and appear in complex region combinations in video sequences, enhancing the feature representation of fine-grained attributes is the key to improving pedestrian attribute recognition. This paper proposes an end-to-end feature-enhanced multi-scale dual-branch model for multiple attributes recognition (FEMDAR) with a feature-enhanced block (FEB) and multi-scale modules. The FEB module fuses the contextual features of each level, stacking different rates of dilated convolutions to expand the receptive field of the feature expression effectively. Simultaneously, the feature-enhanced block module utilizes a dual-branch structure to incorporate enhanced feature representation. Moreover, a new residual module is designed with multi-scale modules to extract the inconsistency in various attribute scales. Hence, the proposed model can achieve robust feature representation that can flexibly adapt to multiple pedestrian attributes of different scales. We validate our model on three public large-scale pedestrian attribute datasets. The experimental results show that the FEMDAR model shows prominent advantages in instance-based measurements. Furthermore, the ablation study shows the effectiveness of the proposed FEB and multi-scale modules in feature presentation.

Gait-Assisted Video Person Retrieval

Video person retrieval aims at matching video clips of the same person across non-overlapping camera views, where video sequences contain more comprehensive information, e.g., temporal cues. How to extract useful temporal cues is the key to the success of a video person retrieval system. Gait, as a unique biometric modality indicating the way people walk, contains informative temporal information. To date, it is not clear how to fully utilize gait to boost the performance of video person retrieval. In this paper, to validate whether gait could help retrieve person in videos, we build a two-stream architecture, named appearance-gait network (AGNet), to jointly learn the appearance features and gait features from RGB video clips and silhouette video clips. We further explore how to fully utilize gait features to enhance the video feature representation. Specifically, we propose an appearance-gait attention module (AGA) to fuse a discriminative feature representation for the person retrieval task. Furthermore, to eliminate the requirement of silhouette video clips during inference, we propose a simple yet effective appearance-gait distillation module (AGD) which transfers the gait knowledge to appearance stream. As such, we are able to perform the enhanced video person retrieval without silhouette video clips, which makes the inference more flexible and practical. To the best of our knowledge, our work is the first to successfully introduce such appearance-gait knowledge distillation design for video person retrieval. We verify the effectiveness of the proposed methods on two large-scale challenging benchmarks of MARS and DukeMTMC-VideoReID. Extensive experiments demonstrate superior or comparable performance compared to the state-of-the-art methods while being much simpler. Source code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/yangyangkiki/Gait-Assisted-Video-Reid</uri> .

Pose matters: Pose guided graph attention network for person re-identification

Person re-Identification (reID), aiming at retrieving a person across different cameras, has been playing a more and more important role in the construction of smart city and social security. For deep-learning-based reID methods, it has been proved that using local feature together with global feature could help to give robust representation for person retrieval. Human pose information can provide the locations of human skeleton to effectively guide the network to pay more attention to these key areas, and can also help to reduce the noise distractions from background or occlusions. Based on human pose, a Pose Guided Graph Attention (PGGA) network is proposed in this paper, which is a multi-branch architecture consisting of one branch for global feature and two branches for local key-point features. A graph attention convolution layer is carefully designed to re-assign the contribution weight of each extracted local feature by modeling the similarity relations. The experimental results demonstrate the effectiveness of our approach on discriminative feature learning. Our model achieves the state-of-the-art performance on several mainstream evaluation datasets. A plenty of ablation studies and different kinds of comparison experiments are conducted to prove the effectiveness of this work, including the tests on occluded datasets and cross-domain datasets. Moreover, we further design supplementary tests in practical scenario to indicate the advantage of our work in real-word applications.

Illumination Unification for Person Re-Identification

The performance of person re-identification (re-ID) is easily affected by illumination variations caused by different shooting times, places and cameras. Existing illumination-adaptive methods usually require annotating cross-camera pedestrians on each illumination scale, which is unaffordable for a long-term person retrieval system. The cross-illumination person retrieval problem presents a great challenge for accurate person matching. In this paper, we propose a novel method to tackle this task, which only needs to annotate pedestrians on one illumination scale. Specifically, (i) we propose a novel Illumination Estimation and Restoring framework (IER) to estimate the illumination scale of testing images taken at different illumination conditions and restore them to the illumination scale of training images, such that the disparities between training images with uniform illumination and testing images with varying illuminations are reduced. IER achieves promising results on illumination-adaptive dataset and proving itself a proper baseline for cross-illumination person re-ID. (ii) we propose a Mixed Training strategy using both Original and Reconstructed images (MTOR) to further improve model performance. We generate reconstructed images that are consistent with the original training images in content but more similar to the restored images in style. The reconstructed images are combined with the original training images for supervised training to further reduce the domain gap between original training images and restored testing images. To verify the effectiveness of our method, some simulated illumination-adaptive datasets are constructed with various illumination conditions. Extensive experimental results on the simulated datasets validate the effectiveness of the proposed method. The source code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FadeOrigin/IUReId</uri> .

Identifying Visible Parts via Pose Estimation for Occluded Person Re-Identification.

We focus on the occlusion problem in person re-identification (re-id), which is one of the main challenges in real-world person retrieval scenarios. Previous methods on the occluded re-id problem usually assume that only the probes are occluded, thereby removing occlusions by manually cropping. However, this may not always hold in practice. This article relaxes this assumption and investigates a more general occlusion problem, where both the probe and gallery images could be occluded. The key to this challenging problem is depressing the noise information by identifying bodies and occlusions. We propose to incorporate the pose information into the re-id framework, which benefits the model in three aspects. First, it provides the location of the body. We then design a Pose-Masked Feature Branch to make our model focus on the body region only and filter those noise features brought by occlusions. Second, the estimated pose reveals which body parts are visible, giving us a hint to construct more informative person features. We propose a Pose-Embedded Feature Branch to adaptively re-calibrate channel-wise feature responses based on the visible body parts. Third, in testing, the estimated pose indicates which regions are informative and reliable for both probe and gallery images. Then we explicitly split the extracted spatial feature into parts. Only part features from those commonly visible parts are utilized in the retrieval. To better evaluate the performances of the occluded re-id, we also propose a large-scale data set for the occluded re-id with more than 35 000 images, namely Occluded-DukeMTMC. Extensive experiments show our approach surpasses previous methods on the occluded, partial, and non-occluded re-id data sets.

Attribute based spatio-temporal person retrieval in video surveillance

Many venues, such as airports, railway stations, and shopping malls, have video surveillance systems for security and monitoring. However, searching for and retrieving people based on attribute descriptions in a large number of videos is difficult, particularly with weather variations and crowded places. Most of the existing attribute-based person retrieval systems consist of two main modules: object detection and person attribute recognition. The common drawbacks of object detection in the existing methods are false-positive, missing detection, and multi bounding boxes for the same object. Moreover, attribute recognition algorithms suffer from low accuracy for a single attribute classifier, while attributes error spread in the cascading multi-attribute classifier. This paper overcomes these issues by applying the ByteTrack algorithm instead of object detection to exploit the person's spatio-temporal information and generate a tube that maintains all the boxes that include the objects and associates high and low score boxes of the objects without raising false positive detection. Also, linking each person bounding boxes together results in more accurate attributes recognition than defining the attributes of each bounding box separately. Moreover, the proposed algorithm merges between selected predictions of two attribute recognition algorithms to improve the recognition performance. An extensive empirical evaluation was carried out on the SoftBioSearch database. The simulation results reveal that the proposed retrieval algorithm provides effective retrieval performance that exceeds the best conventional method by 14%.

SUM: Serialized Updating and Matching for text-based person retrieval

The central problem of text-based person retrieval is how to properly bridge the gap between heterogeneous cross-modal data. Many of the previous works contrive to learn a latent common space to bridge the modality gap and extract modality-invariant feature vectors. Within these methods, the common space mapping and cross-modal information matching operations are conducted in a one-off manner, which aims to extract sufficient discriminative clues from the high-dimensional multi-modal data at first glance, but it is inconsistent with the fact that humans usually follow a step-by-step process to properly recognize and match two objects. Intuitively, the large heterogeneity gap between multi-modal data can be better bridged by gradually analyzing the complex cross-modal relationships. In this paper, we propose a Serialized Updating and Matching (SUM) method for text-based person retrieval to bridge the heterogeneity gap between cross-modal data in a step-by-step manner. The core component of SUM is the proposed Memory Gating Modules (MGM), which can be stacked to gradually update and match features extracted from visual/textual modalities. To fully excavate the correlations lie within multi-granular cross-modal data, two variants are designed to care for both global and fine-grain local information, namely, Global Memory Gating Module (GMGM) and Fine-grained Memory Gating Module (FMGM) with which the updating rate of information at each step is dynamically determined after observing the feature in opposite modality. Moreover, SUM can be flexibly utilized as an add-on to any multi-granular text-based person retrieval methods to further improve the performance. We evaluate our proposed method on two text-based person retrieval datasets CUHK-PEDES and RSTPReid along with two general cross-modal retrieval datasets Flickr8K and Flickr30K to see its generalization ability. Experimental results present that the proposed SUM outperforms existing methods and achieves the state-of-the-art performance.

Loss function search for person re-identification

In recent years, person re-identification, which learns discriminative features for the specific person retrieval problem across non-overlapping cameras, has attracted extensive attention. One of the main challenges in person re-identification with deep neural networks is the design of the loss function, which plays a vital role in improving the discrimination of the learned features. However, most existing models utilize the hand-designed loss functions, which are usually sub-optimal and time-consuming. The search spaces of the two existing AutoML-based methods are either too complicated or too simple to include various forms of loss functions. In order to solve the irrationality of the above search spaces, in this paper, we propose a method of AutoML for loss function search named LFS-ReID for person ReID in the framework of the margin-based softmax loss function. Specifically, we first analyze the margin-based softmax loss function and conclude four key properties. Then we carefully design a sampling distribution based on the non-independent truncated Gaussian distributions to sample the loss function, which conforms to the above four properties. Finally, a method based on reinforcement learning is adopted to optimize the sampling distribution dynamically. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on four commonly used datasets.

Learning discriminative region representation for person retrieval

Region-level representation learning plays a key role in providing discriminative information for person retrieval. Current methods rely on heuristically coarse-grained region strips or directly borrow pixel-level annotations from pretrained human parsing models for region representation learning. How to learn a discriminative region representation within fine-grained segments while avoiding expensive pixel-level annotations is rarely discussed. To that end, we introduce a novel identity-guided human region segmentation (HRS) method for person retrieval. Via learning a set of distinct region bases that are consistent across a given dataset, HRS can predict informative region segments by grouping intermediate feature vectors based on their similarity to these bases. The predicted segments are iteratively refined for discriminative region representation learning. HRS enjoys two advantages: (1) HRS learns region segmentation using only identity labels, making it a much more practical solution to person retrieval. (2) By jointly learning global appearance and local granularity cues, HRS enables a comprehensive feature representation learning. We verify the effectiveness of the proposed HRS on four challenging benchmark datasets of Market1501, DukeMTMC-reID, CUHK03, and Occluded-DukeMTMC. Extensive experiments demonstrate superior performance over the state-of-the-art region-based methods. For instance, on the CUHK03-labeled dataset, the performance increases from 74.1% mAP and 76.5% rank-1 accuracy to 81.5% (+7.4%) mAP and 83.2% (+6.7%) rank-1 accuracy.

Person retrieval in surveillance using textual query: a review

Recent advancement of research in biometrics, computer vision, and natural language processing has discovered opportunities for person retrieval from surveillance videos using textual query. The prime objective of a surveillance system is to locate a person using a description, e.g., a short woman with a pink t-shirt and white skirt carrying a black purse. She has brown hair. Such a description contains attributes like gender, height, type of clothing, colour of clothing, hair colour, and accessories. Such attributes are formally known as soft biometrics. They help bridge the semantic gap between a human description and a machine as a textual query contains the person’s soft biometric attributes. It is also not feasible to manually search through huge volumes of surveillance footage to retrieve a specific person. Hence, automatic person retrieval using vision and language-based algorithms is becoming popular. In comparison to other state-of-the-art reviews, the contribution of the paper is as follows: 1. Recommends most discriminative soft biometrics for specific challenging conditions. 2. Integrates benchmark datasets and retrieval methods for objective performance evaluation. 3. A complete snapshot of techniques based on features, classifiers, number of soft biometric attributes, type of the deep neural networks, and performance measures. 4. The comprehensive coverage of person retrieval from handcrafted features based methods to end-to-end approaches based on natural language description.

Learning deep part-aware embedding for person retrieval

Person retrieval is an important vision task, aiming at matching the images of the same person under various camera views. The key challenge of person retrieval lies in the large intra-class variations among the person images. Therefore, how to learn discriminative feature representations becomes the core problem. In this paper, we propose a deep part-aware representation learning method for person retrieval. First, an improved triplet loss is introduced such that the global feature representations from the same identity are closely clustered. Meanwhile, a localization branch is proposed to automatically localize those discriminative person-wise parts or regions, only using identity labels in a weakly supervised manner. Via the learning simultaneously guided by the global branch and the localization branch, the proposed method can further improve the performance for person retrieval. Through an extensive set of ablation studies, we verify that the localization branch and the improved triplet loss each contributes to the performance boosts of the proposed method. Our model obtains superior (or comparable) performance compared to state-of-the-art methods for person retrieval on the four public person retrieval datasets. On the CUHK03-labeled dataset, for instance, the performance increases from 73.0% mAP and 77.9% rank-1 accuracy to 80.8% (+7.8%) mAP and 83.9% (+6.0%) rank-1 accuracy.