Person Re-identification Task Research Articles

Uncertainty-guided joint attention and contextual relation network for person re-identification

Due to the influence of factors such as camera angle and pose changes, some salient local features are often suppressed in person re-identification tasks. Moreover, many existing person re-identification methods do not consider the relation between features. To address these issues, this paper proposes two novel approaches: (1) To solve the problem of being confused and misidentified when local features of different individuals have similar attributes, we design a contextual relation network that focuses on establishing the relationship between local features and contextual features, so that all local features of the same person both contain contextual information. (2) To fully and correctly express key local features, we propose an uncertainty-guided joint attention module. The module focuses on the joint representation of individual pixels and local spatial features to enhance the credibility of local features. Finally, our method achieves competitive performance on four widely recognized datasets compared with state-of-the-art methods.

EcReID: Enhancing Correlations from Skeleton for Occluded Person Re-Identification

Person re-identification is a challenging task due to the lack of person image information in occluded scenarios. The current methods for person re-identification only take into account global information, neglect local information, and are not responsive to changes in input. Additionally, these methods do not address the issue of inaccurate joint detection caused by occlusion. In this paper, we propose an occluded person re-identification method based on a graph model and deformable method, which is able to simultaneously focus on global and local information and can flexibly adapt to local information and changes in the input, efficiently resolving issues such as occluded or incorrect joint information. Our method consists of three modules: the mutual help denoising module, inter-node aggregation and update module, and graph matching module. The mutual help denoising module acquires global features and person skeleton node features using a CNN backbone network and a pose estimation model, respectively. It uses symmetric deformable graph attention to obtain the local and global features of the joint points in different views, correcting the information of incorrect nodes and extracting favorable human features. The inter-node aggregation and update module employs deformable graph convolution operations to enhance the relations between the nodes in the same view, resulting in higher-order information. The graph matching module uses graph matching methods based on the human topology to obtain a more accurate similarity calculation for masked images. Experimental results on the Occluded-Duck and Occluded-REID datasets show that our proposed method achieves Rank-1 accuracies of 64.8% and 84.5%, respectively, outperforming current mainstream methods such as HOReID. Our method also achieves good results on the MARKET-1501 and DukeMTMC-ReID datasets. These results demonstrate that our proposed method can extract person features well and effectively improve the accuracy of person re-identification tasks.

An adaptive cross‐scale transformer based on graph signal processing for person re‐identification

AbstractExtracting robust feature representation is one of the key challenges for person re‐identification (ReID) task. Although convolution neural network (CNN)‐based methods have achieved great success, they still cannot handle the part occlusion and misalignment caused by limited receptive field. Recently, pure transformer models have shown its power in the person ReID task. However, current transformer models adopt patches of equal‐scale as input, and cannot solve the problem of cross‐scale interaction properly. To overcome this problem, an adaptive cross‐scale transformer from a perspective of the graph signal, named ACSFormer, is proposed. Specifically, the self‐attention module is first treated as an undirected fully connected graph. And then, “node variation” is introduced as an indicator to adaptively merge neighbourhood tokens. To the best of the authors’ knowledge, their ACSFormer is the first work to attempt to combine pure transformers and graph signal processing in the field of person ReID. Extensive evaluations are conducted on three person ReID datasets to validate the performance of ACSFormer. Experiments demonstrate that this ACSFormer performs on par with state‐of‐the‐art CNN‐based methods and consistently improves transformer‐based baseline, for example, surpassing ViT‐baseline by 2.5%, 2.7% and 4.8% mAP on Market1501, DukeMTMC‐reID and MSMT17, respectively.

Unsupervised person re-identification via multi-domain joint learning

Deep learning techniques have achieved impressive progress in the task of person re-identification. However, how to generalize a learned model from the source domain to the target domain remains a long-standing challenge. Inspired by the fact that the enrichment of data diversity and the utilization of miscellaneous semantic features can lead to better generalization ability, we design a model that integrates a novel data augmentation method with a multi-label assignment strategy to achieve semantic features decoupling in the source domain. The pre-trained model is employed to extract several kinds of semantic features from the target dataset, and each kind of semantic features is regarded as a specific domain. We then cluster features of each domain and exploit the connection between different clustering results to perform self-distillation for generating more reliable pseudo labels. Finally, the obtained pseudo labels are used to fine-tune the pre-trained model to achieve model transfer from the source domain to the target one. Extensive experiments demonstrate that our approach outperforms some state-of-the-art methods by a clear margin and even surpass some supervised methods. Source code is available at: https://www.github.com/flychen321/MDJL.

Survey of Cross-Modal Person Re-Identification from a Mathematical Perspective

Person re-identification (Re-ID) aims to retrieve a particular pedestrian’s identification from a surveillance system consisting of non-overlapping cameras. In recent years, researchers have begun to focus on open-world person Re-ID tasks based on non-ideal situations. One of the most representative of these is cross-modal person Re-ID, which aims to match probe data with target data from different modalities. According to the modalities of probe and target data, we divided cross-modal person Re-ID into visible–infrared, visible–depth, visible–sketch, and visible–text person Re-ID. In cross-modal person Re-ID, the most challenging problem is the modal gap. According to the different methods of narrowing the modal gap, we classified the existing works into picture-based style conversion methods, feature-based modality-invariant embedding mapping methods, and modality-unrelated auxiliary information mining methods. In addition, by generalizing the aforementioned works, we find that although deep-learning-based models perform well, the black-box-like learning process makes these models less interpretable and generalized. Therefore, we attempted to interpret different cross-modal person Re-ID models from a mathematical perspective. Through the above work, we attempt to compensate for the lack of mathematical interpretation of models in previous person Re-ID reviews and hope that our work will bring new inspiration to researchers.

Effect of Reconstruction Losses in Discriminative and Generative Learning based Networks for the Person Re-identification

The Person Re-identification (Re-ID) task has gained popularity in recent times. Researchers are continuously looking to improve the accuracy of the existing person Re-ID systems. Identifying the person from the surveillance footage can be essential to security concerns. Currently, there are many state-of-art Person Re-ID systems available. Deep learning frameworks are also adopted for designing Re-ID systems. Apart from deep learning-based approaches, the Generative Adversarial Networks (GAN) based approach also gained substantial interest in Person Re-ID tasks. Augmentation of training data has significantly improved the performance of the system. Our primary objective is to analyze the effect of applying different reconstruction losses and their combinations on the GAN-based approach. The Discriminative and Generative Learning (DG-Net) approach is chosen for carrying out this study from other existing GAN-based systems. DG-Net is currently considered benchmarked in the GAN-based method for person Re-ID. Analysis shows that the proposed idea of using a variety of reconstruction losses simultaneously significantly improves the existing system's performance. Using the proposed technique of fusing multiple Losses simultaneously, we achieved a massive performance gain of 20.57% over the current benchmarked approach on the Market1501 dataset. This paper includes a thorough study of different loss functions and their effect on the generated images for performing Person Re-ID tasks.

Multi-Stage Spatio-Temporal Aggregation Transformer for Video Person Re-Identification

In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specifically, MSTAT consists of three stages to encode the attribute-associated, the identity-associated, and the attribute-identity-associated information from the video clips, respectively, achieving the holistic perception of the input person. We combine the outputs of all the stages for the final identification. In practice, to save the computational cost, the Spatial-Temporal Aggregation (STA) modules are first adopted in each stage to conduct the self-attention operations along the spatial and temporal dimensions separately. We further introduce the Attribute-Aware and Identity-Aware Proxy embedding modules (AAP and IAP) to extract the informative and discriminative feature representations at different stages. All of them are realized by employing newly designed self-attention operations with specific meanings. Moreover, temporal patch shuffling is also introduced to further improve the robustness of the model. Extensive experimental results demonstrate the effectiveness of the proposed modules in extracting the informative and discriminative information from the videos, and illustrate the MSTAT can achieve state-of-the-art accuracies on various standard benchmarks.

Translation, Association and Augmentation: Learning Cross-Modality Re-identification from Single-Modality Annotation.

Daytime visible modality (RGB) and night-time infrared (IR) modality person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. However, training a cross-modality ReID model requires plenty of cross-modality (visible-infrared) identity labels that are more expensive than single-modality person ReID. To alleviate this issue, this paper studies unsupervised domain adaptive visible infrared person re-identification (UDA-VI-ReID) task without the reliance on any cross-modality annotation. To transfer learned knowledge from the labelled visible source domain to the unlabelled visible-infrared target domain, we propose a Translation, Association and Augmentation (TAA) framework. Specifically, the modality translator is firstly utilized to transfer visible image to infrared image, formulating generated visible-infrared image pairs for cross-modality supervised training. A Robust Association and Mutual Learning (RAML) module is then designed to exploit the underlying relations between visible and infrared modalities for label noise modeling. Moreover, a Translation Supervision and Feature Augmentation (TSFA) module is designed to enhance the discriminability by enriching the supervision with feature augmentation and modality translation. The extensive experimental results demonstrate that our method significantly outperforms current state-of-the-art unsupervised methods under various settings, and even surpasses some supervised counterparts, providing a powerful baseline for UDA-VI-ReID.

Attention Map Guided Transformer Pruning for Occluded Person Re-Identification on Edge Device

Due to its significant capability of modeling long-range dependencies, vision transformer (ViT) has achieved promising success in both holistic and occluded person re-identification (Re-ID) tasks. However, the inherent problems of transformers such as the huge computational cost and memory footprint are still two unsolved issues that will block the deployment of ViT based person Re-ID models on resource-limited edge devices. Our goal is to reduce both the inference complexity and model size without sacrificing the comparable accuracy on person Re-ID, especially for tasks with occlusion. To this end, we propose a novel attention map guided (AMG) transformer pruning method, which removes both redundant tokens and heads with the guidance of the attention map in a hardware-friendly way. We first calculate the entropy in the key dimension and sum it up for the whole map, and the corresponding head parameters of maps with high entropy will be removed for model size reduction. Then we combine the similarity and first-order gradients of key tokens along the query dimension for token importance estimation and remove redundant key and value tokens to further reduce the inference complexity. Comprehensive experiments on Occluded DukeMTMC and Market-1501 demonstrate the effectiveness of our proposals. For example, our proposed pruning strategy on ViT-Base enjoys <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">29.4%</b> <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">FLOPs</b> savings with <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.2%</b> drop on Rank-1 and <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0.4%</b> improvement on mAP, respectively. Code and models have been made available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/NUST-Machine-Intelligence-Laboratory/AMG</uri> .

Dual Level Adaptive Weighting for Cloth-Changing Person Re-Identification.

For the long-term person re-identification (ReID) task, pedestrians are likely to change clothes, which poses a key challenge in overcoming drastic appearance variations caused by these cloth changes. However, analyzing how cloth changes influence identity-invariant representation learning is difficult. In this context, varying cloth-changed samples are not adaptively utilized, and their effects on the resulting features are overshadowed. To address these limitations, this paper aims to estimate the effect of cloth-changing patterns at both the image and feature levels, presenting a Dual-Level Adaptive Weighting (DLAW) solution. Specifically, at the image level, we propose an adaptive mining strategy to locate the cloth-changed regions for each identity. This strategy highlights the informative areas that have undergone changes, enhancing robustness against cloth variations. At the feature level, we estimate the degree of cloth-changing by modeling the correlation of part-level features and re-weighting identity-invariant feature components. This further eliminates the effects of cloth variations at the semantic body part level. Extensive experiments demonstrate that our method achieves promising performance on several cloth-changing datasets. Code and models are available at https: //github.com/fountaindream/DLAW.

A Novel Mix-Normalization Method for Generalizable Multi-Source Person Re-Identification

Person re-identification (Re-ID) has achieved great success in the supervised scenario. However, it is difficult to directly transfer the supervised model to arbitrary unseen domains due to the model overfitting to the seen source domains. In this paper, we aim to tackle the generalizable multi-source person Re-ID task (i.e., there are multiple available source domains, and the testing domain is unseen during training) from the data augmentation perspective, thus we put forward a novel method, termed MixNorm. It consists of domain-aware mix-normalization (DMN) and domain-aware center regularization (DCR). Different from the conventional data augmentation, the proposed domain-aware mix-normalization enhances the diversity of features during training from the normalization perspective of the neural network, which can effectively alleviate the model overfitting to the source domains, so as to boost the generalization capability of the model in the unseen domain. To further promote the efficacy of the proposed DMN, we exploit the domain-aware center regularization to better map the diversely generated features into the same space. Extensive experiments on multiple benchmark datasets validate the effectiveness of the proposed method and show that the proposed method can outperform the state-of-the-art methods. Besides, further analysis also reveals the superiority of the proposed method.

Cross-Modality Transformer With Modality Mining for Visible-Infrared Person Re-Identification

Visible-infrared cross-modality person re-identification is a challenging ReID task, which aims to retrieve and match the same identity's images between the heterogeneous visible and infrared modalities. Thus, the core of this task is to bridge the huge gap between these two modalities. The existing convolutional neural network-based methods mainly face the problem of insufficient perception of modalities' information, and can not learn good discriminative modality-invariant embeddings for identities, which limits their performance. To solve these problems, we propose a cross-modality transformer-based method (CMTR) for the visible-infrared person re-identification task, which can explicitly mine the information of each modality and generate better discriminative features based on it. Specifically, to capture modalities' characteristics, we design the novel modality embeddings, which are fused with token embeddings to encode modalities' information. Furthermore, to enhance representation of modality embeddings and adjust matching embeddings' distribution, we propose a modality-aware enhancement loss based on the learned modalities' information, reducing intra-class distance and enlarging inter-class distance. To our knowledge, this is the first work of applying transformer network to the cross-modality re-identification task. We implement extensive experiments on the public SYSU-MM01 and RegDB datasets, and our proposed CMTR model's performance significantly surpasses existing outstanding CNN-based methods.

Describe, Spot and Explain: Interpretable Representation Learning for Discriminative Visual Reasoning.

Despite the recent success achieved by deep neural networks (DNNs), it remains challenging to disclose/explain the decision-making process from the numerous parameters and complex non-linear functions. To address the problem, explainable AI (XAI) aims to provide explanations corresponding to the learning and prediction processes for deep learning models. In this paper, we propose a novel representation learning framework of Describe, Spot and eXplain (DSX). Based on the architecture of Transformer, our proposed DSX framework is composed of two learning stages, descriptive prototype learning and discriminative prototype discovery. Given an input image, the former stage is designed to derive a set of descriptive representations, while the latter stage further identifies a discriminative subset, offering semantic interpretability for the corresponding classification tasks. While our DSX does not require any ground truth attribute supervision during training, the derived visual representations can be practically associated with physical attributes provided by domain experts. Extensive experiments on fine-grained classification and person re-identification tasks qualitatively and quantitatively verify the use our DSX model for offering semantically practical interpretability with satisfactory recognition performances.

Two-phase self-supervised pretraining for object re-identification

Self-supervised pretraining (e.g., MAE) based on reconstructing masked image patches has become increasingly popular in recent months. It commonly pretrains models on the large-scale ImageNet dataset with a considerably wider range of image categories, and then finetunes on downstream tasks such as classification, detection, segmentation, etc. However, we discover that for finetuning domain-specific tasks usually with a few or limited semantic spaces (e.g., object re-identification (ReID)), the model pretrained on a wider variety of sources (e.g., ImageNet) fails to be a good initialization. To address the problem, we propose a second-phase self-supervised pretraining on the specific domain, based on the self-supervised ImageNet pretraining models in the first phase. We perform studies on 2 domains of object ReID, i.e., person and vehicle ReID, where person ReID domain involves four datasets (Market-1501, DukeMTMC, MSMT17, Occluded-Duke) and vehicle ReID domain contains 2 datasets (VeRi-776, VehicleID). Through extensive experiments, we observe significant improvements in every single dataset based on the second-phase self-supervised pretraining over the entire unlabeled domain data. Specifically in person ReID task, compared to the self-supervised model in the first phase, our approach achieves +8.5%/+11.5%/+13.3%/+17.2% mAP improvements in Market-1501, DukeMTMC, MSMT17 and Occluded-Duke datasets, respectively. Furthermore, it also outperforms the supervised pretraining baselines, achieving improved performance by +1.5%/+4.3%/+3.9%/+4.2% mAP. We observe similar inspiring improvements in vehicle ReID tasks. Additionally, we also perform the second-phase pretraining on the specific tasks using its own unlabeled data, namely task-specific pretraining (TSP), which also consistently shows encouraging results across object ReID tasks.

Person re-identification accuracy improvement by training a CNN with the new large joint dataset and re-rank

The paper is aimed to improve person re-identification accuracy in distributed video surveillance systems based on constructing a large joint image dataset of people for training convolutional neural networks (CNN). For this aim, an analysis of existing datasets is provided. Then, a new large joint dataset for person re-identification task is constructed that includes the existing public datasets CUHK02, CUHK03, Market, Duke, MSMT17 and PolReID. Testing for re-identification is performed for such frequently cited CNNs as ResNet-50, DenseNet121 and PCB. Re-identification accuracy is evaluated by using the main metrics Rank, mAP and mINP. The use of the new large joint dataset makes it possible to improve Rank1 mAP, mINP on all test sets. Re-ranking is used to further increase the re-identification accuracy. Presented results confirm the effectiveness of the proposed approach.

Unsupervised person re-identification by patch-based nearest neighbor mining

Unsupervised person re-identification (re-ID) is still challenging due to the difficulty in learning discriminative features without labeling information. Some existing methods use the instance specificity learning to learn image features without direct supervision signals. However, this kind of instance specificity learning treats each sample as an individual instance and can not mine the inter-sample relationships, which limits the ability to explore the category boundaries. To address the unsupervised person re-ID, this paper proposes a patch-based nearest neighbor mining (PNNM) method. The proposed PNNM utilizes a patch-based feature extraction model to capture patch features from person images. Then the patch discrimination mining (PDM) is applied to supervise the model to learn a discriminative patch feature space. Meanwhile, the nearest neighbor mining (NNM) is introduced to guide the model to mine the image-level discrimination from unlabeled person images. The NNM aims to explore nearest neighbor sets with high category consistency on the whole training dataset. A two-stage training process is built with the PDM and NNM to enable the model to capture both patch-level and image-level visual clues. Extensive experiments are conducted to show the effectiveness of the proposed PNNM method in the unsupervised person re-ID task.

Joint Cross-Consistency Learning and Multi-Feature Fusion for Person Re-Identification

To solve the problem of inadequate feature extraction by the model due to factors such as occlusion and illumination in person re-identification tasks, this paper proposed a model with a joint cross-consistency learning and multi-feature fusion person re-identification. The attention mechanism and the mixed pooling module were first embedded in the residual network so that the model adaptively focuses on the more valid information in the person images. Secondly, the dataset was randomly divided into two categories according to the camera perspective, and a feature classifier was trained for the two types of datasets respectively. Then, two classifiers with specific knowledge were used to guide the model to extract features unrelated to the camera perspective for the two types of datasets so that the obtained image features were endowed with domain invariance by the model, and the differences in the perspective, attitude, background, and other related information of different images were alleviated. Then, the multi-level features were fused through the feature pyramid to concern the more critical information of the image. Finally, a combination of Cosine Softmax loss, triplet loss, and cluster center loss was proposed to train the model to address the differences of multiple losses in the optimization space. The first accuracy of the proposed model reached 95.9% and 89.7% on the datasets Market-1501 and DukeMTMC-reID, respectively. The results indicated that the proposed model has good feature extraction capability.

Soft Person Reidentification Network Pruning via Blockwise Adjacent Filter Decaying.

Deep learning has shown significant successes in person reidentification (re-id) tasks. However, most existing works focus on discriminative feature learning and impose complex neural networks, suffering from low inference efficiency. In fact, feature extraction time is also crucial for real-world applications and lightweight models are needed. Prevailing pruning methods usually pay attention to compact classification models. However, these methods are suboptimal for compacting re-id models, which usually produce continuous features and are sensitive to network pruning. The key point of pruning re-id models is how to retain the original filter distribution in continuous features as much as possible. In this work, we propose a blockwise adjacent filter decaying method to fill this gap. Specifically, given a trained model, we first evaluate the redundancy of filters based on the adjacency relationships to preserve the original filter distribution. Second, previous layerwise pruning methods ignore that discriminative information is enhanced block-by-block. Therefore, we propose a blockwise filter pruning strategy to better utilize the block relations in the pretrained model. Third, we propose a novel filter decaying policy to progressively reduce the scale of redundant filters. Different from conventional soft filter pruning that directly sets the filter values as zeros, the proposed filter decaying can keep the pretrained knowledge as much as possible. We evaluate our method on three popular person reidentification datasets, that is: 1) Market-1501; 2) DukeMTMC-reID; and 3) MSMT17_V1. The proposed method shows superior performance to the existing state-of-the-art pruning methods. After pruning over 91.9% parameters on DukeMTMC-reID, the Rank-1 accuracy only drops 3.7%, demonstrating its effectiveness for compacting person reidentification.

Bi‐level deep mutual learning assisted multi‐task network for occluded person re‐identification

An occluded person re-identification (ReID) approach is presented by constructing a Bi-level deep Mutual learning assisted Multi-task network (BMM), where the holistic and occluded person ReID tasks are treated as two related but not identical tasks. This is inspired by the human perception characteristic that there exist both similarities and differences when human views a holistic image and the occluded one. Specifically, a multi-task network with two branches is designed, where the convolutional neural network based feature representation part shares the weights by two tasks for commonality extraction, while the following output layers have respective weights for difference representation. Furthermore, as the non-occluded regions convey discriminative information, a bi-level mutual learning strategy is proposed and applied mutually on two branches to obtain more effective information from the non-occluded regions in the occluded images for better identity recognition. This is achieved by both feature-level and output-level mutual loss functions. Extensive experiments prove the advantages of the BMM for person ReID.

ICMiF: Interactive cascade microformers for cross-domain person re-identification

Person re-identification (ReID) arises in many applications, such as video surveillance and intelligent security. A challenge of cross-domain person ReID is the notorious distribution drift problem. Improving the accuracy of pseudo labels can promote the model to fit the target domain, so it is crucial to extract discriminative person features. The transformer can extract discriminative person features due to its ability to capture long distance dependencies, but its poor generalization ability may aggravate the influence of noise pseudo labels. The isolation between local information modeling and global information modeling further prevents the extraction of the discriminative person features. Therefore, this paper proposes an interactive cascade transformer framework. Firstly, a lightweight transformer structure (microformer) with better generalization is proposed. It maintains the advantages of traditional transformers on extracting discriminative features, and avoids the aggravation of the influence of noise labels. Secondly, an interactive cascade microformer framework (ICMiF) is proposed, which promotes the interaction between local information modeling and global information modeling to enhance the person feature representation. Local information facilitates mining global attribute dependencies, and global information improves the accuracy of local information. The experimental results demonstrate that the proposed ICMiF outperforms state-of-the-art methods for the cross-domain person ReID tasks.