Video-based Person Re-identification Research Articles

Progressive learning in cross-modal cross-scale fusion transformer for visible-infrared video-based person reidentification

Person reidentification (reID) is a complex problem that can be addressed by exploiting the complementary multi-modal information. We accomplish this by mapping these modalities into a shared space, which facilitates 24-hour surveillance systems. However, current methods for visible–infrared-based cross-modal person reID primarily concentrate on image-to-image matching, while the potential of image-to-video and video-to-video matching that offers a rich spatial–temporal representation is yet to be fully investigated. Existing cross-modal reID methods rely on score fusion or feature integration techniques to merge various heterogeneous and complementary multi-modalities, unfortunately, these methods fall short of fully utilizing the complementary information offered by different modalities. To overcome the above drawbacks, this study proposes a Cross-modality Cross-scale Fusion Transformer (CMFT) for multi-scale visible–infrared complementary information interaction, yielding a more comprehensive representation for person reID. The Cross-Modality cross-scale Fusion (CCF) is a fundamental component of the CMFT, designed to capture cross-modal correlations and propagate the fused complementary and discriminative information across multiple scales. The proposed CMFT not only aligns the two modalities into a shared modality-invariant space but also captures the temporal memory to ensure motion-invariance. To mitigate the adverse effects of modality gaps, we propose a progressive learning scheme that first introduces the Modality-Shared Refinement Loss (MSRL) to guide the CMFT towards uncovering more reliable identity-related information from features shared across modalities. Then, a Modality Discriminative Loss (MDCL) is used to tackle the challenges of significant intra-class and minimal inter-class variation. MSRL combined with MDL enhances the discriminative power of reliable features. Importantly, our CMFT model exhibits generality and scalability, as evidenced by significant performance improvements when applied to different combinations of multimodal inputs. Experimental results validate that CMFT effectively leverages the complementary semantic information in visible and infrared inputs, outperforming the existing visible–infrared-based reID methods on the HITSZ-VCM, SYSU-MM01, and RegID datasets.

A review on video person re-identification based on deep learning

Person Re-Identification (ReID) is an essential technology for matching a person across non-overlapping cameras. It has attracted increasing attention in recent years due to its wide range of applications in various real-world scenarios such as security surveillance and criminal investigation. Different from other person ReID tasks, video-based ReID uses a video clip as the retrieval input, which can provide more promising ReID performance because that the video has rich information on appearance, motion cues and pose variations on temporal pipeline. Over the last few years, many deep learning-based video person ReID have been proposed to address various challenges, such as illumination variation, complex background, occlusion, etc. To provide a more comprehensive and readable review on existing video-based person ReID methods, we propose a novel taxonomy method that observes existing methods from four perspectives: data, algorithms, computing power, and applications. Specifically, we first introduce some popular datasets and evaluation criterion used for video-based person ReID. Next, from limited data and little annotation view, we introduce data augmentation and unsupervised learning ReID. From algorithm view, we focus on reviewing supervised methods including spatial feature learning, temporal feature learning and spatio-temporal feature learning, and further discuss and conduct a systematic comparison among these approaches. From complex open-world application view, we mainly summarized domain adaption and multimodal ReID. From insufficient GPU computing power view, we mainly discuss modality-agnostic unified large-scale ReID and their lightweighting. Finally, we provide a discussion of open problems and potential research directions for the community.

In-Depth Analysis of GAF-Net: Comparative Fusion Approaches in Video-Based Person Re-Identification

This study provides an in-depth analysis of GAF-Net, a novel model for video-based person re-identification (Re-ID) that matches individuals across different video sequences. GAF-Net combines appearance-based features with gait-based features derived from skeletal data, offering a new approach that diverges from traditional silhouette-based methods. We thoroughly examine each module of GAF-Net and explore various fusion methods at the both score and feature levels, extending beyond initial simple concatenation. Comprehensive evaluations on the iLIDS-VID and MARS datasets demonstrate GAF-Net’s effectiveness across scenarios. GAF-Net achieves state-of-the-art 93.2% rank-1 accuracy on iLIDS-VID’s long sequences, while MARS results (86.09% mAP, 89.78% rank-1) reveal challenges with shorter, variable sequences in complex real-world settings. We demonstrate that integrating skeleton-based gait features consistently improves Re-ID performance, particularly with long, more informative sequences. This research provides crucial insights into multi-modal feature integration in Re-ID tasks, laying a foundation for the advancement of multi-modal biometric systems for diverse computer vision applications.

Incorporating texture and silhouette for video-based person re-identification

Silhouette is an effective modality in video-based person re-identification (ReID) since it contains features (e.g., stature and gait) complementary to the RGB modality. However, recent silhouette-assisted methods have not fully explored the spatial–temporal relations within each modality or considered the cross-modal complementarity in fusion. To address these two issues, we propose a Complete Relational Framework that includes two key components. The first component, Spatial-Temporal Relational Module (STRM), explores the spatiotemporal relations. STRM decomposes the video’s spatiotemporal context into local/fine-grained and global/semantic aspects, modeling them sequentially to enhance the representation of each modality. The second component, Modality-Channel Relational Module (MCRM), explores the complementarity between RGB and silhouette videos. MCRM aligns two modalities semantically and multiplies them to capture complementary interrelations. With these two modules focusing on intra- and cross-modal relationships, our method achieves superior results across multiple benchmarks with minimal additional parameters and FLOPs. Code and models are available at https://github.com/baist/crf.

Multi-scale spatio-temporal feature adaptive aggregation for video-based Person Re-identification

Video-based person re-identification (Re-ID) aims to retrieve the target person from video sequences captured by a distributed camera system. It remains a challenging task due to the reasons such as occlusion and misalignment in the video. To address above problem, many methods are proposed to exploit multi-scale spatio-temporal features in videos. However, established methods typically assign the equal weights to temporal or spatial features at different scales, which significantly diminishes the distinct roles of each feature. In this paper, we propose a novel Multi-scale Feature Aggregation Network (MFANet) for video-based person Re-ID. Specifically, we propose two flexible modules, Multi-scale Temporal Feature Aggregation (MTFA) and Multi-scale Spatial Feature Aggregation (MSFA). These two modules first extract different scales of temporal (dynamic, static) and spatial (coarse and fine) features, and then adaptively assign weights to each feature according to the video sequence. Both of these lightweight modules can be incorporated with 3D Convolutional Neural Network to build our MFANet. Extensive experiments on four public benchmarks demonstrate that MTFA and MSFA improve the performance of baseline architectures, and our MFANet achieves the best performance compared to other state-of-the-art methods.

Deep video-based person re-identification (Deep Vid-ReID): comprehensive survey

Person re-identification (ReID) aims to find the person of interest across multiple non-overlapping cameras. It is considered an essential step for person tracking applications which is vital for surveillance. Person ReID could be investigated either using image-based or video-based. Video-based person ReID is considered more discriminating and realistic than image-based ReID due to the massive information extracted for each person. Different deep-learning techniques have been used for video-based ReID. In this survey, recently published articles are reviewed according to video-based ReID system pipeline: deep features learning, deep metric learning, and deep learning approaches. The deep feature learning approaches are categorized into spatial and temporal approaches, while deep metric learning is divided into metric and metric learning approaches. The deep learning approaches are differentiated into: supervised, unsupervised, weakly-supervised, and one-shot learning. A detailed analysis is held for the architectures of the state-of-the-art deep learning approaches. And their performance on four benchmark datasets is compared.

Multi-Granularity Aggregation with Spatiotemporal Consistency for Video-Based Person Re-Identification.

Video-based person re-identification (ReID) aims to exploit relevant features from spatial and temporal knowledge. Widely used methods include the part- and attention-based approaches for suppressing irrelevant spatial-temporal features. However, it is still challenging to overcome inconsistencies across video frames due to occlusion and imperfect detection. These mismatches make temporal processing ineffective and create an imbalance of crucial spatial information. To address these problems, we propose the Spatiotemporal Multi-Granularity Aggregation (ST-MGA) method, which is specifically designed to accumulate relevant features with spatiotemporally consistent cues. The proposed framework consists of three main stages: extraction, which extracts spatiotemporally consistent partial information; augmentation, which augments the partial information with different granularity levels; and aggregation, which effectively aggregates the augmented spatiotemporal information. We first introduce the consistent part-attention (CPA) module, which extracts spatiotemporally consistent and well-aligned attentive parts. Sub-parts derived from CPA provide temporally consistent semantic information, solving misalignment problems in videos due to occlusion or inaccurate detection, and maximize the efficiency of aggregation through uniform partial information. To enhance the diversity of spatial and temporal cues, we introduce the Multi-Attention Part Augmentation (MA-PA) block, which incorporates fine parts at various granular levels, and the Long-/Short-term Temporal Augmentation (LS-TA) block, designed to capture both long- and short-term temporal relations. Using densely separated part cues, ST-MGA fully exploits and aggregates the spatiotemporal multi-granular patterns by comparing relations between parts and scales. In the experiments, the proposed ST-MGA renders state-of-the-art performance on several video-based ReID benchmarks (i.e., MARS, DukeMTMC-VideoReID, and LS-VID).

AA-RGTCN: reciprocal global temporal convolution network with adaptive alignment for video-based person re-identification.

Person re-identification(Re-ID) aims to retrieve pedestrians under different cameras. Compared with image-based Re-ID, video-based Re-ID extracts features from video sequences that contain both spatial features and temporal features. Existing methods usually focus on the most attractive image parts, and this will lead to redundant spatial description and insufficient temporal description. Other methods that take temporal clues into consideration usually ignore misalignment between frames and only focus on a fixed length of one given sequence. In this study, we proposed a Reciprocal Global Temporal Convolution Network with Adaptive Alignment(AA-RGTCN). The structure could address the drawback of misalignment between frames and model discriminative temporal representation. Specifically, the Adaptive Alignment block is designed to shift each frame adaptively to its best position for temporal modeling. Then, we proposed the Reciprocal Global Temporal Convolution Network to model robust temporal features across different time intervals along both normal and inverted time order. The experimental results show that our AA-RGTCN can achieve 85.9% mAP and 91.0% Rank-1 on MARS, 90.6% Rank-1 on iLIDS-VID, and 96.6% Rank-1 on PRID-2011, indicating we could gain better performance than other state-of-the-art approaches.

TF-CLIP: Learning Text-Free CLIP for Video-Based Person Re-identification

Large-scale language-image pre-trained models (e.g., CLIP) have shown superior performances on many cross-modal retrieval tasks. However, the problem of transferring the knowledge learned from such models to video-based person re-identification (ReID) has barely been explored. In addition, there is a lack of decent text descriptions in current ReID benchmarks. To address these issues, in this work, we propose a novel one-stage text-free CLIP-based learning framework named TF-CLIP for video-based person ReID. More specifically, we extract the identity-specific sequence feature as the CLIP-Memory to replace the text feature. Meanwhile, we design a Sequence-Specific Prompt (SSP) module to update the CLIP-Memory online. To capture temporal information, we further propose a Temporal Memory Diffusion (TMD) module, which consists of two key components: Temporal Memory Construction (TMC) and Memory Diffusion (MD). Technically, TMC allows the frame-level memories in a sequence to communicate with each other, and to extract temporal information based on the relations within the sequence. MD further diffuses the temporal memories to each token in the original features to obtain more robust sequence features. Extensive experiments demonstrate that our proposed method shows much better results than other state-of-the-art methods on MARS, LS-VID and iLIDS-VID.

Video-based person re-identification with complementary local and global features using a graph transformer.

In recent years, significant progress has been made in video-based person re-identification (Re-ID). The key challenge in video person Re-ID lies in effectively constructing discriminative and robust person feature representations. Methods based on local regions utilize spatial and temporal attention to extract representative local features. However, prior approaches often overlook the correlations between local regions. To leverage relationships among different local regions, we have proposed a novel video person Re-ID representation learning approach based on a graph transformer, which facilitates contextual interactions between relevant region features. Specifically, we construct a local relation graph to model intrinsic relationships between nodes representing local regions. This graph employs the architecture of a transformer for feature propagation, iteratively refining region features and considering information from adjacent nodes to obtain partial feature representations. To learn compact and discriminative representations, we have further proposed a global feature learning branch based on a vision transformer to capture the relationships between different frames in a sequence. Additionally, we designed a dual-branch interaction network based on multi-head fusion attention to integrate frame-level features extracted by both local and global branches. Finally, the concatenated global and local features, after interaction, are used for testing. We evaluated the proposed method on three datasets, namely iLIDS-VID, MARS, and DukeMTMC-VideoReID. Experimental results demonstrate competitive performance, validating the effectiveness of our proposed approach.

Anchor Association Learning for Unsupervised Video Person Re-Identification.

Video-based person re-identification (re-id) has attracted a significant attention in recent years due to the increasing demand of video surveillance. However, existing methods are usually based on the supervised learning, which requires vast labeled identities across cameras and is not suitable for real scenes. Although some unsupervised approaches have been proposed for video re-id, their performance is far from satisfactory. In this article, we propose an unsupervised anchor association learning (UAAL) framework to address the video-based person re-id task, in which the feature representation of each sampled tracklet is regarded as an anchor. Specifically, we first propose an intracamera anchor association learning (IAAL) term that learns the discriminative anchor by utilizing the affiliation relations between an image and the anchors in each camera. Then, the exponential moving average (EMA) strategy is employed to update the anchor and the updated anchors are stored into an anchor memory module. On top of that, a cross-camera anchor association learning (CAAL) term is introduced to mine potential positive anchor pairs across cameras by presenting a cyclic ranking anchor alignment and threshold filtering method. Extensive experiments conducted on two public datasets show the superiority of the proposed method; for example, our method achieves 73.2% for rank-1 accuracy and 60.1% for mean average precision (mAP) score, respectively, on MARS, similarly 89.7% and 87.0% on DukeMTMC-VideoReID.

Graph Convolution Based Efficient Re-Ranking for Visual Retrieval

Visual retrieval tasks such as image retrieval and person re-identification (Re-ID) aim at effectively and thoroughly searching images with similar content or the same identity. After obtaining retrieved examples, re-ranking is a widely adopted post-processing step to reorder and improve the initial retrieval results by making use of the contextual information from semantically neighboring samples. Prevailing re-ranking approaches update distance metrics and mostly rely on inefficient crosscheck set comparison operations while computing expanded neighbors based distances. In this work, we present an efficient re-ranking method which refines initial retrieval results by updating features. Specifically, we reformulate re-ranking based on Graph Convolution Networks (GCN) and propose a novel Graph Convolution based Re-ranking (GCR) for visual retrieval tasks via feature propagation. To accelerate computation for large-scale retrieval, a decentralized and synchronous feature propagation algorithm which supports parallel or distributed computing is introduced. In particular, the plain GCR is extended for cross-camera retrieval and an improved feature propagation formulation is presented to leverage affinity relationships across different cameras. It is also extended for video-based retrieval, and Graph Convolution based Re-ranking for Video (GCRV) is proposed by mathematically deriving a novel profile vector generation method for the tracklet. Without bells and whistles, the proposed approaches achieve state-of-the-art performances on seven benchmark datasets from three different tasks, i.e., image retrieval, person Re-ID and video-based person Re-ID. Code is publicly available: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">github.com/WesleyZhang1991/GCN rerank</uri> .

Unsupervised Domain Adaptation Via Dynamic Clustering and Co-Segment Attentive Learning for Video-Based Person Re-Identification

Unsupervised Domain Adaptation Via Dynamic Clustering and Co-Segment Attentive Learning for Video-Based Person Re-Identification

Multi-Scale Aligned Spatial-Temporal Interaction for Video-Based Person Re-Identification

Multi-Scale Aligned Spatial-Temporal Interaction for Video-Based Person Re-Identification

Spatial-temporal graph-guided global attention network for video-based person re-identification

Spatial-temporal graph-guided global attention network for video-based person re-identification

Pose-Aided Video-Based Person Re-Identification via Recurrent Graph Convolutional Network

Existing methods for video-based person re-identification (ReID) mainly learn the appearance feature of a given pedestrian via a feature extractor and a feature aggregator. However, the appearance models would fail to learn a large inter-class variance when different pedestrians have similar appearances. Considering that different pedestrians have different walking postures and body proportions, we propose to learn the discriminative pose feature beyond the appearance feature for video retrieval. Specifically, we implement a two-branch architecture to separately learn the appearance feature and pose feature, and then concatenate them together for inference. To learn the pose feature, we first detect the pedestrian pose in each frame through an off-the-shelf pose detector, and construct a temporal graph using the pose sequence. We then exploit a recurrent graph convolutional network (RGCN) to learn the node embeddings of the temporal pose graph, which devises a global information propagation mechanism to simultaneously achieve the neighborhood aggregation of intra-frame nodes and message passing among inter-frame graphs. Finally, we propose a dual-attention method (DAM) consisting of node-attention and time-attention to obtain the temporal graph representation from the node embeddings, where the self-attention mechanism is employed to learn the importance of each node and each frame. We verify the proposed method on three video-based ReID datasets, i.e., Mars, DukeMTMC and iLIDS-VID, whose experimental results demonstrate that the learned pose feature can effectively improve the performance of existing appearance models.

Spatial-temporal aware network for video-based person re-identification

Spatial-temporal aware network for video-based person re-identification

Spatiotemporal Interaction Transformer Network for Video-Based Person Reidentification in Internet of Things

Video-based person re-identification, which is a significant application in the Internet of Things, aims to identify the same person in different video sequences across non-overlapping cameras. Existing methods usually utilize temporal cues to enhance spatial features. However, these methods learn the temporal and spatial information separately, which breaks the relationship between them and ignores the positive role of temporal information for learning frame-level spatial representation in the process of spatial representation learning. In this paper, we propose a novel Spatiotemporal Interaction Transformer Network (SITN) to solve this problem. To model the temporal information and the relationship between frames, we introduce a Temporal Interaction Module (TIM) to interact between frame information. Meanwhile, we combine TIM with Spatial Transformer Encoder to explore the positive role of temporal information in the learning procedure of the frame-level spatial feature. Moreover, we propose a Transformer local learning scheme by reconstructing the 2D spatial information of the frame patch sequences and extracting local features in a striped manner to strengthen the discriminative capability of our model. Extensive experiments are conducted on four public benchmarks. The results show that our model is superior compared with state-of-the-art methods.

Video-based person re-identification with scene and person attributes

Video-based person re-identification with scene and person attributes

Unbiased Spatiotemporal Representation With Uncertainty Control for Person Reidentification

For person re-identification, most current research aims to encode the spatial and temporal information by using convolutional neural networks (CNNs) to extract spatial features and recurrent neural networks (RNNs) or their variations to discover the time dependencies. However, it ignores the effect of the complex background, which leads to a biased spatial representation. Further, it often uses the backpropagation through time (BPTT) to train RNNs. Unfortunately, it is hard to learn the long-term dependency via BPTT due to the gradient vanishing or exploding. The significance of a frame should not be biased by its position in a given sequence. In this paper, a new method is proposed to learn an unbiased semantic representation for video-based person re-identification. To handle the background clutter and occlusion, a two-branch CNN model is used to obtain the enriched representation from both the foreground person and original pedestrian images. Then, an unbiased bidirectional convolutional neural network architecture is developed to learn the unbiased spatial and temporal representation. Experimental results on three public datasets demonstrate the effectiveness of the proposed method.