Video Person Re-identification Research Articles

A review on video person re-identification based on deep learning

Person Re-Identification (ReID) is an essential technology for matching a person across non-overlapping cameras. It has attracted increasing attention in recent years due to its wide range of applications in various real-world scenarios such as security surveillance and criminal investigation. Different from other person ReID tasks, video-based ReID uses a video clip as the retrieval input, which can provide more promising ReID performance because that the video has rich information on appearance, motion cues and pose variations on temporal pipeline. Over the last few years, many deep learning-based video person ReID have been proposed to address various challenges, such as illumination variation, complex background, occlusion, etc. To provide a more comprehensive and readable review on existing video-based person ReID methods, we propose a novel taxonomy method that observes existing methods from four perspectives: data, algorithms, computing power, and applications. Specifically, we first introduce some popular datasets and evaluation criterion used for video-based person ReID. Next, from limited data and little annotation view, we introduce data augmentation and unsupervised learning ReID. From algorithm view, we focus on reviewing supervised methods including spatial feature learning, temporal feature learning and spatio-temporal feature learning, and further discuss and conduct a systematic comparison among these approaches. From complex open-world application view, we mainly summarized domain adaption and multimodal ReID. From insufficient GPU computing power view, we mainly discuss modality-agnostic unified large-scale ReID and their lightweighting. Finally, we provide a discussion of open problems and potential research directions for the community.

Situational diversity in video person re-identification: introducing MSA-BUPT dataset

Thanks to the success of deep learning over the past few years, the video person re-identification (ReID) algorithms have achieved high accuracy on multiple public benchmark datasets. However, the available video person ReID datasets cover a limited range of real-world scenarios, and they have several obvious limitations: limited camera viewing angles, tiny variations of the shooting scene, and even errors in manual labels. These disadvantages prevent video person ReID from being widely used in real-life scenarios. In this work, a new high-quality multi-situation video person ReID dataset, named MSA-BUPT, is built to promote the video person ReID task in large-scale urban surveillance. Specifically, MSA-BUPT contains 684 identities, 2,665 trajectories, and nearly 250,000 frames from 200-h videos across various complex scenarios. Person attribute annotations and unannotated video data are also provided for other research perspectives, such as cross-modality ReID, cross-domain ReID, and so on. Furthermore, two plug-and-play components are used to improve retrieval capabilities: a new scenario-based data augmentation method is proposed to alleviate the person misalignment problem; a re-ranking strategy based on person attribute is applied to make secondary adjustments to the content to the results of the model. The extensive experimental results show that the above methods improve the performance of some representative state-of-the-art models on the new dataset.

Temporal Correlation Vision Transformer for Video Person Re-Identification

Video Person Re-Identification (Re-ID) is a task of retrieving persons from multi-camera surveillance systems. Despite the progress made in leveraging spatio-temporal information in videos, occlusion in dense crowds still hinders further progress. To address this issue, we propose a Temporal Correlation Vision Transformer (TCViT) for video person Re-ID. TCViT consists of a Temporal Correlation Attention (TCA) module and a Learnable Temporal Aggregation (LTA) module. The TCA module is designed to reduce the impact of non-target persons by relative state, while the LTA module is used to aggregate frame-level features based on their completeness. Specifically, TCA is a parameter-free module that first aligns frame-level features to restore semantic coherence in videos and then enhances the features of the target person according to temporal correlation. Additionally, unlike previous methods that treat each frame equally with a pooling layer, LTA introduces a lightweight learnable module to weigh and aggregate frame-level features under the guidance of a classification score. Extensive experiments on four prevalent benchmarks demonstrate that our method achieves state-of-the-art performance in video Re-ID.

Video-based person re-identification with complementary local and global features using a graph transformer.

In recent years, significant progress has been made in video-based person re-identification (Re-ID). The key challenge in video person Re-ID lies in effectively constructing discriminative and robust person feature representations. Methods based on local regions utilize spatial and temporal attention to extract representative local features. However, prior approaches often overlook the correlations between local regions. To leverage relationships among different local regions, we have proposed a novel video person Re-ID representation learning approach based on a graph transformer, which facilitates contextual interactions between relevant region features. Specifically, we construct a local relation graph to model intrinsic relationships between nodes representing local regions. This graph employs the architecture of a transformer for feature propagation, iteratively refining region features and considering information from adjacent nodes to obtain partial feature representations. To learn compact and discriminative representations, we have further proposed a global feature learning branch based on a vision transformer to capture the relationships between different frames in a sequence. Additionally, we designed a dual-branch interaction network based on multi-head fusion attention to integrate frame-level features extracted by both local and global branches. Finally, the concatenated global and local features, after interaction, are used for testing. We evaluated the proposed method on three datasets, namely iLIDS-VID, MARS, and DukeMTMC-VideoReID. Experimental results demonstrate competitive performance, validating the effectiveness of our proposed approach.

Anchor Association Learning for Unsupervised Video Person Re-Identification.

Video-based person re-identification (re-id) has attracted a significant attention in recent years due to the increasing demand of video surveillance. However, existing methods are usually based on the supervised learning, which requires vast labeled identities across cameras and is not suitable for real scenes. Although some unsupervised approaches have been proposed for video re-id, their performance is far from satisfactory. In this article, we propose an unsupervised anchor association learning (UAAL) framework to address the video-based person re-id task, in which the feature representation of each sampled tracklet is regarded as an anchor. Specifically, we first propose an intracamera anchor association learning (IAAL) term that learns the discriminative anchor by utilizing the affiliation relations between an image and the anchors in each camera. Then, the exponential moving average (EMA) strategy is employed to update the anchor and the updated anchors are stored into an anchor memory module. On top of that, a cross-camera anchor association learning (CAAL) term is introduced to mine potential positive anchor pairs across cameras by presenting a cyclic ranking anchor alignment and threshold filtering method. Extensive experiments conducted on two public datasets show the superiority of the proposed method; for example, our method achieves 73.2% for rank-1 accuracy and 60.1% for mean average precision (mAP) score, respectively, on MARS, similarly 89.7% and 87.0% on DukeMTMC-VideoReID.

An Adaptive Interference Removal Framework for Video Person Re-Identification

Video person re-identification (V-ReID) can leverage rich spatial-temporal information embedded in sequence data to achieve better accuracy. However, it is vulnerable to the interference of inaccurate and redundant noisy frames in each sequence as well as the background clutter and person-irrelevant pixels in each frame. To solve the above issues, this paper presents an adaptive interference removal framework (IRF) to learn discriminative feature representations by removing various interference. Our IRF mainly consists of two modules including an attention-guided adaptive interference frame removal module (IFRM) and an attention-guided adaptive interference pixel removal module (IPRM). IFRM and IPRM are designed to locate task-relevant keyframes and key pixels, respectively. IFRM adopts the attention mechanism to predict frame-wise scores to characterize the contribution of each frame to the final identification task. IPRM collaboratively utilizes camera identity classification loss, person identity classification loss, target attention loss, and person mask adversarial loss for extracting pure pedestrian representations. A progressive mask augmentation strategy is designed to restrain the data distribution of the generated person masks to further guide model training. Extensive experiments demonstrate that our models outperform state-of-the-art accuracy on seven person ReID datasets.

Where to look: Multi-granularity occlusion aware for video person re-identification

Video person re-identification(re-ID) plays an important role in intelligent video surveillance, which can automatically match the same person across video clips under non-overlapping cameras. Despite great progress in re-ID, the performance of most existing methods still is corrupted severely under partial occlusion. Although some multi-granularity methods have alleviated this dilemma, these methods still suffer from weak diversity of features and conflict between rigid horizontal partition and vertical occlusion. In this paper, we propose a novel video person re-ID framework, called Multi-Granularity Occlusion Aware (MGOA), which extracts multi-granularity features by precisely erasing the occlusion. Different from previous works based on multiple granularities, the proposed MGOA predicts the partial occlusion in a coarse-to-fine manner instead of erasing the occlusion in video clips by one step. Specifically, we first propose the multi-granularity feature extraction to obtain diverse features at different levels of feature maps, which is beneficial for the fine erasure of the occlusion. Moreover, to avoid the limitation of horizontal stripes that cannot handle vertical occlusion, we design Attention-Aware Occlusion Erasure (AA-OE) that can obtain the attention maps with coarse occlusion erasure in the coarse-grained branch and the attention maps with fine occlusion erasure in the fine-grained branch. It is worth noting that each granularity in our network is not independent but relevant through the top-down information transmission between granularities, which transfers the erased occlusion feature maps of the current branch to the next finer-grained branch for guiding AA-OE to obtain more discriminative features. Extensive experiments on three challenging public benchmarks show that our MGOA can deal well with occlusion and achieves state-of-the-art performance.

Progressive unsupervised video person re-identification with accumulative motion and tracklet spatial–temporal correlation

Person re-identification (re-id) methods based on supervised learning require a large number of manually labeled samples for training, resulting in poor scalability in actual re-id applications. Existing unsupervised video person re-id methods typically focus on extracting appearance features from pedestrian videos, ignoring motion information and the fact that people usually move in groups, i.e., pedestrian spatio–temporal co-occurrence patterns. The key factor for unsupervised video person re-id is to effectively exploit both spatio–temporal clues from video sequences and cross-camera tracklet association. In this work, we propose a progressive deep learning method for unsupervised person re-id via tracklet association with spatio–temporal correlation (TASTC). Specifically, we first divide uniformly each tracklet into multiple temporally localized slices according to a time pyramid structure. Then, an initial re-id model is trained based on a two-stream convolutional architecture, and the accumulative motion context information of temporally localized slices of the tracklets per camera is learned. Finally, combining accumulative motion and tracklet spatial–temporal correlation, we associate tracklets across cameras and update the re-id model. The above steps are iterated to optimize the re-id model progressively. Experiment results demonstrate that the proposed method significantly outperforms the current state-of-the-art unsupervised video person re-identification methods on three video-based benchmark datasets, ILIDS-VID, MARS, and DukeMTMC-VideoReID.

Multitask Multigranularity Aggregation With Global-Guided Attention for Video Person Re-Identification

The goal of video-based person re-identification (Re-ID) is to identify the same person across multiple non-overlapping cameras. The key to accomplishing this challenging task is to sufficiently exploit both spatial and temporal cues in video sequences. However, most current methods are incapable of accurately locating semantic regions or efficiently filtering discriminative spatio-temporal features; so it is difficult to handle issues such as spatial misalignment and occlusion. Thus, we propose a novel feature aggregation framework, multi-task and multi-granularity aggregation with global-guided attention (MMA-GGA), which aims to adaptively generate more representative spatio-temporal aggregation features. Specifically, we develop a multi-task multi-granularity aggregation (MMA) module to extract features at different locations and scales to identify key semantic-aware regions that are robust to spatial misalignment. Then, to determine the importance of the multi-granular semantic information, we propose a global-guided attention (GGA) mechanism to learn weights based on the global features of the video sequence, allowing our framework to identify stable local features while ignoring occlusions. Therefore, the MMA-GGA framework can efficiently and effectively capture more robust and representative features. Extensive experiments on four benchmark datasets demonstrate that our MMA-GGA framework outperforms current state-of-the-art methods. In particular, our method achieves a rank-1 accuracy of 91.0% on the MARS dataset, the most widely used database, significantly outperforming existing methods.

STCA: Utilizing a spatio-temporal cross-attention network for enhancing video person re-identification

Video-based re-identification (ReID) is a crucial task in computer vision that draws increasing attention due to advances in deep learning (DL) and modern computational devices. Despite recent success with CNN architectures, single models (e.g., 2D-CNNs or 3D-CNNs) alone failed to leverage temporal information with spatial cues. This is due to uncontrolled surveillance scenarios and variable poses leading to inevitable misalignment of ROIs across the tracklets, which is accompanied by occlusion and motion blur. In this context, designing temporal and spatial cues for two different models and their combinations can be beneficial, considering the global of a video-tracklet. 3D-CNNs allow encoding of temporal information while 2D-CNNs extract spatial or appearance information. In this paper, we propose a Spatio-Temporal Cross Attention (STCA) network to utilize both 2D-CNNs and 3D-CNNs that calculate the cross attention mapping both from the layer of 3D-CNNs and 2D-CNNs along a person's trajectory to gate the following layers of 2D-CNNs; and highlight relevant appearance features for the person ReID. Given an input tracklet, the proposed cross attention (CA) is able to capture the salient regions that propagate throughout the tracklet to obtain the global view. This provides a spatio-temporal attention approach that can be dynamically aggregated with spatial features of 2D-CNNs to perform finer-grained recognition. Additionally, we exploit the advantage of utilizing cosine similarity while triplet sampling as well as for calculating the final recognition score. Experimental analyses on three challenging benchmark datasets indicate that integrating spatio-temporal cross attention into the state-of-the-art video ReID backbone CNN architecture allows for improving their recognition accuracy.

MRRNet: Learning multiple region representation for video person re-identification

Video person re-identification is a crucial component of a robust surveillance system. Within a video clip, different human regions exhibit unique stability characteristics, which would be harmful to generating a discriminative representation. Unfortunately, prior works cannot effectively deal with the stability characteristics of different regions. To tackle this problem, we propose a Multiple Region Representation Network (MRRNet) that aims to discover the discriminative information from different human regions. Firstly, a Stable Region Representation (SRR) layer is proposed to capture important clues from the stable regions and exchange temporal information by cross-relation aware operation. Secondly, a Multiple Region Representation (MRR) layer is designed to address the unstable regions and preserve the attention on stable regions. Thirdly, SRR and MRR can be conveniently inserted into multiple stages of the deep residual networks and significantly improve the performance of the network. Comprehensive experiments validate the effectiveness of our network. Particularly, MRRNet achieves 86.7% mAP and 91.1% Rank-1 accuracy on the MARS dataset, which outperforms state-of-the-arts.

SFGN: Representing the sequence with one super frame for video person re-identification

Video-based person re-identification (V-Re-ID) is more robust than image-based person re-identification (I-Re-ID) because of the additional temporal information. However, the high storage overhead of video sequences largely stems the applications of V-Re-ID. To reduce the storage overhead, we propose to represent each video sequence with only one frame. However, directly picking one frame from each sequence will reduce the performance dramatically. Thus, we propose a brand-new framework called super frame generation network (SFGN), which can encode the spatial–temporal information of a video sequence into a generated frame, which is called “super frame” to distinguish from the directly picked “key frame”. To achieve super frames of high visual quality and representation ability, we carefully design the specific-frame-feature fused skip-connection generator (SFSG). SFSG takes the role of a feature encoder and the co-trained image model can be seen as the corresponding feature decoder. To reduce the information loss in the encoding–decoding process, we further propose the feature recovery loss (FRL). To the best of our knowledge, we are the first to propose and relieve this issue. Extensive experiments on Mars, iLIDS-VID, and PRID2011 show that the proposed SFGN can generate super frames of high visual quality and representation ability. For the code, please visit the project website: .

Video person re-identification using key frame screening with index and feature reorganization based on inter-frame relation

Video person re-identification using key frame screening with index and feature reorganization based on inter-frame relation

Video Person Re-Identification with Frame Sampling-Random Erasure and Mutual Information-Temporal Weight Aggregation.

Partial occlusion and background clutter in camera video surveillance affect the accuracy of video-based person re-identification (re-ID). To address these problems, we propose a person re-ID method based on random erasure of frame sampling and temporal weight aggregation of mutual information of partial and global features. First, for the case in which the target person is interfered or partially occluded, the frame sampling–random erasure (FSE) method is used for data enhancement to effectively alleviate the occlusion problem, improve the generalization ability of the model, and match persons more accurately. Second, to further improve the re-ID accuracy of video-based persons and learn more discriminative feature representations, we use a ResNet-50 network to extract global and partial features and fuse these features to obtain frame-level features. In the time dimension, based on a mutual information–temporal weight aggregation (MI–TWA) module, the partial features are added according to different weights and the global features are added according to equal weights and connected to output sequence features. The proposed method is extensively experimented on three public video datasets, MARS, DukeMTMC-VideoReID, and PRID-2011; the mean average precision (mAP) values are 82.4%, 94.1%, and 95.3% and Rank-1 values are 86.4%, 94.8%, and 95.2%, respectively.

Video person re-identification based on RGB triple pyramid model

Video person re-identification based on RGB triple pyramid model

Sampling and Re-Weighting: Towards Diverse Frame Aware Unsupervised Video Person Re-Identification

Video person re-identification (re-ID) methods extract richer features from video tracklets than image-based ones and have received growing attention. However, existing supervised methods require numerous cross-camera identity labels, which is impractical for large-scale data. Although clustering-based unsupervised methods have been exploited to obtain pseudo labels and train the models iteratively for video person re-ID, they remain in their infancy due to the diversity of person images and uncertainty in the image quality of video tracklets. In this work, we employ two strategies of <u>S</u>ampling and <u>R</u>e-weighting for <u>C</u>lustering (SRC) to obtain robust and discriminative person feature representations. This method considers the influence of two kinds of frames in the tracklet: 1)&#x00A0;Detection errors or heavy occlusions generate noisy frames in the tracklet. These tracklets with noisy frames may be assigned with unreliable annotations during clustering. 2)&#x00A0;Different frames are identified by the model with varying degrees of difficulty, caused by pose changes or partial occlusions. We call them hard frames, which are hard to identify but informative. To alleviate these problems, we propose a dynamic noise trimming module and diverse frame re-weighting module for sampling and re-weighting. The dynamic noise trimming module strengthens the dependability of the tracklet representation by removing noisy frames to enhance the clustering accuracy. The diverse frame re-weighting module focuses on training hard frames to enhance the learning of rich information from tracklet. Experiments on three video datasets, <i>i.e.</i> DukeMTMC-VideoReID, MARS and PRID2011, demonstrate the effectiveness of the proposed SRC under the unsupervised re-ID setting.

Decoupled Pose and Similarity Based Graph Neural Network for Video Person Re-Identification

Decoupled Pose and Similarity Based Graph Neural Network for Video Person Re-Identification

Exploiting robust unsupervised video person re‐identification

Unsupervised video person re-identification (reID) methods usually depend on global-level features. Many supervised reID methods employed local-level features and achieved significant performance improvements. However, applying local-level features to unsupervised methods may introduce an unstable performance. To improve the performance stability for unsupervised video reID, this paper introduces a general scheme fusing part models and unsupervised learning. In this scheme, the global-level feature is divided into equal local-level feature. A local-aware module is employed to explore the potentials of local-level feature for unsupervised learning. A global-aware module is proposed to overcome the disadvantages of local-level features. Features from these two modules are fused to form a robust feature representation for each input image. This feature representation has the advantages of local-level feature without suffering from its disadvantages. Comprehensive experiments are conducted on three benchmarks, including PRID2011, iLIDS-VID, and DukeMTMC-VideoReID, and the results demonstrate that the proposed approach achieves state-of-the-art performance. Extensive ablation studies demonstrate the effectiveness and robustness of proposed scheme, local-aware module and global-aware module. The code and generated features are available at https://github.com/deropty/uPMnet.

Video person reidentification based on neural ordinary differential equations and graph convolution network

Person reidentification rate has become a challenging research topic in the field of computer vision due to the fact that person appearance is easily affected by lighting, posture and perspective. In order to make full use of the continuity of video data on the time line and the unstructured relationship of features, a video person reidentification algorithm combining the neural ordinary differential equation with the graph convolution network is proposed in this paper. First, a continuous time model is constructed by using the ordinary differential equation (ODE) network so as to capture hidden information between video frames. By simulating the hidden space of the hidden variables with the hidden time series model, the hidden information between frames that may be ignored in the discrete model can be obtained. Then, the features of the generated video frames are given to the graph convolution network to reconstruct them. Finally, weak supervision is used to classify the features. Experiments on PRID2011 datasets show that the proposed algorithm can significantly improve person reidentification performance.

SSN3D: Self-Separated Network to Align Parts for 3D Convolution in Video Person Re-Identification

Temporal appearance misalignment is a crucial problem in video person re-identification. The same part of person (e.g. head or hand) appearing on different locations in video sequence weakens its discriminative ability, especially when we apply standard temporal aggregation such as 3D convolution or LSTM. To address this issue, we propose Self-Separated network (SSN) to seek out the same parts in different images. As the name implies, SSN, if trained in an unsupervised strategy, guarantees the selected parts distinct. With a few samples of labeled parts to guide SSN training, this semi-supervised trained SSN seeks out the parts that are human-understandable within a frame and stable across a video snippet. Given the distinct and stable person parts, rather than performing aggregation on features, we then apply 3D convolution across different frames for person re-identification. This SSN + 3D pipeline, dubbed SSN3D, is proved to be efficient through extensive experiments on both synthetic and real data.