Video-based Re-identification Research Articles

Deep video-based person re-identification (Deep Vid-ReID): comprehensive survey

Person re-identification (ReID) aims to find the person of interest across multiple non-overlapping cameras. It is considered an essential step for person tracking applications which is vital for surveillance. Person ReID could be investigated either using image-based or video-based. Video-based person ReID is considered more discriminating and realistic than image-based ReID due to the massive information extracted for each person. Different deep-learning techniques have been used for video-based ReID. In this survey, recently published articles are reviewed according to video-based ReID system pipeline: deep features learning, deep metric learning, and deep learning approaches. The deep feature learning approaches are categorized into spatial and temporal approaches, while deep metric learning is divided into metric and metric learning approaches. The deep learning approaches are differentiated into: supervised, unsupervised, weakly-supervised, and one-shot learning. A detailed analysis is held for the architectures of the state-of-the-art deep learning approaches. And their performance on four benchmark datasets is compared.

Multi-Granularity Aggregation with Spatiotemporal Consistency for Video-Based Person Re-Identification.

Video-based person re-identification (ReID) aims to exploit relevant features from spatial and temporal knowledge. Widely used methods include the part- and attention-based approaches for suppressing irrelevant spatial-temporal features. However, it is still challenging to overcome inconsistencies across video frames due to occlusion and imperfect detection. These mismatches make temporal processing ineffective and create an imbalance of crucial spatial information. To address these problems, we propose the Spatiotemporal Multi-Granularity Aggregation (ST-MGA) method, which is specifically designed to accumulate relevant features with spatiotemporally consistent cues. The proposed framework consists of three main stages: extraction, which extracts spatiotemporally consistent partial information; augmentation, which augments the partial information with different granularity levels; and aggregation, which effectively aggregates the augmented spatiotemporal information. We first introduce the consistent part-attention (CPA) module, which extracts spatiotemporally consistent and well-aligned attentive parts. Sub-parts derived from CPA provide temporally consistent semantic information, solving misalignment problems in videos due to occlusion or inaccurate detection, and maximize the efficiency of aggregation through uniform partial information. To enhance the diversity of spatial and temporal cues, we introduce the Multi-Attention Part Augmentation (MA-PA) block, which incorporates fine parts at various granular levels, and the Long-/Short-term Temporal Augmentation (LS-TA) block, designed to capture both long- and short-term temporal relations. Using densely separated part cues, ST-MGA fully exploits and aggregates the spatiotemporal multi-granular patterns by comparing relations between parts and scales. In the experiments, the proposed ST-MGA renders state-of-the-art performance on several video-based ReID benchmarks (i.e., MARS, DukeMTMC-VideoReID, and LS-VID).

AA-RGTCN: reciprocal global temporal convolution network with adaptive alignment for video-based person re-identification.

Person re-identification(Re-ID) aims to retrieve pedestrians under different cameras. Compared with image-based Re-ID, video-based Re-ID extracts features from video sequences that contain both spatial features and temporal features. Existing methods usually focus on the most attractive image parts, and this will lead to redundant spatial description and insufficient temporal description. Other methods that take temporal clues into consideration usually ignore misalignment between frames and only focus on a fixed length of one given sequence. In this study, we proposed a Reciprocal Global Temporal Convolution Network with Adaptive Alignment(AA-RGTCN). The structure could address the drawback of misalignment between frames and model discriminative temporal representation. Specifically, the Adaptive Alignment block is designed to shift each frame adaptively to its best position for temporal modeling. Then, we proposed the Reciprocal Global Temporal Convolution Network to model robust temporal features across different time intervals along both normal and inverted time order. The experimental results show that our AA-RGTCN can achieve 85.9% mAP and 91.0% Rank-1 on MARS, 90.6% Rank-1 on iLIDS-VID, and 96.6% Rank-1 on PRID-2011, indicating we could gain better performance than other state-of-the-art approaches.

Rethink Motion Information for Occluded Person Re-Identification

Person re-identification aims to identify the same pedestrians captured by various cameras from different viewpoints in multiple scenarios. Occlusion is the toughest problem for practical applications. In video-based ReID tasks, motion information can be easily obtained from sampled frames, and provide discriminative human part representations. However, most motion-based methodologies are designed for video frames which are not suitable for processing single static image input. In this paper, we propose a Motion-Aware Fusion (MAF) network, aiming to acquire motion information from static images in order to improve the performance of ReID tasks. Specifically, a visual adapter is introduced to enable visual feature extraction, either from image or video data. We design a motion consistency task to guide the motion-aware transformer to learn representative human-part motion information and greatly improve the learning quality of features of occluded pedestrians. Extensive experiments on popular holistic, occluded, and video datasets demonstrate the effectiveness of our proposed method. This method outperforms state-of-the-art approaches by improving the mean average precision (mAP) by 1.5% and rank-1 accuracy by 1.2% on the challenging Occluded-REID dataset. At the same time, it surpasses other methods on the MARS dataset with an improvement of 0.2% in mAP and 0.1% in rank-1 accuracy.

Pose-Aided Video-Based Person Re-Identification via Recurrent Graph Convolutional Network

Existing methods for video-based person re-identification (ReID) mainly learn the appearance feature of a given pedestrian via a feature extractor and a feature aggregator. However, the appearance models would fail to learn a large inter-class variance when different pedestrians have similar appearances. Considering that different pedestrians have different walking postures and body proportions, we propose to learn the discriminative pose feature beyond the appearance feature for video retrieval. Specifically, we implement a two-branch architecture to separately learn the appearance feature and pose feature, and then concatenate them together for inference. To learn the pose feature, we first detect the pedestrian pose in each frame through an off-the-shelf pose detector, and construct a temporal graph using the pose sequence. We then exploit a recurrent graph convolutional network (RGCN) to learn the node embeddings of the temporal pose graph, which devises a global information propagation mechanism to simultaneously achieve the neighborhood aggregation of intra-frame nodes and message passing among inter-frame graphs. Finally, we propose a dual-attention method (DAM) consisting of node-attention and time-attention to obtain the temporal graph representation from the node embeddings, where the self-attention mechanism is employed to learn the importance of each node and each frame. We verify the proposed method on three video-based ReID datasets, i.e., Mars, DukeMTMC and iLIDS-VID, whose experimental results demonstrate that the learned pose feature can effectively improve the performance of existing appearance models.

Multi-Context Grouped Attention for Unsupervised Person Re-Identification

Recent advancements like multiple contextual analysis, attention mechanisms, distance-aware optimization, and multi-task guidance have been widely used for supervised person re-identification (ReID), but the implementation and effects of such methods in unsupervised ReID frameworks are non-trivial and unclear, respectively. Moreover, with increasing size and complexity of image-and video-based ReID datasets, manual or semi-automated annotation procedures for supervised ReID are becoming labor intensive and cost prohibitive, which is undesirable especially considering the likelihood of annotation errors increase with scale/complexity of data collections. Therefore, we propose a new iterative clustering framework that is insensitive to annotation errors and over-fitting ReID annotations (i.e., labels). Our proposed unsupervised framework incorporates (a) a novel multi-context group attention architecture that learns a holistic attention map from multiple local and global contexts, (b) an unsupervised clustering loss function that down-weights easily discriminative identities, and (c) a background diversity term that helps cluster persons across different cross-camera views without leveraging any identification or camera labels. We perform extensive analysis using the DukeMTMC-VideoReID and MARS video-based ReID datasets and the MSMT17 image-based ReID dataset. Our approach is shown to provide a new state-of-the-art performance for unsupervised ReID, reducing the rank-1 performance gap between supervised and unsupervised ReID to 1.1%, 12.1%, and 21.9% from 6.1%, 17.9%, and 22.6% for DukeMTMC, MARS, and MSMT17 datasets, respectively.

PolarBearVidID: A Video-Based Re-Identification Benchmark Dataset for Polar Bears

Automated monitoring systems have become increasingly important for zoological institutions in the study of their animals' behavior. One crucial processing step for such a system is the re-identification of individuals when using multiple cameras. Deep learning approaches have become the standard methodology for this task. Especially video-based methods promise to achieve a good performance in re-identification, as they can leverage the movement of an animal as an additional feature. This is especially important for applications in zoos, where one has to overcome specific challenges such as changing lighting conditions, occlusions or low image resolutions. However, large amounts of labeled data are needed to train such a deep learning model. We provide an extensively annotated dataset including 13 individual polar bears shown in 1431 sequences, which is an equivalent of 138,363 images. PolarBearVidID is the first video-based re-identification dataset for a non-human species to date. Unlike typical human benchmark re-identification datasets, the polar bears were filmed in a range of unconstrained poses and lighting conditions. Additionally, a video-based re-identification approach is trained and tested on this dataset. The results show that the animals can be identified with a rank-1 accuracy of 96.6%. We thereby show that the movement of individual animals is a characteristic feature and it can be utilized for re-identification.

An Adaptive Partitioning and Multi-Granularity Network for Video-Based Person Re-Identification

Person re-identification (Re-ID) is a key technology used in the field of intelligent surveillance. The existing Re-ID methods are mainly realized by using convolutional neural networks (CNNs), but the feature information is easily lost in the operation process due to the down-sampling structure design in CNNs. Moreover, CNNs can only process one local neighbourhood at a time, which makes the global perception of the network poor. To overcome these shortcomings, in this study, we apply a pure transformer to a video-based Re-ID task by proposing an adaptive partitioning and multi-granularity (APMG) network framework. To enable the pure transformer structure better at adapting to the Re-ID task, we propose a new correlation-adaptive partitioning (CAP) of feature embedding modules that can adaptively partition person images according to structural correlations and thus retain the structure and semantics of local feature information in the images. To improve the Re-ID performance of the network, we also propose a multi-granularity (MG) module to better capture people feature information at different levels of granularity. We performed validation trials on three video-based benchmark datasets. The results show that the network structure based on the pure transformer can adapt to Re-ID tasks well, and our APMG network outperforms other state-of-the-art methods.

STCA: Utilizing a spatio-temporal cross-attention network for enhancing video person re-identification

Video-based re-identification (ReID) is a crucial task in computer vision that draws increasing attention due to advances in deep learning (DL) and modern computational devices. Despite recent success with CNN architectures, single models (e.g., 2D-CNNs or 3D-CNNs) alone failed to leverage temporal information with spatial cues. This is due to uncontrolled surveillance scenarios and variable poses leading to inevitable misalignment of ROIs across the tracklets, which is accompanied by occlusion and motion blur. In this context, designing temporal and spatial cues for two different models and their combinations can be beneficial, considering the global of a video-tracklet. 3D-CNNs allow encoding of temporal information while 2D-CNNs extract spatial or appearance information. In this paper, we propose a Spatio-Temporal Cross Attention (STCA) network to utilize both 2D-CNNs and 3D-CNNs that calculate the cross attention mapping both from the layer of 3D-CNNs and 2D-CNNs along a person's trajectory to gate the following layers of 2D-CNNs; and highlight relevant appearance features for the person ReID. Given an input tracklet, the proposed cross attention (CA) is able to capture the salient regions that propagate throughout the tracklet to obtain the global view. This provides a spatio-temporal attention approach that can be dynamically aggregated with spatial features of 2D-CNNs to perform finer-grained recognition. Additionally, we exploit the advantage of utilizing cosine similarity while triplet sampling as well as for calculating the final recognition score. Experimental analyses on three challenging benchmark datasets indicate that integrating spatio-temporal cross attention into the state-of-the-art video ReID backbone CNN architecture allows for improving their recognition accuracy.

Relation-based global-partial feature learning network for video-based person re-identification

Video-based person re-identification (Re-ID) aims to match the same pedestrian from the video sequences captured by non-overlapping cameras. It is the key to fully extracting abundant spatial and temporal information from the video frames in video-based Re-ID. In this paper, a novel Relation-Based Global-Partial Feature Learning framework is proposed to explore discriminative spatiotemporal features with the help of the global and partial relationship between frames. Specifically, we propose a Relation-Based Global Feature Learning Module (RGL) to obtain global references for generating features correlation maps between frames in the video sequence and determine the importance of frame-level features. As the supplementary of the global relation-based features, a Relation-Based Partial Feature Learning Module (RPL) is also proposed to obtain the relationship between partial features of the same spatial position in different frames to enhance the frame-level partial representation. Moreover, we design a multi-level training scheme to deeply supervise our model. Extensive experiments are conducted on three public video-based person Re-ID datasets, and the results indicate that our framework achieves state-of-the-art performance on three benchmarks.

Motion Feature Aggregation for Video-Based Person Re-Identification.

Most video-based person re-identification (re-id) methods only focus on appearance features but neglect motion features. In fact, motion features can help to distinguish the target persons that are hard to be identified only by appearance features. However, most existing temporal information modeling methods cannot extract motion features effectively or efficiently for v ideo-based re-id. In this paper, we propose a more efficient Motion Feature Aggregation (MFA) method to model and aggregate motion information in the feature map level for video-based re-id. The proposed MFA consists of (i) a coarse-grained motion learning module, which extracts coarse-grained motion features based on the position changes of body parts over time, and (ii) a fine-grained motion learning module, which extracts fine-grained motion features based on the appearance changes of body parts over time. These two modules can model motion information from different granularities and are complementary to each other. It is easy to combine the proposed method with existing network architectures for end-to-end training. Extensive experiments on four widely used datasets demonstrate that the motion features extracted by MFA are crucial complements to appearance features for video-based re-id, especially for the scenario with large appearance changes. Besides, the results on LS-VID, the current largest publicly available video-based re-id dataset, surpass the state-of-the-art methods by a large margin. The code is available at: https://github.com/guxinqian/Simple-ReID.

Temporal Weighting Appearance-Aligned Network for Nighttime Video Retrieval

Video-based person re-identification (ReID) aims at re-identifying video sequences of a specified person from videos captured by disjoint cameras. Existing datasets and works on this task all focus on daytime scenarios and cannot adapt well to the nighttime scenarios, which is also of significant importance for practical applications. In this paper, we contribute a new dataset for nighttime video-based ReID, termed NIVIR, which contains 800 identities with over 228,000 images. NIVIR contains video shots under various lighting conditions, different weathers, and complex scenarios, which is consistent with the real nighttime outdoor surveillance. Furthermore, we propose a temporal weighting appearance-aligned network (TWAN) for nighttime video-based ReID, which is composed of a correlation-based appearance-aligned module (CAM) and a temporal weighting module (TWM). Specifically, CAM is proposed to reconstruct the adjacent feature maps to guarantee the appearance alignment between the central frame and its adjacent frames. TWM is designed to evaluate the frame quality of a tracklet and generate temporal weights to enhance the video representation. Extensive experiments conducted on our new NIVIR dataset demonstrate that the proposed TWAN outperforms the state-of-the-art methods. We believe that our NIVIR dataset and the comprehensive attempts for solving the nighttime ReID problem will push forward the development of the ReID research community.

Spatial temporal and channel aware network for video-based person re-identification

As a challenging computer vision task, video-based person Re-IDentification (Re-ID) has been intensively studied, and recent works have achieved a series of satisfactory results by capturing spatial temporal relationships. However, extensive observations have found that the same feature vector generated by a convolutional neural network contains considerable redundant information in the channel dimension. This issue is seldom investigated. A Spatial Temporal and Channel Aware Network (STCAN) for video-based ReID is studied in this paper. It jointly considers spatial temporal and channel information. Firstly, the Spatial Attention Enhanced (SAE) convolutional network is developed as the backbone network to learn spatial enhanced features from video frames. Secondly, a Channel Segmentation and Group Shuffle (CSGS) convolution module is designed to jointly address temporal and channel relations. Finally, a Two Branch Weighted Fusion (TBWF) mechanism is introduced to enhance the robustness of the Re-ID network by fusing the output of the SAE backbone network and CSGS. Comprehensive experiments are conducted on three large-scale datasets MARS, LSVID, and P-DESTRE. The experimental results imply that the STCAN can effectively improve the performance of video-based Re-ID and outperform several state-of-the-art methods.

Exploiting Global Camera Network Constraints for Unsupervised Video Person Re-Identification

Many unsupervised approaches have been proposed recently for the video-based re-identification problem since annotations of samples across cameras are time-consuming. However, higher-order relationships across the entire camera network are ignored by these methods, leading to contradictory outputs when matching results from different camera pairs are combined. In this paper, we address the problem of unsupervised video-based re-identification by proposing a consistent cross-view matching (CCM) framework, in which global camera network constraints are exploited to guarantee the matched pairs are with consistency. Specifically, we first propose to utilize the first neighbor of each sample to discover relations among samples and find the groups in each camera. Additionally, a cross-view matching strategy followed by global camera network constraints is proposed to explore the matching relationships across the entire camera network. Finally, we learn metric models for camera pairs progressively by alternatively mining consistent cross-view matching pairs and updating metric models using these obtained matches. Rigorous experiments on two widely-used benchmarks for video re-identification demonstrate the superiority of the proposed method over current state-of-the-art unsupervised methods; for example, on the MARS dataset, our method achieves an improvement of 4.2\% over unsupervised methods, and even 2.5\% over one-shot supervision-based methods for rank-1 accuracy.

Rethinking Temporal Fusion for Video-Based Person Re-Identification on Semantic and Time Aspect

Recently, the research interest of person re-identification (ReID) has gradually turned to video-based methods, which acquire a person representation by aggregating frame features of an entire video. However, existing video-based ReID methods do not consider the semantic difference brought by the outputs of different network stages, which potentially compromises the information richness of the person features. Furthermore, traditional methods ignore important relationship among frames, which causes information redundancy in fusion along the time axis. To address these issues, we propose a novel general temporal fusion framework to aggregate frame features on both semantic aspect and time aspect. As for the semantic aspect, a multi-stage fusion network is explored to fuse richer frame features at multiple semantic levels, which can effectively reduce the information loss caused by the traditional single-stage fusion. While, for the time axis, the existing intra-frame attention method is improved by adding a novel inter-frame attention module, which effectively reduces the information redundancy in temporal fusion by taking the relationship among frames into consideration. The experimental results show that our approach can effectively improve the video-based re-identification accuracy, achieving the state-of-the-art performance.

Instance Hard Triplet Loss for In-video Person Re-identification

Traditional Person Re-identification (ReID) methods mainly focus on cross-camera scenarios, while identifying a person in the same video/camera from adjacent subsequent frames is also an important question, for example, in human tracking and pose tracking. We try to address this unexplored in-video ReID problem with a new large-scale video-based ReID dataset called PoseTrack-ReID with full images available and a new network structure called ReID-Head, which can extract multi-person features efficiently in real time and can be integrated with both one-stage and two-stage human or pose detectors. A new loss function is also required to solve this new in-video problem. Hence, a triplet-based loss function with an online hard example mining designed to distinguish persons in the same video/group is proposed, called instance hard triplet loss, which can be applied in both cross-camera ReID and in-video ReID. Compared with the widely-used batch hard triplet loss, our proposed loss achieves competitive performance and saves more than 30% of the training time. We also propose an automatic reciprocal identity association method, so we can train our model in an unsupervised way, which further extends the potential applications of in-video ReID. The PoseTrack-ReID dataset and code will be publicly released.

Comprehensive feature fusion mechanism for video-based person re-identification via significance-aware attention

Video-based person re-identification (Re-ID) is of important capability for artificial intelligence and human–computer interaction. The spatial and temporal features play indispensable roles to comprehensively represent the person sequences. In this paper, we propose a comprehensive feature fusion mechanism (CFFM) for video-based Re-ID. We use multiple significance-aware attention to learn attention-based spatial–temporal feature fusion to better represent the person sequences. Specifically, CFFM consists of spatial attention, periodic attention, significance attention and residual learning. The spatial attention and periodic attention aim to respectively make the system focus on more useful spatial feature extracted by CNN and temporal feature extracted by the recurrent networks. The significance attention is to measure the two features that contribute to the sequence representation. Then the residual learning plays between the spatial and temporal features based on the significance scores for final significance-aware feature fusion. We apply our approach to different representative state-of-the-art networks, proposing several improved networks for improving the video-based Re-ID task. We conduct extensive experimental results on the widely utilized datasets, PRID-2011, i-LIDS-VID and MARS, for the video-based Re-ID task. Results show that the improved networks perform favorably against existing approaches, demonstrating the effectiveness of our proposed CFFM for comprehensive feature fusion. Furthermore, we compare the performance of different modules in CFFM, investigating the varied significance of the different networks, features and sequential feature aggregation modes.

Video Based Person Re Identification Methods, Datasets, and Deep Learning

Video Analytics applications like security and surveillance face a critical problem of person re-identification abbreviated as re-ID. The last decade witnessed the emergence of large-scale datasets and deep learning methods to use these huge data volumes. Most current re-ID methods are classified into either image-based or video-based re-ID. Matching persons across multiple camera views have attracted lots of recent research attention. Feature representation and metric learning are major issues for person re-identification. The focus of re-ID work is now shifting towards developing end-to-end re-Id and tracking systems for practical use with dynamic datasets. Most previous works contributed to the significant progress of person re-identification on still images using image retrieval models. This survey considers the more informative and challenging video-based person re-ID problem, pedestrian re-ID in particular. Publicly available datasets and codes are listed as a part of this work. Current trends which include open re-identification systems, use of discriminative features and deep learning is marching towards new applications in security and surveillance, typically for tracking.

Video-based person re-identification using a novel feature extraction and fusion technique

Person re-identification has received extensive attention in the academic community. In this paper, a novel multiple feature fusion network (MPFF-Net) is proposed for video-based person re-identification. The proposed network is used to obtain the robust and discriminative feature representation for describing the pedestrian in the video, which contains the hand-crafted and deep-learned parts. First, the image-level features of all consecutive frames are extracted. Then the hand-crafted branch uses these descriptors to obtain the average feature of the video and the information of frame-to-frame differences. The deep-learned branch is based on the bidirectional LSTM (BiLSTM) network. It is responsible for aggregating frame-wise representations of human regions and yielding sequence-level features. Furthermore, the problem of misalignment is taken into account in this branch. Finally, the hand-crafted and deep-learned parts are considered to be complementary, and the fusion of them can help to capture the complete information of the video. Extensive experiments are conducted on the iLIDS-VID, PRID2011 and MARS datasets. The results demonstrate that the proposed algorithm outperforms state-of-the-art video-based re-identification methods.

Weighted triple-sequence loss for video-based person re-identification

Person re-identification (re-id) task has attracted a lot of attention because of the excellent performance powered by the Convolutional Neural Network (CNN). However, video-based person re-id is still challenging and far to be solved. On the one hand, the sequence contains complementary information but also more noise information. On the other hand, the training for video is still mainly based on classifying a single frame via its person identity. The representation of the video is generated by aggregating each frame feature. All these will cause that the robustness of video feature is not adequate at the training stage, and the model is easy to be misled by noise information existing in frames. In order to alleviate the difficulty of training video-based re-id, we propose a novel loss named Weighted Triple-Sequence Loss (WTSL) to optimize the video-based feature and reduce the impact of outliers. Further more, we design a Spatial Transformed Partial Network (STPN) coordinated with jointly optimizing image-level and video-level features to generate more robust representation. Extensive experiments show that our algorithm outperforms the state-of-the-art results and achieves 82.2%, 95.2%, and 85.9% rank-1 accuracy on three popular video-based benchmarks: iLIDS-VID, PRID2011, and MARS, respectively.