Video-based Person Re-identification Research Articles

3-D PersonVLAD: Learning Deep Global Representations for Video-Based Person Reidentification.

We present the global deep video representation learning to video-based person reidentification (re-ID) that aggregates local 3-D features across the entire video extent. Existing methods typically extract frame-wise deep features from 2-D convolutional networks (ConvNets) which are pooled temporally to produce the video-level representations. However, 2-D ConvNets lose temporal priors immediately after the convolutions, and a separate temporal pooling is limited in capturing human motion in short sequences. In this paper, we present global video representation learning, to be complementary to 3-D ConvNets as a novel layer to capture the appearance and motion dynamics in full-length videos. Nevertheless, encoding each video frame in its entirety and computing aggregate global representations across all frames is tremendously challenging due to the occlusions and misalignments. To resolve this, our proposed network is further augmented with the 3-D part alignment to learn local features through the soft-attention module. These attended features are statistically aggregated to yield identity-discriminative representations. Our global 3-D features are demonstrated to achieve the state-of-the-art results on three benchmark data sets: MARS, Imagery Library for Intelligent Detection Systems-Video Re-identification, and PRID2011.

Dynamic Graph Co-Matching for Unsupervised Video-based Person Re-Identification.

Cross-camera label estimation from a set of unlabelled training data is an extremely important component in unsupervised person re-identification (re-ID) systems. With the estimated labels, existing advanced supervised learning methods can be leveraged to learn discriminative re-ID models. In this paper, we utilize the graph matching technique for accurate label estimation due to its advantages in optimal global matching and intra-camera relationship mining. However, the graph structure constructed with non-learnt similarity measurement cannot handle the large cross-camera variations, which leads to noisy and inaccurate label outputs. This paper designs a Dynamic Graph Matching (DGM) framework, which improves the label estimation process by iteratively refining the graph structure with better similarity measurement learnt from intermediate estimated labels. In addition, we design a positive re-weighting strategy to refine the intermediate labels, which enhances the robustness against inaccurate matching output and noisy initial training data. To fully utilize the abundant video information and reduce false matchings, a co-matching strategy is further incorporated into the framework. Comprehensive experiments conducted on three video benchmarks demonstrate that DGM outperforms state-of-the-art unsupervised re-ID methods and yields competitive performance to fully supervised upper bounds.

Independent metric learning with aligned multi-part features for video-based person re-identification

Video-based person re-identification attracts wide attention because it plays a crucial role for many applications in the video surveillance. The task of video-based person re-identification is to match image sequences of the pedestrian recorded by non-overlapping cameras. Like many visual recognition problems, variations in pose, viewpoints, illumination, and occlusion make this task non-trivial. Aiming at increasing the robustness of features to variations and occlusion, this paper designs an aligned multi-part image model inspired by human visual attention mechanism. This model performs a pose estimation method to align the pedestrians. Then, it divides the images to extract multi-part appearance features. Besides, we present independent metric learning to combine the multi-part appearance and spatial-temporal features, which obtains several metric kernels by feeding these features into distance metric learning respectively. These kernels are fused with the weights learned by the attention measure. The novel way of features fusion can achieve better functional complementarity of these features. In experiments, we analyze the effectiveness of the major components. Extensive experiments on two public benchmark datasets, i.e., the iLIDS-VID and PRID-2011 datasets, demonstrate the effectiveness of the proposed method.

Joint Attentive Spatial-Temporal Feature Aggregation for Video-Based Person Re-Identification

Video-based person re-identification (Re-ID) remains to be a promising but challenging computer vision task, suffering from a lack of discriminative features that better aggregate both the spatial and temporal information. In this paper, we propose a joint attentive spatial-temporal feature aggregation network (JAFN) for the video-based person Re-ID, simultaneously learning the quality- and frame-aware model to obtain attention-based spatial-temporal feature aggregation. Specifically, we utilize CNN to learn the spatial features, while introducing the LSTM to separately learn the temporal features. For the feature aggregation, we introduce two attention mechanisms respectively for generating the quality and frame significance score, where the quality score measures the quality of the images for attentive spatial feature aggregation, and the frame score measures the significance of the image frames contributing to the temporal feature. Then, we utilize the set-pooling for both the quality-aware spatial feature and the frame-aware temporal feature aggregation based on the attentive scores. The residual learning is also introduced to play between the LSTM and the CNN for adaptive spatial-temporal feature fusion. Furthermore, we adopt the data balance to alleviate the data disproportions existing in datasets of the video-based Re-ID. The extensive experimental results conducted on the PRID2011, i-LIDS-VID, and MARS datasets demonstrate the effectiveness of the proposed JAFN. Furthermore, comparison results conducted on different modules and features in the JAFN show that our approach is of favorable generalization ability on attentively aggregating both the spatial and temporal features.

A Two-Stage Attribute-Constraint Network for Video-Based Person Re-Identification

Person re-identification has gradually become a popular research topic in many fields such as security, criminal investigation, and video analysis. This paper aims to learn a discriminative and robust spatial–temporal representation for video-based person re-identification by a two-stage attribute-constraint network (TSAC-Net). The knowledge of pedestrian attributes can help re-identification tasks because it contains high-level information and is robust to visual variations. In this paper, we manually annotate three video-based person re-identification datasets with four static appearance attributes and one dynamic appearance attribute. Each attribute is regarded as a constraint that is added to the deep network. In the first stage of the TSAC-Net, we solve the re-identification problem as a classification issue and adopt a multi-attribute classification loss to train the CNN model. In the second stage, two LSTM networks are trained under the constraint of identities and dynamic appearance attributes. Therefore, the two-stage network provides a spatial–temporal feature extractor for pedestrians in video sequences. In the testing phase, a spatial–temporal representation can be obtained by inputting a sequence of images to the proposed TSAC-Net. We demonstrate the performance improvement gained with the use of attributes on several challenging person re-identification datasets (PRID2011, iLIDS-VID, MARS, and VIPeR). Moreover, the extensive experiments show that our approach achieves state-of-the-art results on three video-based benchmark datasets.

联合多级深度特征表示和有序加权距离融合的视频行人再识别方法

联合多级深度特征表示和有序加权距离融合的视频行人再识别方法

Fine-Grained Fusion With Distractor Suppression for Video-Based Person Re-Identification

Video based person re-identification aims to associate video clips with the same identity by designing discriminative and representative features. Existing approaches simply compute representations for video clips via frame-level or region-level feature aggregation, where fine-grained local information is inaccessible. To address this issue, we propose a novel module called fine-grained fusion with distractor suppression (short as FFDS) to fully exploit the local features towards better representation of a specific video clip. Concretely, in the proposed FFDS module, the importance of each local feature of an anchor image is calculated by pixel-wise correlation mining with other intra-sequence frames. In this way, 'good' local features co-exist across the video frames are enhanced in the attention map, while sparse 'distractors' can be suppressed. Moreover, to maintain the high-level semantic information of deep CNN features as well as enjoying the fine-grained local information, we adopt the feature mimicking scheme during the training process. Extensive experiments on two challenging large-scale datasets demonstrate effectiveness of the proposed method.

Extended Global–Local Representation Learning for Video Person Re-Identification

Recently, person re-identification has become one of the research hotspots in the field of computer vision and has received extensive attention in the academic community. Inspired by the part-based research of image ReID, this paper presents a novel feature learning and extraction framework for video-based person re-identification, namely, the extended global-local representation learning network (E-GLRN). Given a video sequence of a pedestrian, the holistic and local features are simultaneously extracted using the E-GLRN network. Specifically, for the global feature learning, we adopt the channel attention convolutional neural network (CNN) and the bidirectional long short-term memory (Bi-LSTM) networks, which are responsible for introducing a CNN-LSTM module to learn the features of consecutive frames. The local feature learning module relies on the key local information extraction, which is based on the Bi-LSTM networks. In order to obtain the local feature more effectively, our work defines a concept of “the main image group” by selecting three representative frames. The local feature representation of a video is obtained by exploiting the spatial contextual and appearance information of this group. The local and global features extracted in this paper are complementary and further combined into a discriminative and robust feature representation of the video sequence. Extensive experiments are conducted on three video-based ReID datasets, including the iLIDS-VID, PRID2011 and MARS datasets. The experimental results demonstrate that the proposed method outperforms state-of-the-art video-based re-identification approaches.

K-Reciprocal Harmonious Attention Network for Video-Based Person Re-Identification

Video-based person re-identification aims to retrieve video sequences of the same person in the multi-camera system. In this paper, we propose a k -reciprocal harmonious attention network (KHAN) to jointly learn discriminative spatiotemporal features and the similarity metrics. In KHAN, the harmonious attention module adaptively calibrates response at each spatial position and each channel by explicitly inspecting position-wise and channel-wise interactions over feature maps. Besides, the k-reciprocal attention module attends key features from all frame-level features with a discriminative feature selection algorithm; thus, useful temporal information within contextualized key features can be assimilated to produce more robust clip-level representation. Compared with commonly used local-context based approaches, the proposed KHAN captures long dependency of different spatial regions and visual patterns while incorporating informative context at each time-step in a non-parametric manner. The extensive experiments on three public benchmark datasets show that the performance of our proposed approach outperforms the state-of-the-art methods.

Pose-Guided Spatial Alignment and Key Frame Selection for One-Shot Video-Based Person Re-Identification

One-shot video-based person re-identification exploits the unlabeled data by using a single-labeled sample for each individual to train a model and to reduce the need for laborious labeling. Although recent works focusing on this task have made some achievements, most state-of-the-art models are vulnerable to misalignment, pose variation and corrupted frames. To address these challenges, we propose a one-shot video-based person re-identification model based on pose-guided spatial alignment and KFS. First, a spatial transformer sub-network trained using pose-guided regression is employed to perform the spatial alignment. Second, we propose a novel training strategy based on KFS. Key frames with abruptly changing poses are deliberately identified and selected to make the network adaptive to pose variation. Finally, we propose a frame feature pooling method by incorporating long short-term memory with an attention mechanism to reduce the influence of corrupted frames. Comprehensive experiments are presented based on the MARS and DukeMTMC-VideoReID datasets. The mAP values for these datasets reach 46.5% and 68.4%, respectively, demonstrating that the proposed model achieves significant improvements over state-of-the-art one-shot person re-identification methods.

Spatial-Transformed Regional Quality Estimation Network for Large-Variance Person Re-Identification

The video-based person re-identification aims to search a query video in a large video gallery. The major restrictions on the performance of this task are image misalignment and partial noises caused by detection error, occlusion, blur, and illumination. The misalignment between the different frames in a video caused by excessive background or part missing may play the devil with pedestrian matching. In addition, the influence of partial noises on pedestrian matching performance is also unfavorable. Since different spatial regions of a single frame have various qualities, and the quality of the same region also varies across frames in a tracklet. A good way to address the problem is to effectively aggregate complementary information from all frames in a sequence, using better regions from other frames to compensate for the influence of an image region with poor quality. To achieve this, we propose a novel Spatial-transformed Regional Quality Estimation Network (SRQEN), where a well-designed spatial-transformed unit is used to automatically learn the alignment from an identification procedure and another ingenious training mechanism enables the effective learning to extract the complementary region-based information between different frames. Visual examples indicate that pedestrians are better aligned with SRQEN and the proposed method can learn complementary information. Extensive experiments show that compared with other feature extraction methods, we achieved comparable results of 93.5%, 79.8%, and 74.85% on the PRID 2011, iLIDS-VID, and MARS, respectively.

Video-Based Person Re-Identification Using Unsupervised Tracklet Matching

Despite the significant improvement in accuracy supervised learning has brought into person re-identification (re-id), the availability of sufficient fully annotated data from concerned camera-views poses a problem for real-life applications. To alleviate the burden of intensive data annotation, one way is to resort to unsupervised methods. This has motivated us to propose a novel algorithm for unsupervised video-based person re-id applications. To achieve this, the frames of a person video tracklet are divided into a set of clusters that are subsequently matched using a distance measure based on the Naive Bayes nearest neighbor algorithm and Spearman distance. Knowing that person's sequences may suffer from substantial changes in viewpoint, pose, and illumination distortions, our technique allows the rejection of poor and noisy clusters while retaining the most discriminative ones for matching. Experiments on three widely used datasets for video person re-id PRID2011, iLIDS-VID and MARS have been carried out, and the results demonstrate the superiority of the proposed approach.

Gallery based k-reciprocal-like re-ranking for heavy cross-camera discrepancy in person re-identification

Recently more and more attention is paid to the re-ranking step for person re-identification in computer vision community, especially those fully automatic, unsupervised solutions. Among them, k-reciprocal re-ranking (KR) method achieved big success. However, when the heavy cross-camera discrepancy exists between query and gallery datasets, it may degrade the performance. To alleviate the heavy cross-camera discrepancy between query and gallery datasets, we propose a gallery based k-reciprocal-like re-ranking (GKR) method. GKR adopts graph matching to construct the matching correspondence between query and gallery datasets. Then the proposed k-reciprocal-like neighbors are computed only on gallery dataset instead of on the union of query and gallery datasets like KR does. Moreover, to perform unsupervised video-based person re-identification, we incorporate our proposed GKR method into the dynamic label graph matching (DGM) framework, which can improve the cross-camera labels estimating in training step but also can improve the re-identification accuracy by re-ranking in testing step. Experimental results by supervised and unsupervised solutions on some benchmarks, demonstrate the effectiveness of our GKR method to handle the cross-camera discrepancy problem for person re-identification.

Video-Based Person Re-Identification by an End-To-End Learning Architecture with Hybrid Deep Appearance-Temporal Feature.

Video-based person re-identification is an important task with the challenges of lighting variation, low-resolution images, background clutter, occlusion, and human appearance similarity in the multi-camera visual sensor networks. In this paper, we propose a video-based person re-identification method called the end-to-end learning architecture with hybrid deep appearance-temporal feature. It can learn the appearance features of pivotal frames, the temporal features, and the independent distance metric of different features. This architecture consists of two-stream deep feature structure and two Siamese networks. For the first-stream structure, we propose the Two-branch Appearance Feature (TAF) sub-structure to obtain the appearance information of persons, and used one of the two Siamese networks to learn the similarity of appearance features of a pairwise person. To utilize the temporal information, we designed the second-stream structure that consisting of the Optical flow Temporal Feature (OTF) sub-structure and another Siamese network, to learn the person’s temporal features and the distances of pairwise features. In addition, we select the pivotal frames of video as inputs to the Inception-V3 network on the Two-branch Appearance Feature sub-structure, and employ the salience-learning fusion layer to fuse the learned global and local appearance features. Extensive experimental results on the PRID2011, iLIDS-VID, and Motion Analysis and Re-identification Set (MARS) datasets showed that the respective proposed architectures reached 79%, 59% and 72% at Rank-1 and had advantages over state-of-the-art algorithms. Meanwhile, it also improved the feature representation ability of persons.

Video-Based Person Re-Identification With Accumulative Motion Context

Video based person re-identification plays a central role in realistic security and video surveillance. In this paper we propose a novel Accumulative Motion Context (AMOC) network for addressing this important problem, which effectively exploits the long-range motion context for robustly identifying the same person under challenging conditions. Given a video sequence of the same or different persons, the proposed AMOC network jointly learns appearance representation and motion context from a collection of adjacent frames using a two-stream convolutional architecture. Then AMOC accumulates clues from motion context by recurrent aggregation, allowing effective information flow among adjacent frames and capturing dynamic gist of the persons. The architecture of AMOC is end-to-end trainable and thus motion context can be adapted to complement appearance clues under unfavorable conditions (e.g. occlusions). Extensive experiments are conduced on three public benchmark datasets, i.e., the iLIDS-VID, PRID-2011 and MARS datasets, to investigate the performance of AMOC. The experimental results demonstrate that the proposed AMOC network outperforms state-of-the-arts for video-based re-identification significantly and confirm the advantage of exploiting long-range motion context for video based person re-identification, validating our motivation evidently.

Learning Bidirectional Temporal Cues for Video-Based Person Re-Identification

This paper presents an end-to-end learning architecture for video-based person re-identification by integrating convolutional neural networks (CNNs) and bidirectional recurrent neural networks (BRNNs). Given a video with consecutive frames, features of each frame are extracted with CNN and then are fed into the BRNN to get a final spatio-temporal representation about the video. Specifically, CNN acts as a Spatial Feature Extractor, while BRNN is expected to capture the temporal cues of sequential frames in both forward and backward directions, simultaneously. The whole network is trained end-to-end with a joint identification and verification manner. Experimental results on benchmark data sets show that the proposed model can effectively learn spatio-temporal features relevant for re-identification and outperforms existing video-based person re-identification methods.

Semi-Supervised Cross-View Projection-Based Dictionary Learning for Video-Based Person Re-Identification

Video-based person re-identification (re-id) has attracted a lot of research interest. When facing dramatic growth in new pedestrian videos, existing video-based person re-id methods usually need large quantities of labeled pedestrian videos to train a discriminative model. In practice, labeling large quantities of pedestrian videos is a costly and time-consuming task, which will limit the application of these methods in the real environment. Therefore, it is valuable and necessary to investigate how to learn a discriminative re-id model by using limited labeled training pedestrian videos. In this paper, we propose a semi-supervised cross-view projection-based dictionary learning (SCPDL) approach for video-based person re-id. Specifically, SCPDL jointly learns a pair of feature projection matrices and a pair of dictionaries by integrating the information contained in labeled and unlabeled pedestrian videos. With the learned feature projection matrices, the influence of variations within each video to the re-id can be reduced. With the learned dictionary pair, pedestrian videos from two different cameras can be converted into coding coefficients in a common representation space, such that the differences between different cameras can be bridged. In the learning process, the labeled pedestrian videos are used to ensure that the learned dictionaries have favorable discriminability; the large quantities of unlabeled pedestrian videos are used to ensure that SCPDL can better capture the variations between pedestrian videos, such that the learned dictionaries can own stronger representative capability. Experiments on two public pedestrian sequence data sets (iLIDS-VID and PRID 2011) demonstrate the effectiveness of the proposed approach.

Video-based Person Re-identification by Simultaneously Learning Intra-video and Inter-video Distance Metrics.

Video-based person re-identification (re-id) is an important application in practice. Since large variations exist between different pedestrian videos, as well as within each video, it's challenging to conduct re-identification between pedestrian videos. In this paper, we propose a simultaneous intra-video and inter-video distance learning (SI2DL) approach for video-based person re-id. Specifically, SI2DL simultaneously learns an intravideo distance metric and an inter-video distance metric from the training videos. The intra-video distance metric is used to make each video more compact, and the inter-video one is used to ensure that the distance between truly matching videos is smaller than that between wrong matching videos. Considering that the goal of distance learning is to make truly matching video pairs from different persons be well separated with each other, we also propose a pair separation based SI2DL (P-SI2DL). P-SI2DL aims to learn a pair of distance metrics, under which any two truly matching video pairs can be well separated. Experiments on four public pedestrian image sequence datasets show that our approaches achieve the state-of-the-art performance.

Video Person Re-Identification for Wide Area Tracking Based on Recurrent Neural Networks

In this paper, we propose a video-based person re-identification system for wide area tracking based on a recurrent neural network architecture. Given short video sequences of a person, generated by a tracking algorithm, our video re-identification algorithm links these tracklets in full trajectories across a network of non-overlapping cameras in an open-world scenario. In our system, features are first extracted from each frame using a convolutional neural network. Then, a recurrent layer combines information across time-steps. The features from all time-steps are finally combined using temporal pooling to give an overall appearance feature for the complete sequence. Our system is trained to perform re-identification using a Siamese network architecture. Experiments are conducted on the iLIDS-VID and PRID-2011 video re-identification data sets as well as in the DukeMTMC multi-camera tracking data set.

Low illumination person re-identification

Low illumination is a common problem for recognition and tracking. Low illumination video-based person re identification (re-id) is an important application in practice. Low illumination usually results in severe loss of visual appearance and space-time information contained in pedestrian image or video, which brings large difficulty to re-identification. However, the problem of low illumination video-based person re-id (LIVPR) has not been well studied. In this paper, we propose a novel triplet-based manifold discriminative distance learning (TMD2L) approach for LIVPR. By regarding each video as an image set, TMD2L aims to learn a manifold-based distance metric, under which the intrinsic structure of image sets can be preserved, and the distance between truly matching sets is smaller than that between wrong matching sets. Experiment results on the new collected low illumination person sequence (LIPS) dataset, as well as two simulated datasets LI-PRID 2011 and LI-iLIDS-VID show that our proposed approach TMD2L outperforms existing representative person re-id methods.