Cross-modality Person Re-identification Research Articles

Cross-Modality Person Re-Identification Method with Joint-Modality Generation and Feature Enhancement.

In order to minimize the disparity between visible and infrared modalities and enhance pedestrian feature representation, a cross-modality person re-identification method is proposed, which integrates modality generation and feature enhancement. Specifically, a lightweight network is used for dimension reduction and augmentation of visible images, and intermediate modalities are generated to bridge the gap between visible images and infrared images. The Convolutional Block Attention Module is embedded into the ResNet50 backbone network to selectively emphasize key features sequentially from both channel and spatial dimensions. Additionally, the Gradient Centralization algorithm is introduced into the Stochastic Gradient Descent optimizer to accelerate convergence speed and improve generalization capability of the network model. Experimental results on SYSU-MM01 and RegDB datasets demonstrate that our improved network model achieves significant performance gains, with an increase in Rank-1 accuracy of 7.12% and 6.34%, as well as an improvement in mAP of 4.00% and 6.05%, respectively.

Bridge Gap in Pixel and Feature Level for Cross-Modality Person Re-Identification

Bridge Gap in Pixel and Feature Level for Cross-Modality Person Re-Identification

IGIE-net: Cross-modality person re-identification via intermediate modality image generation and discriminative information enhancement

Given an RGB image (or an IR image) of a pedestrian, the task of cross-modality person re-identification aims to retrieve images of a specific pedestrian from an IR (or RGB) gallery. However, the huge modality discrepancy between RGB and IR images significantly affects the performance of cross-modality person re-identification. For the purpose of reducing the impact of modality discrepancy and extracting more discriminative pedestrian features, a new cross-modality person re-identification network is proposed by us, which is based on intermediate modality image generation and discriminative information enhancement (IGIE-Net). Specifically, we design an intermediate modality image generation module (IMIGM) to effectively weaken the impact of modality discrepancy, which first extracts the modality-specific and modality-shared information contained in RGB and IR images separately, and then generates intermediate RGB and intermediate IR images by adaptively fusing original RGB and original IR images with their corresponding modality-specific and modality-shared information separately. In addition, we also design a discriminative information enhancement module (DIEM) to obtain more discriminative pedestrian representations by enhancing the discriminative information contained in the deep features of pedestrians. Extensive experiments on the publicly available SYSU-MM01 and RegDB datasets indicate that the performance of IGIE-Net reaches the current advanced level.

Homogeneous and Heterogeneous Optimization for Unsupervised Cross-Modality Person Reidentification in Visual Internet of Things

Homogeneous and Heterogeneous Optimization for Unsupervised Cross-Modality Person Reidentification in Visual Internet of Things

Enhanced Invariant Feature Joint Learning via Modality-Invariant Neighbor Relations for Cross-Modality Person Re-Identification

Enhanced Invariant Feature Joint Learning via Modality-Invariant Neighbor Relations for Cross-Modality Person Re-Identification

Cross modality person re-identification via mask-guided dynamic dual-task collaborative learning

Cross modality person re-identification via mask-guided dynamic dual-task collaborative learning

Cross-modality person re-identification based on intermediate modal generation

In the context of cross-modal person re-identification, researchers often employ methods that utilize visible modality information to generate both an ‘X’ modality and a grayscale modality, enhancing the accuracy of person re-identification models. A lightweight network causes the ‘X’ modality through self-supervised learning of labels from visible images. In contrast, the grayscale modality is obtained through simple linear accumulation of the three RGB color channels from visual images. It can be observed that both the ‘X’ modality and grayscale modality are derived from visible images, which fails to establish a connection between the visible and infrared modalities. Therefore, this paper proposes an intermediate modality generation module to produce intermediate modality representations dynamically. By combining information from the visible, infrared, and intermediate modalities, the model is encouraged to capture modality-invariant features with cross-modal consistency. This enables person of the same identity to exhibit similar feature representations across different modalities, thereby mitigating the impact of distribution differences between visible and infrared modalities. Additionally, to facilitate the learning of appropriate intermediate modality representations, a distribution migration strategy is introduced. It guides the intermediate modality to maintain the correct distance from the visible and infrared modalities by optimizing the weights of the loss functions, preventing inadequate feature learning caused by an excessive focus on specific modality information. Furthermore, a mixed augmentation approach is proposed to alleviate disparities among multiple modalities further. By randomly cropping and combining regions of visible (infrared) modality images with infrared (visible) modality images, the generalization ability of the model in heterogeneous modalities is enhanced. Extensive comparative experiments are conducted on the SYSU-MM01 and RegDB datasets, yielding mAP values of 57.2% and 85.82%, respectively. The superior mAP performance on the RegDB dataset compared to most existing methods validates the effectiveness of the proposed approach.

Mind the Gap: Learning Modality-Agnostic Representations With a Cross-Modality UNet.

Cross-modality recognition has many important applications in science, law enforcement and entertainment. Popular methods to bridge the modality gap include reducing the distributional differences of representations of different modalities, learning indistinguishable representations or explicit modality transfer. The first two approaches suffer from the loss of discriminant information while removing the modality-specific variations. The third one heavily relies on the successful modality transfer, could face catastrophic performance drop when explicit modality transfers are not possible or difficult. To tackle this problem, we proposed a compact encoder-decoder neural module (cmUNet) to learn modality-agnostic representations while retaining identity-related information. This is achieved through cross-modality transformation and in-modality reconstruction, enhanced by an adversarial/perceptual loss which encourages indistinguishability of representations in the original sample space. For cross-modality matching, we propose MarrNet where cmUNet is connected to a standard feature extraction network which takes as inputs the modality-agnostic representations and outputs similarity scores for matching. We validated our method on five challenging tasks, namely Raman-infrared spectrum matching, cross-modality person re-identification and heterogeneous (photo-sketch, visible-near infrared and visible-thermal) face recognition, where MarrNet showed superior performance compared to state-of-the-art methods. Furthermore, it is observed that a cross-modality matching method could be biased to extract discriminant information from partial or even wrong regions, due to incompetence of dealing with modality gaps, which subsequently leads to poor generalization. We show that robustness to occlusions can be an indicator of whether a method can well bridge the modality gap. This, to our knowledge, has been largely neglected in the previous works. Our experiments demonstrated that MarrNet exhibited excellent robustness against disguises and occlusions, and outperformed existing methods with a large margin (>10%). The proposed cmUNet is a meta-approach and can be used as a building block for various applications.

A Generative-Based Image Fusion Strategy for Visible-Infrared Person Re-Identification

Cross-modality person re-identification task is a challenging task aiming to recognize images of the same identity between different modalities. To alleviate the cross-modality discrepancies between images, existing approaches mainly guide models to mine modality invariant features. Although those approaches are effective, they lose the modality-specific features that include important information beneficial to VI-ReID. Therefore, some approaches are using generative adversarial networks to compensate for modality information. However, the quality of images generated by these methods is usually poor, and most of them focus only on the learning of modality-sharable features. To solve these problems, this paper proposes a generative-based cross-modality image fusion strategy (GC-IFS), which can generate high-quality cross-modality paired images and fuse the information of the two modalities. Firstly, considering the importance of the identity discriminative information of the generated image, we propose a contrastive-learning image generation (CLIG) network to generate cross-modality paired images. Meanwhile, to fully integrate and utilize the information of the two modalities and eliminate the influence of cross-modality discrepancies, we design a part-based dual multi-modality feature fusion (P-DMFF) module to extract the unified feature representation. Extensive experiments on SYSU-MM01 and RegDB datasets demonstrate that our strategy outperforms the state-of-the-art methods for the VI-ReID task.

Co-segmentation assisted cross-modality person re-identification

We present a deep learning-based method for Visible-Infrared person Re-Identification (VI-ReID). The major contribution lies in the incorporation of co-segmentation into a multi-task learning framework for VI-ReID, where the co-segmentation concept aids in making the distributions of RGB images and IR images the same for the same identity but diverse for different identities. Accordingly, a novel multi-task learning based model, i.e., co-segmentation assisted VI-ReID (CSVI), is proposed in this paper. Specifically, the co-segmentation network first takes as the inputs the modality-shared features extracted from a set of RGB and IR images by using the VI-ReID model. Then, it exploits their semantic similarities for predicting the person masks of the common identities within the input RGB and IR images by using a cross-modality center based weight generation module and a segmentation decoder. Doing so enables the VI-ReID model to extract more additional modality-shared shape features for boosting performance. Meanwhile, the co-segmentation network implicitly establishes the interactions among the set of RGB and IR images, thus further bridging the large modality discrepancies. Our model’s effectiveness and superiority are verified through experimental comparisons with state-of-the-art algorithms on several benchmark datasets.

Parameter sharing and multi-granularity feature learning for cross-modality person re-identification

Visible-infrared person re-identification aims to match pedestrian images between visible and infrared modalities, and its two main challenges are intra-modality differences and cross-modality differences between visible and infrared images. To address these issues, many advanced methods attempt to design new network structures to extract modality-sharing features, mitigate modality differences, or learn part-level features to overcome background interference. However, they ignore the parameter sharing of the convolutional layers to obtain more modality-sharing features. At the same time, only using part-level features lack discriminative pedestrian representations such as body structure and contours. To handle these problems, a parameter sharing and feature learning network is proposed in this paper to mitigate modality differences and further enhance feature discrimination. Firstly, a new two-stream parameter sharing network is proposed, by sharing the convolutional layers parameters to obtain more modality-sharing features. Secondly, a multi-granularity feature learning module is designed to reduce modality differences at both coarse and fine-grained levels while further enhancing feature discriminability. In addition, a center alignment loss is proposed to learn relationships between identities and to reduce modality differences by clustering features into their centers. For the part-level feature learning, the hetero-center triplet loss is adopted to alleviate the strict constraints of triplet loss. Finally, extensive experiments are conducted to validate our method outperforms state-of-the-art methods on two challenging datasets. In the SYSU-MM01 dataset, the Rank-1 and mAP reach 74.0%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$74.0\\%$$\\end{document} and 70.51%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$70.51\\%$$\\end{document} in the all-search mode, which is an increase of 3.4%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$3.4\\%$$\\end{document} and 3.61%\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$3.61\\%$$\\end{document} to baseline, respectively.

MSIF: multi-spectrum image fusion method for cross-modality person re-identification

MSIF: multi-spectrum image fusion method for cross-modality person re-identification

Cross-Modality Person Re-Identification Algorithm Based on Two-Branch Network

Person re-identification is the technique of identifying the same person in different camera shots, known as ReID for short. Most existing models focus on single-modality person re-identification involving only visible images. However, the visible modality is not suitable for low-light environments or at night, when crime is frequent. In contrast, infrared images can reflect the nighttime environment, and most surveillance systems are equipped with dual-mode cameras that can automatically switch between visible and infrared modalities based on light conditions. In contrast to visible-light cameras, infrared (IR) cameras can still capture enough information from the scene in those dark environments. Therefore, the problem of visible-infrared cross-modality person re-identification (VI-ReID) is proposed. To improve the identification rate of cross-modality person re-identification, a cross-modality person re-identification method based on a two-branch network is proposed. Firstly, we use infrared image colorization technology to convert infrared images into color images to reduce the differences between modalities and propose a visible-infrared cross-modality person re-identification algorithm based on Two-Branch Network with Double Constraints (VI-TBNDC), which consists of two main components: a two-branch network for feature extraction and a double-constrained identity loss for feature learning. The two-branch network extracts the features of both data sets separately, and the double-constrained identity loss ensures that the learned feature representations are discriminative enough to distinguish different people from two different patterns. The effectiveness of the proposed method is verified by extensive experimental analysis, and the method achieves good recognition accuracy on the visible-infrared image person re-identification standard dataset SYSU-MM01.

Cross-Modality Person Re-identification with Memory-Based Contrastive Embedding

Visible-infrared person re-identification (VI-ReID) aims to retrieve the person images of the same identity from the RGB to infrared image space, which is very important for real-world surveillance system. In practice, VI-ReID is more challenging due to the heterogeneous modality discrepancy, which further aggravates the challenges of traditional single-modality person ReID problem, i.e., inter-class confusion and intra-class variations. In this paper, we propose an aggregated memory-based cross-modality deep metric learning framework, which benefits from the increasing number of learned modality-aware and modality-agnostic centroid proxies for cluster contrast and mutual information learning. Furthermore, to suppress the modality discrepancy, the proposed cross-modality alignment objective simultaneously utilizes both historical and up-to-date learned cluster proxies for enhanced cross-modality association. Such training mechanism helps to obtain hard positive references through increased diversity of learned cluster proxies, and finally achieves stronger ``pulling close'' effect between cross-modality image features. Extensive experiment results demonstrate the effectiveness of the proposed method, surpassing state-of-the-art works significantly by a large margin on the commonly used VI-ReID datasets.

A Cross-Modality Person Re-Identification Method Based on Joint Middle Modality and Representation Learning

Modality differences and intra-class differences have been hot research problems in the field of cross-modality person re-identification currently. In this paper, we propose a cross-modality person re-identification method based on joint middle modality and representation learning. To reduce the modality differences, a middle modal generator is used to map different modal images to a unified feature space to generate middle modality images. A two-stream network with parameter sharing is used to extract the combined features of the original image and the middle modality image. In addition, a multi-granularity pooling strategy combining global features and local features is used to improve the representation learning capability of the model and further reduce the modality differences. To reduce the intra-class differences, the model is further optimized by combining distribution consistency loss, label smoothing cross-entropy loss, and hetero-center triplet loss to reduce the intra-class distance and accelerate the model convergence. In this paper, we use the publicly available datasets RegDB and SYSU-MM01 for validation. The results show that the proposed approach in this paper reaches 68.11% mAP in All Search mode for the SYSU-MM01 dataset and 86.54% mAP in VtI mode for the RegDB dataset, with a performance improvement of 3.29% and 3.29%, respectively, which demonstrate the effectiveness of the proposed method.

Dynamic feature weakening for cross-modality person re-identification

Cross-modality person re-identification (Re-ID) can be used to perform all-weather pedestrian monitoring tasks by matching the visual features captured in both visible and infrared images. However, the existing mainstream methods mainly rely on a few high response modality-invariant features, leading to problems such as easy loss of person cues and poor algorithm robustness in the recognition process. In this paper, we propose a cross-modality recognition method with dynamic feature weakening (DFW) and flexible segmentation learning as the core mechanisms, specifically: (1) DFW is introduced into the modality-shared network to weaken the model’s over-reliance on high response regions of the image, thereby globally collecting and enriching diverse modality-invariant features. (2) A segmented learning network (SLN) with overlap penalty constraint is used for fine-grained mining of each modality-invariant feature, while avoiding redundant focus on a region and maintaining complete learning of diversity information. The experimental results show that the method in this paper greatly enriches the feature representation, and significantly improves the recognition accuracy of the model. On the SYSU-MM01 dataset, compared with the traditional modality feature learning method, the Rank1 and mAP of this method are increased by 2.56% and 4.53%, respectively.

Cross-Modality Person Re-Identification via Local Paired Graph Attention Network.

Cross-modality person re-identification (ReID) aims at searching a pedestrian image of RGB modality from infrared (IR) pedestrian images and vice versa. Recently, some approaches have constructed a graph to learn the relevance of pedestrian images of distinct modalities to narrow the gap between IR modality and RGB modality, but they omit the correlation between IR image and RGB image pairs. In this paper, we propose a novel graph model called Local Paired Graph Attention Network (LPGAT). It uses the paired local features of pedestrian images from different modalities to build the nodes of the graph. For accurate propagation of information among the nodes of the graph, we propose a contextual attention coefficient that leverages distance information to regulate the process of updating the nodes of the graph. Furthermore, we put forward Cross-Center Contrastive Learning (C3L) to constrain how far local features are from their heterogeneous centers, which is beneficial for learning the completed distance metric. We conduct experiments on the RegDB and SYSU-MM01 datasets to validate the feasibility of the proposed approach.

Counterfactual attention alignment for visible-infrared cross-modality person re-identification

Visible-infrared person re-identification (VI-ReID) copes with cross-modality matching between the daytime visible and night-time infrared images. Existing methods try to use attention modules to enhance multi-modality feature representations, but ignore measures of attention quality and lack direct and effective supervision of the attention learning process. To solve these problems, we propose a counterfactual attention alignment (CAA) strategy by mining intra-modality attention information with counterfactual causality and aligning the cross-modality attentions. Specifically, a self-weighted part attention module is designed to extract the pairwise attention information in local parts. The counterfactual attention alignment strategy obtains the learning results of the attention module through counterfactual intervention, and aligns the attention maps of the two modalities to find better shared cross-modality attention regions. Then the effect of the aligned attention on network prediction is used as a supervision signal to directly guide the attention module to learn more effective attention information. Extensive experimental results demonstrate that the proposed approach outperforms other state-of-the-art methods on two standard benchmarks.

Cross-Modality Transformer With Modality Mining for Visible-Infrared Person Re-Identification

Visible-infrared cross-modality person re-identification is a challenging ReID task, which aims to retrieve and match the same identity's images between the heterogeneous visible and infrared modalities. Thus, the core of this task is to bridge the huge gap between these two modalities. The existing convolutional neural network-based methods mainly face the problem of insufficient perception of modalities' information, and can not learn good discriminative modality-invariant embeddings for identities, which limits their performance. To solve these problems, we propose a cross-modality transformer-based method (CMTR) for the visible-infrared person re-identification task, which can explicitly mine the information of each modality and generate better discriminative features based on it. Specifically, to capture modalities' characteristics, we design the novel modality embeddings, which are fused with token embeddings to encode modalities' information. Furthermore, to enhance representation of modality embeddings and adjust matching embeddings' distribution, we propose a modality-aware enhancement loss based on the learned modalities' information, reducing intra-class distance and enlarging inter-class distance. To our knowledge, this is the first work of applying transformer network to the cross-modality re-identification task. We implement extensive experiments on the public SYSU-MM01 and RegDB datasets, and our proposed CMTR model's performance significantly surpasses existing outstanding CNN-based methods.

Person Tracking by Detection Using Dual Visible-Infrared Cameras

We study the problem of cross-modality person reidentification (ReID) and tracking with dual visible-infrared (VI) cameras, while most existing efforts on tracking-by-detection have been paid on single-modality visible ReID which is inapplicable for poor-light environments. The major difficulties for cross-modality (e.g., VI) ReID stem from the large modality gap between three-channel visible images and one-channel infrared images and such unknown environmental factors as background clutter, occlusions, etc. To tackle these issues, we propose to enrich the diversities of visible and infrared images for intra- and cross-modality matching by using both the channel-aware data augmentation (DA) techniques (e.g., channel exchanged augmentation and random occlusions) and standard DA techniques. On top of these DA techniques, we incorporate ResNet50 and vision transformer (ViT) into the feature extraction backbone network and apply the dynamic weight average (DWA) strategy for learning loss weights by regarding the minimization of identity loss and triplet loss as a multitask learning problem. We then apply the proposed ReID approach for person tracking in the field of interests. The experiments on two public data sets, i.e., RegDB and SYSU-MM01, show that our approach can improve the performance of state-of-the-art rank-1, mAP, and mINP for cross-modality matching. In addition, the experiments on our data set show that tracking by VI-ReID using dual VI cameras can achieve an accuracy of around 0.24 m.