Cross-modality Discrepancy Research Articles

Dual-granularity feature alignment for cross-modality person re-identification

RGB-infrared cross-modality person re-identification is a challenging image retrieval task due to the intra-class variations and cross-modality discrepancy. However, it is insufficient for existing methods to identify discriminative patterns with different granularities and align them across granularity and modalities in a synergistic manner. In this paper, we propose Dual-granularity Feature Alignment (DFA) approach for cross-modality re-ID, which accommodates dual-granularity feature extraction and cross-granularity feature alignment. More specifically, a feature extractor is introduced to generate discriminative patterns by dividing feature map into different granularities. Based on the above features, prototype learning is developed to assign different granularity features of the same identity to the same prototype and adaptively build the alignment across different granularities. To mine contextual relationships between training samples, a similarity inference is presented to enforce the similarity matrix closing to identity matrix. This allows model to find feature that conserves as much modality-shared information as possible while being least possible modality-specific information. Both modules work in a collaborative way to obtain the optimal intra-class compactness and inter-class separability for cross-modality image matching. Extensive experiments on several benchmarks demonstrate that our DAF surpasses the state-of-the-art methods. The code is available at https://github.com/PRIS-CV/DFA.

Visible–Infrared Person Re-Identification via Global Feature Constraints Led by Local Features

Smart security is needed for complex scenarios such as all-weather and multi-scene environments, and visible–infrared person re-identification (VI Re-ID) has become a key technique in this field. VI Re-ID is usually modeled as a pattern recognition issue, which faces the problems of inter-modality and intra-modality discrepancies. To alleviate these problems, we designed the Local Features Leading Global Features Network (LoLeG-Net), a representation learning network. Specifically, for cross-modality discrepancies, we employed a combination of ResNet50 and non-local attention blocks to obtain the modality-shareable features and convert the problem to a single-modality person re-identification (Re-ID) problem. For intra-modality variations, we designed global feature constraints led by local features. In this method, the identity loss and hetero-center loss were employed to alleviate intra-modality variations of local features. Additionally, hard sample mining triplet loss combined with identity loss was used, ensuring the effectiveness of global features. With this method, the final extracted global features were much more robust against the background environment, pose differences, occlusion and other noise. The experiments demonstrate that LoLeG-Net is superior to existing works. The result for SYSU-MM01 was Rank-1/mAP 51.40%/51.41%, and the result for RegDB was Rank-1/mAP 76.58%/73.36%.

Part-facial relational and modality-style attention networks for heterogeneous face recognition

The purpose of heterogeneous face recognition (HFR) is to match face images of the same person from different modalities. Most HFR methods bridge the cross-modality variations with feature alignment by global feature representation learning, but ignore the content information of local features and modality-style information of face image for each modality, which limits the performance for HFR. The content information of local features not only contains the invariance of modality face features, but also can improve the stability of global face features, i.e., local features such as eyes, nose and mouth are steady and invariant. With this motivation, we propose a cross-modality dual-constraint (CMDC) approach that includes the part-facial relational attention network (PRAN) and modality-style attention network (MSAN). First, PRAN is designed to estimate the intrinsic structural relationships of local content features on each modality. It can extract discriminative local face features by capturing correlations within the face space of individual modality, and strengthen representations by contextual relationships across modalities. Secondly, we design the MSAN to capture the modality-style information for each modality, and then reduce the inter-modality differences by minimizing the distance of two modality-style features. Thirdly, to alleviate cross-modality variances and enhance intra-class compactness and inter-class divisibility, we propose the cross-modality dual-constrained loss (DCLoss) in the CMDC approach, which adds a global constraint to each sample distribution in the embedding space. Meanwhile, on the basis of focusing on modality-style information, DCLoss emphasizes the significance of category information. Extensive experiments on four datasets demonstrate the superior performance of our approach over the existing state-of-the-art. The code is available at https://github.com/JianYu777/CMDC.

Visible–infrared person re-identification based on key-point feature extraction and optimization

Feature extraction for visible–infrared person re-identification (VI-ReID) is challenging because of the cross-modality discrepancy in the images taken by different spectral cameras. Most of the existing VI-ReID methods often ignore the potential relationship between features. In this paper, we intend to transform low-order person features into high-order graph features, and make full use of the hidden information between person features. Therefore, we propose a multi-hop attention graph convolution network (MAGC) to extract robust person joint feature information using residual attention mechanism while reducing the impact of environmental noise. The transfer of higher order graph features within MAGC enables the network to learn the hidden relationship between features. We also introduce the self-attention semantic perception layer (SSPL) which can adaptively select more discriminant features to further promote the transmission of useful information. The experiments on VI-ReID datasets demonstrate its effectiveness.

Visible-Infrared Person Re-Identification via Partially Interactive Collaboration.

Visible-infrared person re-identification (VI-ReID) task aims to retrieve the same person between visible and infrared images. VI-ReID is challenging as the images captured by different spectra present large cross-modality discrepancy. Many methods adopt a two-stream network and design additional constraint conditions to extract shared features for different modalities. However, the interaction between the feature extraction processes of different modalities is rarely considered. In this paper, a partially interactive collaboration method is proposed to exploit the complementary information of different modalities to reduce the modality gap for VI-ReID. Specifically, the proposed method is achieved in a partially interactive-shared architecture: collaborative shallow layers and shared deep layers. The collaborative shallow layers consider the interaction between modality-specific features of different modalities, encouraging the feature extraction processes of different modalities constrain each other to enhance feature representations. The shared deep layers further embed the modality-specific features to a common space to endow them the same identity discriminability. To ensure the interactive collaborative learning implement effectively, the conventional loss and collaborative loss are utilized jointly to train the whole network. Extensive experiments on two publicly available VI-ReID datasets verify the superiority of the proposed PIC method. Specifically, the proposed method achieves a rank-1 accuracy of 83.6% and 57.5% on RegDB and SYSU-MM01 datasets, respectively.

An Intrinsic Structured Graph Alignment Module With Modality-Invariant Representations for NIR-VIS Face Recognition

Most existing near-infrared to visible (NIR-VIS) face recognition (FR) methods rely on global feature representations to reduce cross-modality discrepancies, but ignore the structural relationships between local features, e.g., the relative positions of eyes, nose, and mouth. Precise alignment of these local features can enhance the learning of modality-invariant face representations, thereby improving the performance of NIR-VIS FR. Therefore, in this letter, we propose an intrinsic structured graph alignment (ISGA) module that aims to obtain a graph-level local feature alignment across modalities. To this end, we first construct an intrinsic structure graph to model the inherent structural relationships of local features, and then enhance the discriminative feature representation by aligning the graphs between modalities. To jointly encourage cross-modality class consistency between semantics and structural relationships, a cross-modality class distribution (CMCD) loss is proposed to add an identity-preserving constraint to each class distribution in the embedding space between two modalities. To solve the resulting problem of suppressed class divisibility, we maximize the mutual information between inputs and class predictions. Extensive experiments on challenging NIR-VIS datasets indicate that our approach outperforms the state-of-the-arts. The code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/JianYu777/ISGA-CMCD</uri> .

MFCNet: Mining Features Context Network for RGB–IR Person Re-Identification

RGB–IR cross modality person re-identification (RGB–IR Re-ID) is an important task for video surveillance in poorly illuminated or dark environments. In addition to the common challenge of Re-ID, the large cross-modality variations between RGB and IR images must be considered. The existing RGB–IR Re-ID methods use different network structures to learn the global shared features associated with multi-modalities. However, most global shared feature learning methods are sensitive to background clutter, and contextual feature relationships are not considered among the mined features. To solve these problems, this paper proposes a dual-path attention network architecture MFCNet. SGA (Spatial-Global Attention) module embedded in MFCNet includes spatial attention and global attention branches to mine discriminative features. First, the SGA module proposed in this paper focuses on the key parts of the input image to obtain robust features. Next, the module mines the contextual relationships among features to obtain discriminative features and improve network performance. Finally, extensive experiments demonstrate that the performance of the network architecture proposed in this paper is better than that of state-of-the-art methods under various settings. In the all-search mode of the SYSU and RegDB data sets, the rank-1 accuracy reaches 51.64% and 69.76%, respectively.

Triplet interactive attention network for cross-modality person re-identification

Cross-modality (RGB-Infrared) person re-identification (ReID) has extreme superiority on low illumination ReID over conventional RGB-RGB matching due to its excellent capability on bridging cross-modality discrepancy. Previous works often focus on extracting modality-invariant global features, whereas the interaction among pairwise samples provides the most important clues for identifying persons. To explore the interactive clues among cross-modality pedestrian images, we propose a Triplet Interactive Attention Network (TIAN) to imitate the pairwise interactions and generate the attention scores for triplet samples. It can discover the fine-grained difference between pedestrian images and is optimized by successive attention losses to strengthen the feature learning capability of the backbone network. Meanwhile, the triplet interactive attention loss further alleviates the cross-modality discrepancy, based on the maximum mean discrepancy constraint. To demonstrate the effectiveness of our proposed TIAN, experiments on SYSU-MM01 and RegDB datasets show obvious superiority over popular cross-modality ReID methods, and the evaluation of main modules also reveals the novelties of our TIAN method.

Guidance Cleaning Network for Sketch-Based 3D Shape Retrieval

The sketch-based query for 3D shapes has drawn growing attention from the academic community due to its flexibility and accessibility. However, most recent works focus on reducing the cross-modality discrepancy between 2D sketches and 3D shapes, which neglect the influence caused by noise information in some low-quality free-hand sketches. To address this issue, we proposed a novel network to decrease the impact in two ways: 1) an attention weighting module to detect the noisy samples by a self-attention mechanism; 2) a data cleaning module to clear up low-quality sketches according to a ranking regularization. Experiments on two widely-used datasets demonstrate the effectiveness of our method.

Dual adaptive alignment and partitioning network for visible and infrared cross-modality person re-identification

Visible and infrared person re-identification (VI-ReID) describes the task of matching the images of a person, captured by visible-light and infrared cameras; this is a particular challenge in night time surveillance applications. Existing cross-modality recognition studies have been conducted mainly with a focus on learning the global and shareable feature representation of pedestrians to handle cross-modality discrepancies. However, the global features of pedestrian images cannot solve the unaligned image pairs efficiently, particularly when encountering the human appearance or posture misalignment caused by inaccurate pedestrian detection boxes. To mitigate the impact of these problems, we propose an end-to-end dual alignment and partitioning network to simultaneously learn global and local modal invariant features of pedestrians. First, we use two adaptive spatial transform modules to align the visible and infrared input images. Subsequently, the aligned image is divided horizontally, and the features of each local block are extracted. Then, we fuse these local features with global features. To alleviate the differences between heterogeneous modals and learn the common feature representation of heterogeneous modals, we map the features of heterogeneous modes into the same feature embedding space. Finally, we use the combination of identity loss and weighted regularized TriHard loss to improve the recognition accuracy. Extensive experimental results on two cross-modality datasets, RegDB and SYSU-MM01, demonstrate the superiority of the proposed method over other existing state-of-the-art methods.

Intra-class low-rank regularization for supervised and semi-supervised cross-modal retrieval

Cross-modal retrieval aims to retrieve related items across different modalities, for example, using an image query to retrieve related text. The existing deep methods ignore both the intra-modal and inter-modal intra-class low-rank structures when fusing various modalities, which decreases the retrieval performance. In this paper, two deep models (denoted as ILCMR and Semi-ILCMR) based on intra-class low-rank regularization are proposed for supervised and semi-supervised cross-modal retrieval, respectively. Specifically, ILCMR integrates the image network and text network into a unified framework to learn a common feature space by imposing three regularization terms to fuse the cross-modal data. First, to align them in the label space, we utilize semantic consistency regularization to convert the data representations to probability distributions over the classes. Second, we introduce an intra-modal low-rank regularization, which encourages the intra-class samples that originate from the same space to be more relevant in the common feature space. Third, an inter-modal low-rank regularization is applied to reduce the cross-modal discrepancy. To enable the low-rank regularization to be optimized using automatic gradients during network back-propagation, we propose the rank-r approximation and specify the explicit gradients for theoretical completeness. In addition to the three regularization terms that rely on label information incorporated by ILCMR, we propose Semi-ILCMR in the semi-supervised regime, which introduces a low-rank constraint before projecting the general representations into the common feature space. Extensive experiments on four public cross-modal datasets demonstrate the superiority of ILCMR and Semi-ILCMR over other state-of-the-art methods.

Grayscale Enhancement Colorization Network for Visible-Infrared Person Re-Identification

Visible-infrared person re-identification (VI-ReID) is an emerging and challenging cross-modality image matching problem because of the explosive surveillance data in night-time surveillance applications. To handle the large modality gap, various generative adversarial network models have been developed to eliminate the cross-modality variations based on a cross-modal image generation framework. However, the lack of point-wise cross-modality ground-truths makes it extremely challenging to learn such a cross-modal image generator. To address these problems, we learn the correspondence between single-channel infrared images and three-channel visible images by generating intermediate grayscale images as auxiliary information to colorize the single-modality infrared images. We propose a grayscale enhancement colorization network (GECNet) to bridge the modality gap by retaining the structure of the colored image which contains rich information. To simulate the infrared-to-visible transformation, the point-wise transformed grayscale images greatly enhance the colorization process. Our experiments conducted on two visible-infrared cross-modality person re-identification datasets demonstrate the superiority of the proposed method over the state-of-the-arts.

Cross-Visual Attention Fusion Network with Dual-Constrained Marginal-Ranking for Visible-Infrared Person Re-Identification

Visible-Infrared Person re-identification(VI-REID) is extremely important for night-time surveillance applications. It is a challenging problem due to large cross-modality discrepancies and intra-modality variations caused by different illuminations, human poses, viewpoints, etc. In this paper, we propose a cross visual attention fusion dual-path neural network with dual-constrained marginal ranking(DCAF) to solve the problem. First, we utilize cross-visual attention to learn discriminative feature of high-level semantic information in their respective modals. Second, in order to establish the relationship between modals, we fuse attentional weight of two modals and add it into backpropagation to obtain those regions that are distinctive for classification. Third, a dual-constrained marginal-ranking loss is introduced to narrow the gap between different networks and to learn strongly the similarities between two modals. Extensive experiments demonstrate that the proposed approach effectively improves the performance of VI-REID task and remarkably outperforms the state-of-the-art methods.

Mask-guided dual attention-aware network for visible-infrared person re-identification

Given a person of interest in RGB images, Visible-Infrared Person Re-identification (VI-REID) aims at searching for this person in infrared images. It faces a number of challenges due to large cross-modality discrepancies and intra-modality variations caused by illuminations, human poses, viewpoints and cluttered backgrounds, etc. This paper proposes a Mask-guided Dual Attention-aware Network (MDAN) for VI-REID. MDAN consists of two individual networks for two different modalities respectively, whose feature representations are driven by mask-guided attention-aware information and multi-loss constraints. Specifically, we first utilize masked image as a supplement to the original image, so as to enhance the contour and appearance information which are extremely important clues for matching the features of pedestrians from visible and infrared modalities. Second, a Residual Attention Module (RAM) is put forward to capture fine-grained features and subtle differences among pedestrians, so as to learn more discriminative features of pedestrians from heterogeneous modalities by adaptively calibrating feature responses along channel and spatial dimensions. Third, features from two individual streams of two modalities will be directly aggregated to form a cross-modality identity representation. Extensive experiments demonstrate that the proposed approach effectively improves the performance of VI-REID task and remarkably outperforms the state-of-the-art methods.

Aggregation-Based Graph Convolutional Hashing for Unsupervised Cross-Modal Retrieval

Cross-modal hashing has sparked much attention in large-scale information retrieval for its storage and query efficiency. Despite the great success achieved by supervised approaches, existing unsupervised hashing methods still suffer from the lack of reliable learning guidance and cross-modal discrepancy. In this paper, we propose Aggregation-based Graph Convolutional Hashing (AGCH) to tackle these obstacles. First, considering that a single similarity metric can hardly represent data relationships comprehensively, we develop an efficient aggregation strategy that utilises multiple metrics to construct a more precise affinity matrix for learning. Specifically, we apply various similarity measures to exploit the structural information of multiple modalities from different perspectives and then aggregate the obtained information to produce a joint similarity matrix. Furthermore, a novel deep model is designed to learn unified binary codes across different modalities, where the key components include modality-specific encoders, Graph Convolutional Networks (GCNs) and a fusion module. The modality-specific encoders are tasked to learn feature embeddings for each individual modality. On this basis, we leverage GCNs to further excavate the semantic structure of data, along with a fusion module to correlate different modalities. Extensive experiments on three real-world datasets demonstrate that the proposed method significantly outperforms the state-of-the-art competitors.

Dual-modality hard mining triplet-center loss for visible infrared person re-identification

Visible infrared person re-identification (VI-reid) has gradually increased in popularity as an crucial branch of person re-identification (reid). It not only has intra-class variations caused by different viewpoint, pedestrian posture changes, complex backgrounds, resolution, occlusions existing in traditional visible visible person re-identification (VV-reid), but also has been subjected to enormous cross-modality discrepancies resulted from the difference in the reflection spectrum of visible light and infrared cameras. In this paper, we put forward a novel loss function, called dual-modality hard mining triplet-center loss (DTCL), to optimize intra-class and inter-class distance and to supervise the network to learn discriminative feature representations. The DTCL learns a visible modality center and an infrared modality center separately for each class and selects online novel cross-modality triplets and intra-modality triplets for each sample respectively. In particular, it makes the samples from the same class closer to the class center and pushes them away from the centers of the different classes. Besides, we also propose a dual-path part-based feature learning network (DPFLN) framework, to learn the local features and solve the problem of cross-modality discrepancies. We conduct experiments on two cross-modality reid datasets, and obtain promising results.

Efficient Cross-Modality Graph Reasoning for RGB-Infrared Person Re-Identification

The modality and pose variance between RGB and infrared (IR) images are two key challenges for RGB-IR person re-identification. Existing methods mainly focus on leveraging pixel or feature alignment to handle the intra-class variations and cross-modality discrepancy. However, these methods are hard to keep semantic identity consistency between global and local representation, which the consistency is important for the cross-modality pedestrian re-identification task. In this work, we propose a novel cross-modality graph reasoning method (CGRNet) to globally model and reason over relations between modalities and context, and to keep semantic identity consistency between global and local representation. Specifically, we propose a local modality-similarity module to put the distribution of modality-specific features into a common subspace without losing identity information. Besides, we squeeze the input feature of RGB and IR images into a channel-wise global vector, and through graph reasoning, the identity relationship and modality relationship in each vector are inferred. Extensive experiments on two datasets demonstrate the superior performance of our approach over the existing state-of-the-art. The code is available at <uri>https://github.com/fegnyujian/CGRNet</uri>.

Parameter Sharing Exploration and Hetero-Center Triplet Loss for Visible-Thermal Person Re-Identification

This paper focuses on the visible-thermal cross-modality person re-identification (VT Re-ID) task, whose goal is to match person images between the daytime visible modality and the nighttime thermal modality. The two-stream network is usually adopted to address the cross-modality discrepancy, the most challenging problem for VT Re-ID, by learning the multi-modality person features. In this paper, we explore how many parameters of two-stream network should share, which is still not well investigated in the existing literature. By well splitting the ResNet50 model to construct the modality-specific feature extracting network and modality-sharing feature embedding network, we experimentally demonstrate the effect of parameters sharing of two-stream network for VT Re-ID. Moreover, in the framework of part-level person feature learning, we propose the hetero-center based triplet loss to relax the strict constraint of traditional triplet loss through replacing the comparison of anchor to all the other samples by anchor center to all the other centers. With the extremely simple means, the proposed method can significantly improve the VT Re-ID performance. The experimental results on two datasets show that our proposed method distinctly outperforms the state-of-the-art methods by large margins, especially on RegDB dataset achieving superior performance, rank1/mAP/mINP 91.05%/83.28%/68.84%. It can be a new baseline for VT Re-ID, with a simple but effective strategy.

LLM: Learning Cross-Modality Person Re-Identification via Low-Rank Local Matching

Existing methods of cross-modality person re-identification usually use set-level global feature constraints to reduce the cross-modality discrepancy. However they often ignore the modality-specific and modality-shared local feature matching. Modality-specific local matching could help extract discriminative identity-consistent features and alleviate spatial misalignment, and modality-shared local matching could make identity-consistent features be correlated in two modalities and mitigate the modality misalignment. Therefore, we establish the intra-modality and inter-modality co-occurrence relation between identity parts. We observe the modality-specific and modality-shared local feature matching of identity parts as intra-modality and inter-modality low-rank relation-finding problems. Therefore, in this work, we propose low-rank local matching (LLM) approach to establish the intra-modality and inter-modality co-occurrence relation between identity parts. First, in order to reinforce identity-consist features in two modalities and correlate these cross-modality identity-consist features together, the local matching (LM) module is designed to estimate partial co-occurrence probability by measuring local feature similarity. Moreover, we propose cross-modality triplet-center loss (CTLoss) which adds global constraint for each class distribution in embedding space to alleviate dramatic data expansion due to modality variance. Extensive experiments on two datasets demonstrate the superior performance of our approach over the existing state-of-the-art. Our code is released on https://github.com/fegnyujian/LLM .

Multi-level similarity learning for image-text retrieval

Image-text retrieval task has been a popular research topic and attracts a growing interest due to it bridges computer vision and natural language processing communities and involves two different modalities. Although a lot of methods have made a great progress in image-text task, it remains challenging because of the difficulty to learn the correspondence between two heterogeneous modalities. In this paper, we propose a multi-level representation learning for image-text retrieval task, which utilizes semantic-level, structural-level and contextual information to improve the quality of visual and textual representation. To utilize semantic-level information, we firstly extract the nouns, adjectives and number with high frequency as the semantic labels and adopt multi-label convolutional neural network framework to encode the semantic-level information. To explore the structure-level information of image-text pair, we firstly construct two graphs to encode the visual and textual information with respect to the corresponding modality and then, we apply graph matching with triplet loss to reduce the cross-modality discrepancy. To further improve the retrieval results, we utilize the contextual-level information from two modalities to refine the rank list and enhance the retrieval quality. Extensive experiments on Flickr30k and MSCOCO, which are two commonly datasets for image-text retrieval, have demonstrated the superiority of our proposed method.