Retrieval Performance Research Articles

Hypercube Pooling for Visual Semantic Embedding

Visual Semantic Embedding ( VSE ) is a primary model for cross-modal retrieval, wherein the global feature aggregator is a crucial component of the VSE model. In recent research, the General Pooling Operator ( GPO ) aggregator, which weighs the features reconstructed from the local feature set to aggregate, facilitates the related models to achieve good retrieval performance. However, the reason for the effectiveness remains to be explored. To enhance the rationality of aggregator designs, we analyze the reason from the perspective of feature space. Indeed, for each data, the local feature set forms a hypercube containing abundant data information, and the feature learned by GPO measures the hypercube, thereby representing the data. The geometric structure of the hypercube implies that the set containing all points within the hypercube is a convex set, so the feature learned by weighted aggregation is an interior point of the hypercube. However, using the interior point to measure the hypercube leads to some problems in feature representation and model optimization, as well as the reduction of retrieval efficiency caused by weight computation. For example, the related pair’s features may be far, while the unrelated ones may be close. To measure the hypercube more clearly and alleviate the problems mentioned above, we propose Hypercube Pooling ( HCP ) aggregator. Specifically, HCP concatenates the Max and Min Pooling features as the global features. This aggregation method has multiple advantages, e.g., the learned global feature represents all hyperplanes of the hypercube that contain critical information and hypercube geometric structure. Moreover, HCP adds normalization-before-concatenation and reduces the usual setting of margin in the loss function by half to avoid gradient loss caused by the difference in the feature value and dimensionality doubling. The experimental results on the Flickr30K and MSCOCO datasets show that the HCP model has excellent performance with high efficiency, confirming the correctness of the spatial analysis and the effectiveness of the HCP aggregator.

Image Retrievable Encryption Based on Linear Fitting and Orthogonal Transformation

Abstract With the development of cloud computing, an increasing number of resource-constrained image owners tend to store their images in the cloud and rely on image retrieval services to obtain the images they desire. However, the security of the cloud cannot be fully guaranteed. To ensure image security while achieving good retrieval performance, we have designed a retrievable image encryption algorithm based on linear fitting and orthogonal transformation. This algorithm first generates encryption and feature extraction domains through orthogonal decomposition, and then applies a modified ResNet50 network for feature extraction in the feature extraction domain. The encryption process employs an improved affine transformation based on linear fitting, where part of the fitting values comes from the original image data and the other part comes from data generated by a chaotic system. Additionally, to simplify the measurement of feature similarity in the cloud, we have designed a hierarchical feature index tree to narrow the retrieval scope, thereby reducing retrieval complexity. Experimental results show that the proposed algorithm effectively protects image privacy and achieves high retrieval accuracy. The F-score reached 6.7634% on the Ghim10k dataset and 25.514% on the Corel 1K dataset, significantly improving upon traditional methods. This algorithm has potential application value in the fields of secure image storage and efficient retrieval in the cloud.

Capturing cell heterogeneity in representations of cell populations for image-based profiling using contrastive learning.

Image-based cell profiling is a powerful tool that compares perturbed cell populations by measuring thousands of single-cell features and summarizing them into profiles. Typically a sample is represented by averaging across cells, but this fails to capture the heterogeneity within cell populations. We introduce CytoSummaryNet: a Deep Sets-based approach that improves mechanism of action prediction by 30-68% in mean average precision compared to average profiling on a public dataset. CytoSummaryNet uses self-supervised contrastive learning in a multiple-instance learning framework, providing an easier-to-apply method for aggregating single-cell feature data than previously published strategies. Interpretability analysis suggests that the model achieves this improvement by downweighting small mitotic cells or those with debris and prioritizing large uncrowded cells. The approach requires only perturbation labels for training, which are readily available in all cell profiling datasets. CytoSummaryNet offers a straightforward post-processing step for single-cell profiles that can significantly boost retrieval performance on image-based profiling datasets.

Biomedical Information Retrieval with Positive-Unlabeled Learning and Knowledge Graphs

The rapid growth of biomedical publications has presented significant challenges in the field of information retrieval. Most existing work focuses on document retrieval given explicit queries. However, in real applications such as curated biomedical database maintenance, explicit queries are missing. In this paper, we propose a two-step model for biomedical information retrieval in the case that only a small set of example documents is available without explicit queries. Initially, we extract keywords from the observed documents using large pre-trained language models and biomedical knowledge graphs. These keywords are then enriched with domain-specific entities. Information retrieval techniques can subsequently use the collected entities to rank the documents. Following this, we introduce an iterative Positive-Unlabeled learning method to classify all unlabeled documents. Experiments conducted on the PubMed dataset demonstrate that the proposed technique outperforms the state-of-the-art positive-unlabeled learning methods. The results underscore the effectiveness of integrating large language models and biomedical knowledge graphs in improving zero-shot information retrieval performance in the biomedical domain.

Preclinical Evaluation of MK-8189: A Novel Phosphodiesterase 10A Inhibitor for the Treatment of Schizophrenia.

MK-8189 is a novel phosphodiesterase 10A (PDE10A) inhibitor being evaluated in clinical studies for the treatment of schizophrenia. PDE10A is a cyclic nucleotide phosphodiesterase enzyme highly expressed in medium spiny neurons of the striatum. MK-8189 exhibits sub-nanomolar potency on the PDE10A enzyme and has excellent pharmaceutical properties. Oral administration of MK-8189 significantly increased cGMP and pGluR1 in rat striatal tissues. Activation of the dopamine D1 direct and D2 indirect pathways was demonstrated by detecting significant elevation of mRNA encoding substance P (Sub P) and enkephalin (ENK) after MK-8189 administration. The PDE10A tracer [3H]MK-8193 was used determine the PDE10A enzyme occupancy (EO) required for efficacy in behavioral models. In the rat conditioned avoidance responding assay, MK-8189 significantly decreased avoidance behavior at PDE10A EO greater than ~48%. MK-8189 significantly reversed an MK-801-induced deficit in pre-pulse inhibition at PDE10A EO of ~47% and higher. Target engagement of MK-8189 in rhesus monkeys was examined with [11C]MK-8193 in PET studies and plasma concentrations of 127nM MK-8189 yielded ~50% EO in the striatum. The impact of MK-8189 on cognitive symptoms was evaluated using the objective retrieval task in rhesus monkeys. MK-8189 significantly attenuated a ketamine-induced deficit in object retrieval performance at exposure that yielded ~29% PDE10A EO. These findings demonstrate the robust impact of MK-8189 on striatal signaling and efficacy in preclinical models of symptoms associated with schizophrenia. Data from these studies were used to establish the relationship between preclinical efficacy, plasma exposures, and PDE10A EO to guide dose selection of MK-8189 in clinical studies. Significance Statement We describe the primary pharmacology of MK-8189 a PDE10A inhibitor under evaluation for the treatment of schizophrenia. We report efficacy in preclinical models that have been used to characterize other PDE10A inhibitors and atypical antipsychotics. The PDE10A occupancy achieved by MK-8189 in behavioral studies was used to support dose selection in clinical trials. This work provides evidence to support exploration of higher levels of PDE10A occupancy in clinical trials to determine if this translates to improved efficacy in patients.

Deep semantics-preserving cross-modal hashing

Cross-modal hashing has been paid widespread attention in recent years due to its outstanding performance in cross-modal data retrieval. Cross-modal hashing can be decomposed into two steps, i.e., the feature learning and the binarization. However, most existing cross-modal hash methods do not take the supervisory information of the data into consideration during binary quantization, and thus often fail to adequately preserve semantic information. To solve these problems, this paper proposes a novel deep cross-modal hashing method called deep semantics-preserving cross-modal hashing (DSCMH), which makes full use of intra and inter-modal semantic information to improve the model's performance. Moreover, by designing a label network for semantic alignment during the binarization process, DSCMH's performance can be further improved. In order to verify the performance of the proposed method, extensive experiments were conducted on four big datasets. The results show that the proposed method is better than most of the existing cross-modal hashing methods. In addition, the ablation experiment shows that the proposed new regularized terms all have positive effects on the model's performances in cross-modal retrieval. The code of this paper can be downloaded from http://www.scholat.com/laizhihui.

Vector space model, term frequency-inverse document frequency with linear search, and object-relational mapping Django on hadith data search

For Muslims, the Hadith ranks as the secondary legal authority following the Quran. This research leverages hadith data to streamline the search process within the nine imams’ compendium using the vector space model (VSM) approach. The primary objective of this research is to enhance the efficiency and effectiveness of the search process within Hadith collections by implementing pre-filtering techniques. This study aims to demonstrate the potential of linear search and Django object-relational mapping (ORM) filters in reducing search times and improving retrieval performance, thereby facilitating quicker and more accurate access to relevant Hadiths. Prior studies have indicated that VSM is efficient for large data sets because it assigns weights to every term across all documents, regardless of whether they include the search keywords. Consequently, the more documents there are, the more protracted the weighting phase becomes. To address this, the current research pre-filters documents prior to weighting, utilizing linear search and Django ORM as filters. Testing on 62,169 hadiths with 20 keywords revealed that the average VSM search duration was 51 seconds. However, with the implementation of linear and Django ORM filters, the times were reduced to 7.93 and 8.41 seconds, respectively. The recall@10 rates were 79% and 78.5%, with MAP scores of 0.819 and 0.814, accordingly.

Text–video retrieval re-ranking via multi-grained cross attention and frozen image encoders

State-of-the-art methods for text–video retrieval generally leverage CLIP embeddings and cosine similarity for efficient retrieval. Meanwhile, recent advancements in cross-attention techniques introduce transformer decoders to facilitate attention computation between text queries and visual tokens extracted from video frames, enabling a more comprehensive interaction between textual and visual information. In this study, we combine the advantages of both approaches and propose a fine-grained re-ranking approach incorporating a multi-grained text–video cross attention module. Specifically, the re-ranker enhances the top K similar candidates identified by the cosine similarity network. To explore video and text interactions efficiently, we introduce frame and video token selectors to obtain salient visual tokens at both frame and video levels. Then, a multi-grained cross-attention mechanism is applied between text and visual tokens at these levels to capture multimodal information. To reduce the training overhead associated with the multi-grained cross-attention module, we freeze the vision backbone and only train the multi-grained cross attention module. This frozen strategy allows for scalability to larger pre-trained vision models such as ViT-G, leading to enhanced retrieval performance. Experimental evaluations on text–video retrieval datasets showcase the effectiveness and scalability of our proposed re-ranker combined with existing state-of-the-art methodologies.

Cross-modal similar clinical case retrieval using a modular model based on contrastive learning and k-nearest neighbor search

ObjectiveElectronic health record systems have made it possible for clinicians to use previously encountered similar cases to support clinical decision-making. However, most studies for similar case retrieval were based on single-modal data. The existing studies on cross-modal clinical case retrieval were limited. We aimed to develop a CRoss-Modal Retrieval (CRMR) model to retrieve similar clinical cases recorded in different data modalities. Materials and methodsThe publically available Medical Information Mart for Intensive Care-Chest X-ray (MIMIC-CXR) dataset was used for model development and testing. The CRMR model was designed as a modular model containing two feature extraction models, two feature transformation models, one feature transformation optimization model, and one case retrieval model. The ability to retrieve similar clinical cases recorded in different data modalities was facilitated by the use of contrastive deep learning and k-nearest neighbor search. ResultsThe average retrieval precision, denoted as AP@k, of the developed CRMR model, were 76.9 %@5, 76.7 %@10, 76.5 %@20, 76.3 %@50, and 77.9 %@100, respectively. Here k is the number of similar cases returned after retrieval. The average retrieval time varied from 0.013 ms to 0.016 ms with k varying from 5 to 100. Moreover, the model can retrieve similar cases with the same multiple radiographic manifestations as the query case. DiscussionThe CRMR model has shown promising cross-modal retrieval performance in clinical case analysis, with the potential for future scalability and improvement in handling diverse disease types and data modalities. The CRMR model has promising potential to aid clinicians in making optimal and explainable clinical decisions.

Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval

Deep hashing technology, known for its low-cost storage and rapid retrieval, has become a focal point in cross-modal retrieval research as multimodal data continue to grow. However, existing supervised methods often overlook noisy labels and multiscale features in different modal datasets, leading to higher information entropy in the generated hash codes and features, which reduces retrieval performance. The variation in text annotation information across datasets further increases the information entropy during text feature extraction, resulting in suboptimal outcomes. Consequently, reducing the information entropy in text feature extraction, supplementing text feature information, and enhancing the retrieval efficiency of large-scale media data are critical challenges in cross-modal retrieval research. To tackle these, this paper introduces the Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval (TEGAH) framework. TEGAH incorporates a deep text feature extraction network and a multiscale label region fusion network to minimize information entropy and optimize feature extraction. Additionally, a Graph-Attention-based modal feature fusion network is designed to efficiently integrate multimodal information, enhance the affinity of the network for different modes, and retain more semantic information. Extensive experiments on three multilabel datasets demonstrate that the TEGAH framework significantly outperforms state-of-the-art cross-modal hashing methods.

Multi-view visual semantic embedding for cross-modal image–text retrieval

Visual Semantic Embedding (VSE) is a dominant method for cross-modal image–text retrieval. The purpose of VSE is to learn an embedding space where images can be embedded close to the corresponding captions. However, there are large intra-class variations in image–text data. Multiple captions describing the same image may be described from different views, and descriptions of different views are often dissimilar. The VSE method embeds samples from the same class in similar positions, which suppresses intra-class variations and leads to inferior generalization. This paper proposes a Multi-View Visual Semantic Embedding (MV-VSE) framework that learns multiple embeddings for an image, explicitly modeling intra-class variation. To optimize the MV-VSE framework, a multi-view triplet loss is proposed, which jointly optimizes multi-view embeddings while retaining intra-class variation. Recently, large-scale Vision-Language Pre-training (VLP) has become a new paradigm for cross-modal image–text retrieval. To allow our framework to be flexibly applied to the traditional VSE models and VSE-based VLP models, we incorporate the contrastive loss commonly used in VLP and the triplet loss into a unified loss, and further propose a multi-view unified loss. Our framework can be applied plug-and-play to traditional VSE models and VSE-based VLP models without excessively increasing model complexity. Experimental results on the image–text retrieval benchmark datasets demonstrate that applying our framework can boost the retrieval performance of current VSE models. The code is available at https://github.com/AAA-Zheng/MV-VSE.

A Self-Distilled Learning to Rank Model for Ad Hoc Retrieval

Learning to rank models are broadly applied in ad hoc retrieval for scoring and sorting documents based on their relevance to textual queries. The generalizability of the trained model in the learning to rank approach, however, can have an impact on the retrieval performance, particularly when data includes noise and outliers, or is incorrectly collected or measured. In this paper, we introduce a Self-Distilled Learning to Rank (SDLR) framework for ad hoc retrieval, and analyze its performance over a range of retrieval datasets and also in the presence of features’ noise. SDLR assigns a confidence weight to each training sample, aiming at reducing the impact of noisy and outlier data in the training process. The confidence weight is approximated based on the feature’s distributions derived from the values observed for the features of the documents labeled for a query in a listwise training sample. SDLR includes a distillation process that facilitates passing on the underlying patterns in assigning confidence weights from the teacher model to the student one. We empirically illustrate that SDLR outperforms state-of-the-art learning to rank models in ad hoc retrieval. We thoroughly investigate the SDLR performance in different settings including when no distillation strategy is applied; when different portion of data are used for training the teacher and the student models, and when both teacher and student models are trained over identical data. We show that SDLR is more effective when training data are split between a teacher and a student model. We also show that SDLR’s performance is robust when data features are noisy.

The Ice Cloud Imager: retrieval of frozen water column properties

Abstract. The Ice Cloud Imager (ICI) aboard the second generation of the EUMETSAT Polar System (EPS-SG) will provide novel measurements of ice hydrometeors. ICI is a passive conically scanning radiometer that will operate within a frequency range of 183 to 664 GHz, helping to cover the present wavelength gap between microwave and infrared observations. Reliable global data will be produced on a daily basis. This paper presents the retrieval database to be used operationally and performs a final pre-launch assessment of ICI retrievals. Simulations are performed within atmospheric states that are consistent with radar reflectivities and represent the three-dimensional (3D) variability of clouds. The radiative transfer calculations use empirically based hydrometeor models. Azimuthal orientation of particles is mimicked, allowing for the consideration of polarisation. The degrees of freedom (DoFs) of the ICI retrieval database are shown to vary according to cloud type. The simulations are considered to be the most detailed performed to this date. Simulated radiances are shown to be statistically consistent with real observations. Machine learning is applied to perform inversions of the simulated ICI observations. The method used allows for the estimation of non-Gaussian uncertainties for each retrieved case. Retrievals of ice water path (IWP), mean mass height (Zm), and mean mass diameter (Dm) are presented. Distributions and zonal means of both database and retrieved IWP show agreement with DARDAR. Retrieval tests indicate that ICI will be sensitive to IWP between 10−2 and 101 kg m−2. Retrieval performance is shown to vary with climatic region and surface type, with the best performance achieved over tropical regions and over ocean. As a consequence of this study, retrievals from real observations will be possible from day one of the ICI operational phase.

Analysis of document retrieval for online public administrative procedure services.

Public Administrative Procedures (APs) are processes, implementation methods, documents, and requirements or conditions prescribed by gov­ernment agencies or authorized individuals to ad­dress specific tasks related to individuals or orga­nizations. However, organizations and citizens still face challenges in easily and conveniently accessing information and public administrative services. This paper investigates advanced techniques to address the challenges of document retrieval for public admin­istrative services. We implement a hybrid retrieval process, combining traditional retrieval models such as TF-IDF and BM25 with modern models such as SBERT and fine-tuning models. Results demonstrate that combining these models significantly enhances retrieval performance. The ensemble model of BM25 and finetuned SBERT achieved the highest F2 score, indicating superior effectiveness in information re­trieval.

A novel Deep High-level Concept-mining Jointing Hashing Model for unsupervised cross-modal retrieval

Unsupervised cross-modal hashing has achieved great success in various information retrieval applications owing to its efficient storage usage and fast retrieval speed. Recent studies have primarily focused on training the hash-encoded networks by calculating a sample-based similarity matrix to improve the retrieval performance. However, there are two issues remain to solve: (1) The current sample-based similarity matrix only considers the similarity between image-text pairs, ignoring the different information densities of each modality, which may introduce additional noise and fail to mine key information for retrieval; (2) Most existing unsupervised cross-modal hashing methods only consider alignment between different modalities, while ignoring consistency between each modality, resulting in semantic conflicts. To tackle these challenges, a novel Deep High-level Concept-mining Jointing Hashing (DHCJH) model for unsupervised cross-modal retrieval is proposed in this study. DHCJH is able to capture the essential high-level semantic information from image modalities and integrate into the text modalities to improve the accuracy of guidance information. Additionally, a new hashing loss with a regularization term is introduced to avoid the cross-modal semantic collision and false positive pairs problems. To validate the proposed method, extensive comparison experiments on benchmark datasets are conducted. Experimental findings reveal that DHCJH achieves superior performance in both accuracy and efficiency. The code of DHCJH is available at Github.

Treadmill Exercise Facilitates Synaptic Plasticity in APP/PS1 Mice by Regulating Hippocampal AMPAR Activity.

Accumulating evidence underscores exercise as a straightforward and cost-effective lifestyle intervention capable of mitigating the risk and slowing the emergence and progression of Alzheimer's disease (AD). However, the intricate cellular and molecular mechanisms mediating these exercise-induced benefits in AD remain elusive. The present study delved into the impact of treadmill exercise on memory retrieval performance, hippocampal synaptic plasticity, synaptic morphology, and the expression and activity of α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic receptors (AMPARs) in 6-month-old APP/PS1 mice. APP/PS1 mice (4-month-old males) were randomly assigned to either a treadmill exercise group or a sedentary group, with C57BL/6J mice (4-month-old males) as the control group (both exercise and sedentary). The exercise regimen spanned 8 weeks. Our findings revealed that 8-week treadmill exercise reversed memory retrieval impairment in step-down fear conditioning in 6-month-old APP/PS1 mice. Additionally, treadmill exercise enhanced basic synaptic strength, short-term potentiation (STP), and long-term potentiation (LTP) of the hippocampus in these mice. Moreover, treadmill exercise correlated with an augmentation in synapse numbers, refinement of synaptic structures, and heightened expression and activity of AMPARs. Our findings suggest that treadmill exercise improves behavioral performance and facilitates synaptic transmission by increasing structural synaptic plasticity and the activity of AMPARs in the hippocampus of 6-month-old APP/PS1 mice, which is involved in pre- and postsynaptic processes.

Parameterization before Meta-Analysis: Cross-Modal Embedding Clustering for Forest Ecology Question-Answering

In the field of forestry ecology, image data capture factual information, while literature is rich with expert knowledge. The corpus within the literature can provide expert-level annotations for images, and the visual information within images naturally serves as a clustering center for the textual corpus. However, both image data and literature represent large and rapidly growing, unstructured datasets of heterogeneous modalities. To address this challenge, we propose cross-modal embedding clustering, a method that parameterizes these datasets using a deep learning model with relatively few annotated samples. This approach offers a means to retrieve relevant factual information and expert knowledge from the database of images and literature through a question-answering mechanism. Specifically, we align images and literature across modalities using a pair of encoders, followed by cross-modal information fusion, and feed these data into an autoregressive generative language model for question-answering with user feedback. Experiments demonstrate that this cross-modal clustering method enhances the performance of image recognition, cross-modal retrieval, and cross-modal question-answering models. Our method achieves superior performance on standardized tasks in public datasets for image recognition, cross-modal retrieval, and cross-modal question-answering, notably achieving a 21.94% improvement in performance on the cross-modal question-answering task of the ScienceQA dataset, thereby validating the efficacy of our approach. Essentially, our method targets cross-modal information fusion, combining perspectives from multiple tasks and utilizing cross-modal representation clustering of images and text. This approach effectively addresses the interdisciplinary complexity of forestry ecology literature and the parameterization of unstructured heterogeneous data encapsulating species diversity in conservation images. Building on this foundation, intelligent methods are employed to leverage large-scale data, providing an intelligent research assistant tool for conducting forestry ecological studies on larger temporal and spatial scales.

Improving Scene Text Retrieval via Stylized Middle Modality

Scene text retrieval addresses the challenge of localizing and searching for all text instances within scene images based on a query text. This cross-modal task has significant applications in various domains, such as intelligent transportation systems and social media analysis. In practice, ensuring consistency of the same content between two modalities is crucial in improving retrieval accuracy. This paper addresses the issue by introducing a stylized middle modality, which fuses the graphical query text with the style of the extracted text proposal. To this end, we propose a stylized middle modality learning (SM \({}^{2}\) L) framework. The proposed stylized middle modality enables the network to jointly enforce constraints on visual feature coherence and text semantic feature consistency in the optimization phase, thereby minimizing the modality gap in the retrieval space. This brings in two major advantages: 1) SM \({}^{2}\) L will pave the way to seamlessly benefit the scene text retrieval and 2) the proposed learning paradigm enables the machine to avoid adding redundant computing resources in the inference phase. Substantial experiments demonstrate that the proposed method outperforms the state-of-the-art retrieval performance considerably.

GNSS-IR Soil Moisture Retrieval Using Multi-Satellite Data Fusion Based on Random Forest

The accuracy and reliability of soil moisture retrieval based on Global Positioning System (GPS) single-star Signal-to-Noise Ratio (SNR) data is low due to the influence of spatial and temporal differences of different satellites. Therefore, this paper proposes a Random Forest (RF)-based multi-satellite data fusion Global Navigation Satellite System Interferometric Reflectometry (GNSS-IR) soil moisture retrieval method, which utilizes the RF Model’s Mean Decrease Impurity (MDI) algorithm to adaptively assign arc weights to fuse all available satellite data to obtain accurate retrieval results. Subsequently, the effectiveness of the proposed method was validated using GPS data from the Plate Boundary Observatory (PBO) network sites P041 and P037, as well as data collected in Lamasquere, France. A Support Vector Machine model (SVM), Radial Basis Function (RBF) neural network model, and Convolutional Neural Network model (CNN) are introduced for the comparison of accuracy. The results indicated that the proposed method had the best retrieval performance, with Root Mean Square Error (RMSE) values of 0.032, 0.028, and 0.003 cm3/cm3, Mean Absolute Error (MAE) values of 0.025, 0.022, and 0.002 cm3/cm3, and correlation coefficients (R) of 0.94, 0.95, and 0.98, respectively, at the three sites. Therefore, the proposed soil moisture retrieval model demonstrates strong robustness and generalization capabilities, providing a reference for achieving high-precision, real-time monitoring of soil moisture.

Implicit Modality Knowledge Alignment and Uncertainty Estimation for visible-infrared person re-identification

Visible-infrared person re-identification (VI-ReID) is a crucial cross-modal matching task in computer vision. Existing research typically relies on a single auxiliary modality to address the semantic gap in cross-modal data, but this approach fails to fully exploit the information inherent in both modalities. To tackle this issue, we propose a novel framework called Implicit Modality Knowledge Alignment (IMKA) and Uncertainty Estimation (UE). This framework aims to enhance the robustness of the learned common embedding space and improve the prediction accuracy of the VI-ReID model. The IMKA module is designed to extract valuable supplementary information from the original modality data to generate implicit modality data. The SKA module then aligns the distribution of significant knowledge learned from this generated implicit modality data. Meanwhile, the UE strategy is implemented to mitigate the overconfidence in incorrect predictions. The uncertainty of predicted probabilities is evaluated in a Dirichlet space, and the designed evidence loss bolsters the confidence in uncertainty. This uncertain information can enhance retrieval performance. Extensive experimental results on two publicly available VI-ReID datasets demonstrate that our IMKA-UE framework significantly outperforms state-of-the-art methods. The code for our IMKA-UE framework is available at https://github.com/SWU-CS-MediaLab/IMKA-UE.