Multimodal Retrieval Research Articles

Identifying Implicit Social Biases in Vision-Language Models

Vision-language models, like CLIP (Contrastive Language Image Pretraining), are becoming increasingly popular for a wide range of multimodal retrieval tasks. However, prior work has shown that large language and deep vision models can learn historical biases contained in their training sets, leading to perpetuation of stereotypes and potential downstream harm. In this work, we conduct a systematic analysis of the social biases that are present in CLIP, with a focus on the interaction between image and text modalities. We first propose a taxonomy of social biases called So-B-It, which contains 374 words categorized across ten types of bias. Each type can lead to societal harm if associated with a particular demographic group. Using this taxonomy, we examine images retrieved by CLIP from a facial image dataset using each word as part of a prompt. We find that CLIP frequently displays undesirable associations between harmful words and specific demographic groups, such as retrieving mostly pictures of Middle Eastern men when asked to retrieve images of a "terrorist". Finally, we conduct an analysis of the source of such biases, by showing that the same harmful stereotypes are also present in a large image-text dataset used to train CLIP models for examples of biases that we find. Our findings highlight the importance of evaluating and addressing bias in vision-language models, and suggest the need for transparency and fairness-aware curation of large pre-training datasets.

Application of Multimedia Information Retrieval Technology in Japanese Text Content Information Query Platform

The research and development of Japanese operating platform has become a new industry. At the same time, it is necessary to strengthen the research on query and feedback technology in MIR(Multimedia Information Retrieval). Based on this, this paper introduces the research status and progress of query and feedback technology in multimedia retrieval, and introduces the overall framework and technical links of MIR. This paper discusses the application of MIR technology in Japanese text content information query platform. A new multimodal retrieval mechanism is proposed, which combines cross-reference index and underlying features in retrieval and related feedback to realize unified processing of multimodal data. The time consumption of the algorithm is lower than that of the two comparative algorithms, which also verifies the superiority. It is feasible to apply the MIR technology proposed in this paper to the Japanese text content information query platform. It provides a new path for Japanese text content information retrieval.

Flexible Dual Multi-Modal Hashing for Incomplete Multi-Modal Retrieval

Multimodal hashing aims to efficiently integrate multi-source data into a unified discrete Hamming space, facilitating fast similarity searches with minimal query and storage overhead. Traditional multimodal hashing assumes that data from different sources are fully observed, an assumption that fails in real-world scenarios involving large-scale multimodal data, thereby compromising conventional methods. To address these limitations during both training and retrieval, our approach manages dual-stage data missing, occurring in both phases. In this paper, we introduce a novel framework called Flexible Dual Multimodal Hashing (FDMH), which recovers missing data at both stages by jointly leveraging low-dimensional data relations and semantic graph structural relationships in multi-source data, achieving promising performance in incomplete multimodal retrieval. We transform the original features into anchor graphs and use existing modalities to reconstruct the anchor graphs of missing modalities. Based on these anchor graphs, we perform weight-adaptive fusion in the semantic space, supervised by original semantic labels, and apply a tensor nuclear norm to enforce consistency constraints on the projection matrices across different modalities. Furthermore, our method flexibly fuses existing and recovered modalities during retrieval. We validate the effectiveness of our approach through extensive experiments on four large-scale multimodal datasets, demonstrating its robust performance in real-world dual-missing retrieval scenarios.

An Interactive Multi-Modal Query Answering System with Retrieval-Augmented Large Language Models

Retrieval-augmented Large Language Models (LLMs) have reshaped traditional query-answering systems, offering unparalleled user experiences. However, existing retrieval techniques often struggle to handle multi-modal query contexts. In this paper, we present an interactive M ulti-modal Q uery A nswering (MQA) system, empowered by our newly developed multi-modal retrieval framework and navigation graph index, integrated with cutting-edge LLMs. It comprises five core components: Data Preprocessing, Vector Representation, Index Construction, Query Execution, and Answer Generation, all orchestrated by a dedicated coordinator to ensure smooth data flow from input to answer generation. One notable aspect of MQA is its utilization of contrastive learning to assess the significance of different modalities, facilitating precise measurement of multimodal information similarity. Furthermore, the system achieves efficient retrieval through our advanced navigation graph index, refined using computational pruning techniques. Another highlight of our system is its pluggable processing framework, allowing seamless integration of embedding models, graph indexes, and LLMs. This flexibility provides users diverse options for gaining insights from their multi-modal knowledge base. A preliminary video introduction of MQA is available at https://youtu.be/xvUuo2ZIqWk.

Fast unsupervised multi-modal hashing based on piecewise learning

Unsupervised hashing has been extensively applied in large-scale multi-modal retrieval by mapping original data from heterogeneous modalities into unified binary codes. However, there still remain challenges especially how to balance the individual modality-specific representations and common representation preserving intrinsic linkages among heterogeneous modalities. In this paper, we propose a novel fast Unsupervised Multi-modal Hashing based on Piecewise Learning, denoted as UMHPL, to deal with the mentioned issue. Initially, we formulate the problem as matrix factorization to derive the individual modality-specific latent representations and common latent representation with consensus matrices in a brief time. To maintain the integrality of multi-modal data, we integrate them by adaptive weight factors and nuclear norm minimization. Subsequently, we establish a connection between the individual modality-specific latent representations and common latent representation based on the piecewise hash learning framework to reinforce the discriminative competency of model, which leads the hash codes more compact. Finally, an effective discrete optimization algorithm in mathematical logic and functional analysis is proposed. Comprehensive experiments on Wiki, MIRFlirck, NUS-WIDE, and MSCOCO datasets demonstrate the superior performance of UMHPL to state-of-the-art hashing methods.

A Web Knowledge-Driven Multimodal Retrieval Method in Computational Social Systems: Unsupervised and Robust Graph Convolutional Hashing

Multimodal retrieval has received widespread consideration since it can commendably provide massive related data support for the development of computational social systems (CSSs). However, the existing works still face the following challenges: 1) rely on the tedious manual marking process when extended to CSS, which not only introduces subjective errors but also consumes abundant time and labor costs; 2) only using strongly aligned data for training, lacks concern for the adjacency information, which makes the poor robustness and semantic heterogeneity gap difficult to be effectively fit; and 3) mapping features into real-valued forms, which leads to the characteristics of high storage and low retrieval efficiency. To address these issues in turn, we have designed a multimodal retrieval framework based on web-knowledge-driven, called <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">unsupervised and robust graph convolutional hashing</i> (URGCH). The specific implementations are as follows: first, a “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">secondary semantic self-fusion</i> ” approach is proposed, which mainly extracts semantic-rich features through pretrained neural networks, constructs the joint semantic matrix through semantic fusion, and eliminates the process of manual marking; second, a “ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">adaptive computing</i> ” approach is designed to construct enhanced semantic graph features through the knowledge-infused of neighborhoods and uses graph convolutional networks for knowledge fusion coding, which enables URGCH to sufficiently fit the semantic modality gap while obtaining satisfactory robustness features; Third, combined with hash learning, the multimodality data are mapped into the form of binary code, which reduces storage requirements and improves retrieval efficiency. Eventually, we perform plentiful experiments on the web dataset. The results evidence that URGCH exceeds other baselines about <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1\%$</tex-math> </inline-formula> – <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.7\%$</tex-math> </inline-formula> in mean average precisions (MAPs), displays superior performance in all the aspects, and can meaningfully provide multimodal data retrieval services to CSS.

Invisible Black-Box Backdoor Attack against Deep Cross-Modal Hashing Retrieval

Deep cross-modal hashing has promoted the field of multi-modal retrieval due to its excellent efficiency and storage, but its vulnerability to backdoor attacks is rarely studied. Notably, current deep cross-modal hashing methods inevitably require large-scale training data, resulting in poisoned samples with imperceptible triggers that can easily be camouflaged into the training data to bury backdoors in the victim model. Nevertheless, existing backdoor attacks focus on the uni-modal vision domain, while the multi-modal gap and hash quantization weaken their attack performance. In addressing the aforementioned challenges, we undertake an invisible black-box backdoor attack against deep cross-modal hashing retrieval in this article. To the best of our knowledge, this is the first attempt in this research field. Specifically, we develop a flexible trigger generator to generate the attacker’s specified triggers, which learns the sample semantics of the non-poisoned modality to bridge the cross-modal attack gap. Then, we devise an input-aware injection network, which embeds the generated triggers into benign samples in the form of sample-specific stealth and realizes cross-modal semantic interaction between triggers and poisoned samples. Owing to the knowledge-agnostic of victim models, we enable any cross-modal hashing knockoff to facilitate the black-box backdoor attack and alleviate the attack weakening of hash quantization. Moreover, we propose a confusing perturbation and mask strategy to induce the high-performance victim models to focus on imperceptible triggers in poisoned samples. Extensive experiments on benchmark datasets demonstrate that our method has a state-of-the-art attack performance against deep cross-modal hashing retrieval. Besides, we investigate the influences of transferable attacks, few-shot poisoning, multi-modal poisoning, perceptibility, and potential defenses on backdoor attacks. Our codes and datasets are available at https://github.com/tswang0116/IB3A.

Multimodal bird information retrieval system

Multimodal bird information retrieval system can help people popularize bird knowledge and help bird conservation. In this paper, we use the self-built bird dataset, the ViT-B/32 model in CLIP model as the training model, python as the development language, and PyQT5 to complete the interface development. The system mainly realizes the uploading and displaying of bird pictures, the multimodal retrieval function of bird information, and the introduction of related bird information. The results of the trial run show that the system can accomplish the multimodal retrieval of bird information, retrieve the species of birds and other related information through the pictures uploaded by the user, or retrieve the most similar bird information through the text content described by the user.

Visual Language – Let the Product Say What You Want

Visual Language is a multitasking on-line system focusing on e-commerce, which involves in generating accurate product descriptions for sellers and providing convenient product retrieval service for customers. To achieve this goal, the system adopts image description technology and multi-modal retrieval technology. By utilizing cross-modal generation technique, we could help sellers on rapid uploading products and customers on rapid retrieval, which could improve the experience of both sellers and customers.

Learning to disentangle and fuse for fine-grained multi-modality ship image retrieval

Multi-modality ship image retrieval aims to retrieve ship images from a large dataset, encompassing various modalities, when provided with a query ship image. One of the key challenges is addressing intra-modality variations and cross-modality discrepancies resulting from complex image content and different types of imaging systems. Current multi-modality retrieval methods primarily focus on extracting heterogeneous features from diverse modalities and mapping them into a shared feature space. However, these methods fail to determine the meaningfulness of the extracted features for retrieval tasks, as the features extracted across different modalities exhibit highly complex interactions. To overcome this limitation, we propose a novel framework, dubbed Disentangled Fusion Network (DFN), to decompose ship images into attribute and identity features, capturing modality-specific details and facilitating ship identification. This is achieved through an image disentanglement and fusion module, separating features across different modalities and converting the challenging multi-modality retrieval problem into a more manageable single-modality problem. To enable fine-grained retrieval, a region mining module identifies discriminative areas within ship images. The training process utilizes a bilevel optimization strategy to learn parameters, optimizing both image fusion and retrieval. Extensive experiments conducted on a publicly available multi-modality ship image dataset demonstrate the superior performance of the proposed DFN in image fusion and retrieval tasks.

Asymmetric Supervised Fusion-Oriented Hashing for Cross-Modal Retrieval.

Hashing technologies have been widely applied for large-scale multimodal retrieval tasks owing to their excellent performance in search and storage tasks. Although some effective hashing methods have been proposed, it is still difficult to handle the intrinsic linkages that exist among different heterogeneous modalities. Moreover, optimizing the discrete constraint problem through a relaxation-based strategy results in a large quantization error and leads to a suboptimal solution. In this article, we present a novel asymmetric supervised fusion-oriented hashing method, named (ASFOH), which investigates three novel schemes to remedy the above issues. Specifically, we first explicitly formulate the problem as matrix decomposition into a common latent representation and a transformation matrix, combined with an adaptive weight scheme and nuclear norm minimization to ensure the information completeness of multimodal data. Then, we associate the common latent representation with the semantic label matrix, thereby increasing the discriminative capability of the model by constructing an asymmetric hash learning framework, thus, making the generated hash codes more compact. Finally, an efficient discrete optimization iterative algorithm based on nuclear norm minimization is proposed to decompose the nonconvex multivariate optimization problem into several subproblems with analytical solutions. Comprehensive experiments on the MIRFlirck, NUS-WIDE, and IARP-TC12 datasets testify that ASFOH outperforms the compared state-of-the-art approaches.

Multiple Pseudo-Siamese Network with Supervised Contrast Learning for Medical Multi-modal Retrieval

Medical multi-modal retrieval aims to provide doctors with similar medical images from different modalities, which can greatly promote the efficiency and accuracy of clinical diagnosis. However, most existing medical retrieval methods hardly support the retrieval of multi-modal medical images, i.e., the number of modalities is greater than 2, and just convert retrieval to classification or clustering. It futilely breaks the gap between the visual information and the semantic information in different medical image modalities. To solve the problem, a S upervised C ontrast L earning method based on a M ultiple P seudo- S iamese network (SCL-MPS) is proposed for multi-modal medical image retrieval. In order to make the samples with semantic similarity close neighbors on Riemann manifold, the multiple constraints based on semantic consistency and modal invariance are designed in different forward stages of SCL-MPS. We theoretically demonstrate the feasibility of the designed constraints. Finally, experiments on four benchmark datasets (ADNI1, ADNI2, ADNI3, and OASIS3) show that SCL-MPS achieves state-of-the-art performance compared to 15 retrieval methods. Especially, SCL-MPS achieves a 100% mAP score in medical cross-modal retrieval on ADNI1.

LCEMH: Label Correlation Enhanced Multi-modal Hashing for efficient multi-modal retrieval

Supervised multi-modal hashing can effectively improve the discriminative power of hash codes by leveraging semantic label information. However, most existing supervised multi-modal hashing models generally rely on discrete labels for supervision. They only provide limited guidance and may not fully capture the potential semantic similarity between multi-modal data. Moreover, the binary constraints of hash codes further restrict their ability to fully capture the rich semantic information embedded in the labels. To address these limitations, we propose a Label Correlation Enhanced Multi-modal Hashing (LCEMH) approach for efficient multi-modal retrieval. Specifically, our proposed model can effectively expand the inter-class boundaries of discrete labels to further enrich the supervision of similarity. Additionally, we directly transform the acquired richer label information into hash codes simply and effectively, thus retaining more comprehensive information in the labels with minimal quantization loss, and enhancing the discriminant ability of the generated hash codes. Experimental results on three public multimedia retrieval datasets demonstrate the superiority of LCEMH in performance. The source code of LCEMH is available online at: https://github.com/ChaoqunZheng/LCEMH.

Constrained Bipartite Graph Learning for Imbalanced Multi-Modal Retrieval

Constrained Bipartite Graph Learning for Imbalanced Multi-Modal Retrieval

Hashing Fake: Producing Adversarial Perturbation for Online Privacy Protection Against Automatic Retrieval Models

The wide application of deep neural networks (DNNs) has significantly improved the performance of hashing models on multimodal retrieval issues. DNN-based deep models can automatically learn semantic features from raw data to make human-level decisions. However, the superior generalization leads to potential privacy leakage risks. Strong DNN-based retrieval models enable malicious crawlers to search for nontag private information based on semantic similarity matching. Hence, executing effective privacy protection mechanisms against those retrieval software is essential for reliable social website construction. In this article, we propose a retrieval task-based adversarial perturbation generation method called Hashing Fake to meet this request. Specifically, DNNs are recently found to be vulnerable to a specific set of attacks called adversarial perturbations, which denote some magnitude-restricted signals added on objective samples to misguide well-crafted DNN models, and perturbations’ magnitudes are small enough that will not induce humans’ perception. Moreover, since existing adversarial perturbation generation methods are designed for supervised tasks, Hashing Fake constructs a differential approximation substitution for perturbation production on unsupervised retrieval tasks. Through extensive experiments on several deep retrieval benchmarks, we demonstrate that well-crafted perturbations using Hashing Fake can effectively misguide objective models’ recognitions to make false predictions. The added norm-restricted perturbations on objective samples will not alter humans’ perception; hence, Hashing Fake can be applied on real-world social websites to protect subscribers’ privacy against malicious retrieval software.

Annotate and retrieve in vivo images using hybrid self-organizing map

Multimodal retrieval has gained much attention lately due to its effectiveness over uni-modal retrieval. For instance, visual features often under-constrain the description of an image in content-based retrieval; however, another modality, such as collateral text, can be introduced to abridge the semantic gap and make the retrieval process more efficient. This article proposes the application of cross-modal fusion and retrieval on real in vivo gastrointestinal images and linguistic cues, as the visual features alone are insufficient for image description and to assist gastroenterologists. So, a cross-modal information retrieval approach has been proposed to retrieve related images given text and vice versa while handling the heterogeneity gap issue among the modalities. The technique comprises two stages: (1) individual modality feature learning; and (2) fusion of two trained networks. In the first stage, two self-organizing maps (SOMs) are trained separately using images and texts, which are clustered in the respective SOMs based on their similarity. In the second (fusion) stage, the trained SOMs are integrated using an associative network to enable cross-modal retrieval. The underlying learning techniques of the associative network include Hebbian learning and Oja learning (Improved Hebbian learning). The introduced framework can annotate images with keywords and illustrate keywords with images, and it can also be extended to incorporate more diverse modalities. Extensive experimentation has been performed on real gastrointestinal images obtained from a known gastroenterologist that have collateral keywords with each image. The obtained results proved the efficacy of the algorithm and its significance in aiding gastroenterologists in quick and pertinent decision making.

A Multi-Modal Retrieval Model for Mathematical Expressions Based on ConvNeXt and Hesitant Fuzzy Set

Mathematical expression retrieval is an essential component of mathematical information retrieval. Current mathematical expression retrieval research primarily targets single modalities, particularly text, which can lead to the loss of structural information. On the other hand, multimodal research has demonstrated promising outcomes across different domains, and mathematical expressions in image format are adept at preserving their structural characteristics. So we propose a multi-modal retrieval model for mathematical expressions based on ConvNeXt and HFS to address the limitations of single-modal retrieval. For the image modal, mathematical expression retrieval is based on the similarity of image features and symbol-level features of the expression, where image features of the expression image are extracted by ConvNeXt, while symbol-level features are obtained by the Symbol Level Features Extraction (SLFE) module. For the text modal, the Formula Description Structure (FDS) is employed to analyze expressions and extract their attributes. Additionally, the application of the Hesitant Fuzzy Set (HFS) theory facilitates the computation of hesitant fuzzy similarity between mathematical queries and candidate expressions. Finally, Reciprocal Rank Fusion (RRF) is employed to integrate rankings from image modal and text modal retrieval, yielding the ultimate retrieval list. The experiment was conducted on the publicly accessible ArXiv dataset (containing 592,345 mathematical expressions) and the NTCIR-mair-wikipedia-corpus (NTCIR) dataset.The MAP@10 values for the multimodal RRF fusion approach are recorded as 0.774. These substantiate the efficacy of the multi-modal mathematical expression retrieval approach based on ConvNeXt and HFS.

Towards Human–Machine Recognition Alignment: An Adversarilly Robust Multimodal Retrieval Hashing Framework

The multimodality nature of web data has necessitated complex multimodal information retrieval for a wide range of web applications. Deep neural networks (DNNs) have been widely employed to extract semantic features from raw samples to improve retrieval accuracy. In addition, hashing is widely used to improve computational and storage efficiency. As such, deep hashing frameworks have been applied for multimodal retrieval tasks. However, there is still a great recognitive gap between primate brain structure-inspired DNNs and humans. On computer vision tasks, well-crafted DNN models can be easily defeated by invisible small attacks, and this phenomenon indicates a large recognition gap between DNN models and humans. Recently, adversarial defense methods have been shown to improve the human&#x2013;machine recognition alignment in several classification tasks. However, the robustness problem on the retrieval tasks, especially on the deep hashing-based multimodal retrieval models, is still not well studied. Therefore, in this article, we present an adversarially robust training mechanism to improve model robustness for the purpose of human&#x2013;machine recognition alignment on retrieval tasks. Through extensive experimental results on several social multimodal retrieval benchmarks, we show that the robust training hashing framework proposed can mitigate the recognition gap on retrieval tasks. Our study highlights the necessity of robustness enhancement on deep hashing models.

Feature Fusion Based on Transformer for Cross-modal Retrieval

With the popularity of the Internet and the rapid growth of multimodal data, multimodal retrieval has gradually become a hot area of research. As one of the important branches of multimodal retrieval, image-text retrieval aims to design a model to learn and align two modal data, image and text, in order to build a bridge of semantic association between the two heterogeneous data, so as to achieve unified alignment and retrieval. The current mainstream image-text cross-modal retrieval approaches have made good progress by designing a deep learning-based model to find potential associations between different modal data. In this paper, we design a transformer-based feature fusion network to fuse the information of two modalities in the feature extraction process, which can enrich the semantic connection between the modalities. Meanwhile, we conduct experiments on the benchmark dataset Flickr30k and get competitive results, where recall at 10 achieves 96.2% accuracy in image-to-text retrieval.

One for more: Structured Multi-Modal Hashing for multiple multimedia retrieval tasks

Hashing has been widely studied in support of efficient multimedia retrieval due to its high computation and storage efficiency. However, existing multimedia hashing models are designed only for specific multimedia retrieval tasks, e.g., cross-modal retrieval and composite multi-modal retrieval. To simultaneously support multiple multimedia retrieval tasks, multiple different hashing models should be equipped in a multimedia retrieval system. This paper proposes a Structured Multi-Modal Hashing (SMMH) method, which can learn only a hash model for multiple multimedia retrieval tasks. Specifically, we generate structured hash codes by independently projecting the modality data into the Hamming space, where each part of hash codes preserves the specific characteristic of the specific modality data, and they are collaborated to represent the whole multi-modal data. To bridge the heterogeneous gap between multiple modalities, a fusion multi-modal space is constructed based on structured hash codes with label and pair-wise semantic supervision. Besides, we learn hash functions for composite multi-modal retrieval by projecting heterogeneous modalities into the collaborative multi-modal features. Finally, we develop a discrete hash optimization strategy to efficiently solve binary hash codes. Extensive experiments show that our model outperforms the strong baseline methods on multiple multimedia retrieval tasks. Source codes are available at https://github.com/ChaoqunZheng/SMMH.