Cross-modal Retrieval Tasks Research Articles

Semantics-Aware Spatial-Temporal Binaries for Cross-Modal Video Retrieval.

With the current exponential growth of video-based social networks, video retrieval using natural language is receiving ever-increasing attention. Most existing approaches tackle this task by extracting individual frame-level spatial features to represent the whole video, while ignoring visual pattern consistencies and intrinsic temporal relationships across different frames. Furthermore, the semantic correspondence between natural language queries and person-centric actions in videos has not been fully explored. To address these problems, we propose a novel binary representation learning framework, named Semantics-aware Spatial-temporal Binaries ( [Formula: see text]Bin), which simultaneously considers spatial-temporal context and semantic relationships for cross-modal video retrieval. By exploiting the semantic relationships between two modalities, [Formula: see text]Bin can efficiently and effectively generate binary codes for both videos and texts. In addition, we adopt an iterative optimization scheme to learn deep encoding functions with attribute-guided stochastic training. We evaluate our model on three video datasets and the experimental results demonstrate that [Formula: see text]Bin outperforms the state-of-the-art methods in terms of various cross-modal video retrieval tasks.

Wildfish++: A Comprehensive Fish Benchmark for Multimedia Research

In this paper, we develop a large-scale vision-language fish benchmark, namely WildFish++, for comprehensive studies in multimedia research. Concretely, WildFish++ consists of 2,348 fish categories with 103,034 images in the wild, and 3,817 fish descriptions with 213,858 words. Based on these distinct characteristics, we mainly introduce four challenging research tasks on WildFish++. (1) Fine-Grained Recognition with Comparison Texts . WildFish++ naturally contains subtle difference among fish categories, which leads to fine-grained classification. Most approaches resort to tackle this problem by capturing discriminative regions in the view of each image. However, this paradigm may be still far way from extracting the most distinct features when the context on visual difference is not available. In this case, we propose to introduce comparison fish descriptions, a unique corpus that can directly point out subtle difference between highly-confused species and naturally serve as a kind of valuable context information. With such texts, we creatively elaborate a multi-modal fish network, aiming at incorporating those comparison textual information as prior knowledge and consequently leveraging it to guide CNNs to find subtle yet distinct regions in the context of comparison texts. (2) Open-Set Classification . We often confront with unknown categories in practice, e.g., there may still exist unknown fishes in our planet. Hence, we creatively adapt WildFish++ for a novel open-set classification task, which aims at correctly assigning each test image into the unknown class or one of known classes. More importantly, we investigate a number of practical designs to boost accuracy of deep learning models in open-set scenarios. (3) Cross-Modal Retrieval . WildFish++ not only contains diversified fish images in the wild but also has rich fish descriptions about morphology diagnosis, biology information, etc. Hence, we design a challenging cross-modal retrieval task, which leverages three subtasks such as text-to-text, text-to-image, image-to-text retrieval in a unified end-to-end framework. (4) Automatic Fish Classification . Automatic fish classification is a long-term research in marine biology, while current studies are unsatisfactory due to the lack of large-scale data. In this case, we train a number of CNNs with WildFish++, and use its pre-trained models to boost fish classification on most existing benchmarks of wild fishes. We will release WildFish++ with codes/protocols ( https://github.com/PeiqinZhuang/WildFish++ ). We believe it can promote relevant studies in multimedia and beyond.

Hybrid Attention Network for Language-Based Person Search.

Language-based person search retrieves images of a target person using natural language description and is a challenging fine-grained cross-modal retrieval task. A novel hybrid attention network is proposed for the task. The network includes the following three aspects: First, a cubic attention mechanism for person image, which combines cross-layer spatial attention and channel attention. It can fully excavate both important midlevel details and key high-level semantics to obtain better discriminative fine-grained feature representation of a person image. Second, a text attention network for language description, which is based on bidirectional LSTM (BiLSTM) and self-attention mechanism. It can better learn the bidirectional semantic dependency and capture the key words of sentences, so as to extract the context information and key semantic features of the language description more effectively and accurately. Third, a cross-modal attention mechanism and a joint loss function for cross-modal learning, which can pay more attention to the relevant parts between text and image features. It can better exploit both the cross-modal and intra-modal correlation and can better solve the problem of cross-modal heterogeneity. Extensive experiments have been conducted on the CUHK-PEDES dataset. Our approach obtains higher performance than state-of-the-art approaches, demonstrating the advantage of the approach we propose.

Semi-Supervised Cross-Modal Retrieval With Label Prediction

Due to abundance of data from multiple modalities, cross-modal retrieval tasks with image-text, audio-image, etc. are gaining increasing importance. Of the different approaches proposed, supervised methods usually give significant improvement over their unsupervised counterparts at the additional cost of labeling or annotation of the training data. Semi-supervised methods are recently becoming popular as they provide an elegant framework to balance the conflicting requirement of labeling cost and accuracy. In this work, we propose a novel deep semi-supervised framework which can seamlessly handle both labeled as well as unlabeled data. The network has two important components: (a) the label prediction component predicts the labels for the unlabeled portion of the data and then (b) a common modality-invariant representation is learned for cross-modal retrieval. The two parts of the network are trained sequentially one after the other. Extensive experiments on three standard benchmark datasets, Wiki, Pascal VOC and NUS-WIDE demonstrate that the proposed framework outperforms the state-of-the-art for both supervised and semi-supervised settings.

DRSL: Deep Relational Similarity Learning for Cross-modal Retrieval

Cross-modal retrieval aims to retrieve relevant samples across different media modalities. Existing cross-modal retrieval approaches are contingent on learning common representations of all modalities by assuming that an equal amount of information exists in different modalities. However, since the quantity of information among cross-modal samples is unbalanced and unequal, it is inappropriate to directly match the obtained modality-specific representations across different modalities in a common space. In this paper, we propose a new method called Deep Relational Similarity Learning (DRSL) for cross-modal retrieval. Unlike existing approaches, the proposed DRSL aims to effectively bridge the heterogeneity gap of different modalities by directly learning the natural pairwise similarities instead of explicitly learning a common space. DRSL is a deep hybrid framework that integrates the relation networks module for relation learning, capturing the implicit nonlinear distance metric. To the best of our knowledge, DRSL is the first approach that incorporates relation networks into the cross-modal learning scenario. Comprehensive experimental results show that the proposed DRSL model achieves state-of-the-art results in cross-modal retrieval tasks on four widely-used benchmark datasets, i.e., Wikipedia, Pascal Sentences, NUS-WIDE-10K, and XMediaNet.

Cross-modal retrieval via label category supervised matrix factorization hashing

• The proposed model can achieve efficient and accurate cross-modal retrieval task. • The proposed model integrates data similarity and category information to improve performance. • The author validates the effectiveness of the proposed model through experiments on several public data sets. Due to the emergence and development of big data, cross-modal hash retrieval has become progressively more important in large-scale multi-modal retrieval tasks depending on its accuracy and efficiency. It completes the retrieval task in a common low-dimensional space by finding a common semantic space for heterogeneous data of different modalities. Recently, many works have concentrated on supervised cross-modal hashing and achieved higher retrieval accuracy. However, there are still many challenges in how to maintain the local geometric structure of the original space in the public space and how to use the supervision information efficiently. To deal with such issues, this paper proposes a hash retrieval method that incorporates supervised information based on matrix factorization (LCSMFH) by maintain the inter-modal and the intra-modal similarity in the original space and make the most of the label information to improve the retrieval task effect. Through experiments on two benchmark data sets, our method is more effective and outperforms state-of-the-art methods.

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-Modal Retrieval

Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio. Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of the audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose TNN-C-CCA, a novel deep triplet neural network with cluster canonical correlation analysis, which is an end-to-end supervised learning architecture with an audio branch and a video branch. We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made. First, a better representation by constructing a deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. Second, positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over fivefold cross validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show that the proposed learning architecture with two branches outperforms existing six canonical correlation analysis–based methods and four state-of-the-art-based cross-modal retrieval methods.

Adversarial Tri-Fusion Hashing Network for Imbalanced Cross-Modal Retrieval

Cross-modal retrieval has received increasing attentions for efficient retrieval across different modalities, and hashing technique has made significant progress recently due to its low storage cost and high query speed. However, most existing cross-modal hashing works still face the challenges of narrowing down the semantic gap between different modalities and training with imbalanced multi-modal data. This article presents an efficient Adversarial Tri-Fusion Hashing Network (ATFH-N) for cross-modal retrieval, which lies among the early attempts to incorporate adversarial learning for working with imbalanced multi-modal data. Specifically, a triple fusion network associated with zero padding operation is proposed to adapt either balanced or imbalanced multi-modal training data. At the same time, an adversarial training mechanism is leveraged to maximally bridge the semantic gap of the common representations between balanced and imbalanced data. Further, a label prediction network is utilized to guide the feature learning process and promote hash code learning, while additionally embedding the manifold structure to preserve both inter-modal and intra-modal similarities. Through the joint exploitation of the above, the underlying semantic structure of multimedia data can be well preserved in Hamming space, which can benefit various cross-modal retrieval tasks. Extensive experiments on three benchmark datasets show that the proposed ATFH-N method yields the comparable performance in balanced scenario and brings substantial improvements over the state-of-the-art methods in imbalanced scenarios.

Learning Feature Representation and Partial Correlation for Multimodal Multi-Label Data

User-provided annotations in existing multimodal datasets sometimes are inappropriate for model learning and can hinder the task of cross-modal retrieval. To handle this issue, we propose a discriminative and noise-robust cross-modal retrieval method, called FLPCL, which consists of deep feature learning and partial correlation learning. Deep feature learning is implemented by utilizing label supervised information to guide the training of deep neural network for each modality, which aims to find modality-specific deep feature representations that preserve the similarity and discrimination information among multimodal data. Based on deep feature learning, partial correlation learning is proposed to infer direct association between different modalities by removing the effect of common underlying semantics from each modality. It is achieved by maximizing the canonical correlation of the feature representations of different modalities conditioned on the label modality. Different from existing works that build indirect association between modalities via incorporating semantic labels, our FLPCL method can learn more effective and robust multimodal latent representations by explicitly preserving both intra-modal and inter-modal relationship among multimodal data. Extensive experiments on three cross-modal datasets show that our method outperforms state-of-the-art methods on cross-modal retrieval tasks.

Cross-modal learning with prior visual relation knowledge

Visual relational reasoning is a central component in recent cross-modal analysis tasks, which aims at reasoning about the visual relationships between objects and their properties. These relationships provide rich semantics and help to enhance the visual representation for improving cross-modal learning. Previous works have succeeded in modeling latent visual relationships or rigid-categorized visual relationships. However, these kinds of methods leave out the problem of ambiguity inherent in the visual relationships because of the diverse relational semantics of different visual appearances. In this work, we explore to model the visual relationships by context-aware representations based on human prior knowledge. Based on such representations, we novelly propose a plug-and-play visual relational reasoning module to enhance image encoding. Specifically, we design an Anisotropic Graph Convolution to utilize the information of relation embeddings and relation directionality between objects for generating relation-aware image representations. We demonstrate the effectiveness of the relational reasoning module by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments are conducted on VQA 2.0 and CMPlaces datasets and superior performance is reported when comparing with state-of-the-art works.

Reasoning on the Relation: Enhancing Visual Representation for Visual Question Answering and Cross-Modal Retrieval

Cross-modal analysis has become a promising direction for artificial intelligence. Visual representation is crucial for various cross-modal analysis tasks that require visual content understanding. Visual features which contain semantical information can disentangle the underlying correlation between different modalities, thus benefiting the downstream tasks. In this paper, we propose a Visual Reasoning and Attention Network (VRANet) as a plug-and-play module to capture rich visual semantics and help to enhance the visual representation for improving cross-modal analysis. Our proposed VRANet is built based on the bilinear visual attention module which identifies the critical objects. We propose a novel Visual Relational Reasoning (VRR) module to reason about pair-wise and inner-group visual relationships among objects guided by the textual information. The two modules enhance the visual features at both relation level and object level. We demonstrate the effectiveness of the proposed VRANet by applying it to both Visual Question Answering (VQA) and Cross-Modal Information Retrieval (CMIR) tasks. Extensive experiments conducted on VQA 2.0, CLEVR, CMPlaces, and MS-COCO datasets indicate superior performance comparing with state-of-the-art work.

Exploiting Subspace Relation in Semantic Labels for Cross-Modal Hashing

Hashing methods have been extensively applied to efficient multimedia data indexing and retrieval on account of the explosion of multimedia data. Cross-modal hashing usually learns binary codes by mapping multi-modal data into a common Hamming space. Most supervised methods utilize relation information like class labels as pairwise similarities of cross-modal data pair to narrow intra-modal and inter-modal gap. In this paper, we propose a novel supervised cross-modal hashing method dubbed Subspace Relation Learning for Cross-modal Hashing (SRLCH), which exploits relation information of labels in semantic space to make similar data from different modalities closer in the low-dimension Hamming subspace. SRLCH preserves the modality relationships, the discrete constraints and nonlinear structures, while admitting a closed-form binary codes solution, which effectively enhances the training efficiency. An iterative alternative optimization algorithm is developed to simultaneously learn both hash functions and unified binary codes. With these binary codes and hash functions, we can index multimedia data and search them in an efficient way. Evaluations in two cross-modal retrieval tasks on several widely-used datasets show that the proposed SRLCH outperforms most cross-modal hashing methods. Theoretical analysis also illustrates reasons for our method’s promotion in subspace relation learning.

Cross-modal retrieval by an end to end way

Cross-modal retrieval has attracted most attention in the recent years. For the image and text, how to measure the semantic similarity among them is still a challenging problem in cross-modal retrieval task. In our paper, we propose a cross-modal retrieval method that uses a CNN to obtain the semantic similarity between image and text. Most methods to solve this problem employ two separate parts for each modality to obtain the semantic similarity between them. However, in our work we just use a CNN to get the semantic similarity without having to use separate networks. In addition, we are aiming to solve the problem between long text and image, so we use the topic model to process the text. We evaluate our approach on Wikepedia dataset.

Progressive Cross-Modal Semantic Network for Zero-Shot Sketch-Based Image Retrieval.

Zero-shot sketch-based image retrieval (ZS-SBIR) is a specific cross-modal retrieval task that involves searching natural images through the use of free-hand sketches under the zero-shot scenario. Most previous methods project the sketch and image features into a low-dimensional common space for efficient retrieval, and meantime align the projected features to their semantic features (e.g., category-level word vectors) in order to transfer knowledge from seen to unseen classes. However, the projection and alignment are always coupled; as a result, there is a lack of alignment that consequently leads to unsatisfactory zero-shot retrieval performance. To address this issue, we propose a novel progressive cross-modal semantic network. More specifically, it first explicitly aligns the sketch and image features to semantic features, then projects the aligned features to a common space for subsequent retrieval. We further employ cross-reconstruction loss to encourage the aligned features to capture complete knowledge about the two modalities, along with multi-modal Euclidean loss that guarantees similarity between the retrieval features from a sketch-image pair. Extensive experiments conducted on two popular large-scale datasets demonstrate that our proposed approach outperforms state-of-the-art competitors to a remarkable extent: by more than 3% on the Sketchy dataset and about 6% on the TU-Berlin dataset in terms of retrieval accuracy.

Rich Features Embedding for Cross-Modal Retrieval: A Simple Baseline

During the past few years, significant progress has been made on cross-modal retrieval, benefiting from the development of deep neural networks. Meanwhile, the overall frameworks are becoming more and more complex, making the training as well as the analysis more difficult. In this paper, we provide a Rich Features Embedding (RFE) approach to tackle the cross-modal retrieval tasks in a simple yet effective way. RFE proposes to construct rich representations for both images and texts, which is further leveraged to learn the rich features embedding in the common space according to a simple hard triplet loss. Without any bells and whistles in constructing complex components, the proposed RFE is concise and easy to implement. More importantly, our RFE obtains the state-of-the-art results on several popular benchmarks such as MS COCO and Flickr 30 K. In particular, the image-to-text and text-to-image retrieval achieve 76.1% and 61.1% (R@1) on MS COCO, which outperform others more than 3.4% and 2.3%, respectively. We hope our RFE will serve as a solid baseline and help ease future research in cross-modal retrieval.

Supervised Robust Discrete Multimodal Hashing for Cross-Media Retrieval

The hashing-based approximate nearest neighbors search is able to reduce the storage cost and improve query speed. Therefore, they have attracted much attention in these years. Moreover, some hashing methods have been proposed for cross-modal retrieval tasks. However, there are still some issues that need to be further addressed. For example, some of them only construct a simple similarity matrix when learning hash functions or binary codes, which may lose some useful information. Some of them solve the hard discrete optimization problem by relaxing the binary constraints and quantizing the solution to obtain the final results, which may generate large quantization errors. To address these challenges, we present a new supervised cross-modal hashing method, named supervised robust discrete multimodal hashing (SRDMH). Specifically, it incorporates full label information into the hash functions learning to preserve the similarity in the original space. In addition, instead of relaxing the binary constraints, it is able to learn the binary codes and hash functions simultaneously. Moreover, it adopts a flexible ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2,p</sub> loss with nonlinear kernel embedding and introduces an intermediate presentation of the binary codes. In light of this, it becomes more robust and easier to solve by an iterative algorithm presented in this paper. To evaluate its performance, we conduct extensive experiments on three benchmark datasets. The results verify that SRDMH outperforms seven state-of-the-art cross-modal hashing methods. In addition, we also extend it to the classification task. Compared with other hashing methods, SRDMH also obtains better results when its binary codes are used for classification.

Label consistent locally linear embedding based cross-modal hashing

Hashing methods have gained widespread attention in cross-modal retrieval applications due to their efficiency and effectiveness. Many works have been done but they fail to capture the feature based similarity consistency or the discriminative semantics of label consistency. In addition, most of them suffer from large quantization loss, resulting in low retrieval performance. To address these issues, we propose a novel cross-modal hashing method named Label Consistent Locally Linear Embedding based Cross-modal Hashing (LCLCH). LCLCH preserves the non-linear manifold structure of different modality data by Locally Linear Embedding, and transforms heterogeneous data into a latent common semantic space to reduce the semantic gap and support cross-modal retrieval tasks. Therefore, it not only discovers the potential correlation of heterogeneous cross-modal data but also maintains label consistency. To further ensure the effectiveness of hash code learning, we utilize an iterative quantization method to handle the discrete optimization task and obtain the hash codes directly. We compare LCLCH with some advanced supervised and unsupervised methods on three datasets to evaluate its effectiveness.

Dual-supervised attention network for deep cross-modal hashing

Cross-modal hashing has received intensive attention due to its low computation and storage efficiency in cross-modal retrieval task. Most previous cross-modal hashing methods mainly focus on extracting correlated binary codes from the pairwise label, but largely ignore the semantic categories of cross-modal data. On the other hand, human perception exploits category information to connect cross-modal samples. Inspired by this fact, we propose to embed category information into hash codes. More specifically, we introduce semantic prediction loss into our framework to enhance hash codes with category supervision. In addition, there always exists a large gap between features from different modalities (e.g. text and images), leading cross-modal hashing to link irrelevant features for retrieval task. To address this issue, this paper proposes Dual-Supervised Attention Network for Deep Hashing (DSADH) to learn the cross-modal relationship via an elaborately-designed attention mechanism. Our cross-modal network applies cross-modal attention block to efficiently encode rich and relevant features to learn compact hash codes. Extensive experiments on three challenging benchmarks demonstrate that our proposed method significantly improves the retrieval results.

A New Approach to Cross-Modal Retrieval

The semantic correlation matching algorithm, which attempts to integrate semantic information based on the unified space obtained by CCA learning, achieves an improvement in the effect of cross-modal retrieval. However, there is still a lot of room for the algorithm to make further progress in the generation of shared subspace. This paper proposes a semantic correlation matching algorithm based on the single-layer feed forward network. It generates features through neural networks to represent weight and bias. Meanwhile, it maximizes the correlation between different modalities by means of cross-coupling. The experimental results show that the algorithm proposed in this paper is superior to the compared method in terms of cross-modal retrieval task of image-text and has better cross-modal retrieval ability.

Adversarial task-specific learning

In this paper, we investigate a principle way to learn a common feature space for data of different modalities (e.g. image and text), so that the similarity between different modal items can be directly measured for benefiting cross-modal retrieval task. To effectively keep semantic/distribution consistent for common feature embeddings, we propose a new Adversarial Task-Specific Learning (ATSL) approach to learn distinct embeddings for different retrieval tasks, i.e. images retrieve texts (I2T) or texts retrieve images (T2I). In particular, the proposed ATSL is with the following advantages: (a) semantic attributes are leveraged to encourage the learned common feature embeddings of couples to be semantic consistent; (b) adversarial learning is applied to relieve the inconsistent distribution of common feature embeddings for different modalities; (c) triplet optimization is employed to guarantee that similar items from different modalities are with smaller distances in the learned common space compared with the dissimilar ones; (d) task-specific learning produces better optimized common feature embeddings for different retrieval tasks. Our ATSL is embedded in a deep neural network, which can be learned in an end-to-end manner. We conduct extensive experiments on two popular benchmark datasets, e.g. Flickr30K and MS COCO. We achieve R@1 accuracy of 57.1% and 38.4% for I2T and 56.5% and 38.6% T2I on MS COCO and Flickr30K respectively, which are the new state-of-the-arts.