Cross-modal Retrieval Methods Research Articles

Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-Modal Retrieval

Cross-modal retrieval aims to retrieve data in one modality by a query in another modality, which has been a very interesting research issue in the field of multimedia, information retrieval, and computer vision, and database. Most existing works focus on cross-modal retrieval between text-image, text-video, and lyrics-audio. Little research addresses cross-modal retrieval between audio and video due to limited audio-video paired datasets and semantic information. The main challenge of the audio-visual cross-modal retrieval task focuses on learning joint embeddings from a shared subspace for computing the similarity across different modalities, where generating new representations is to maximize the correlation between audio and visual modalities space. In this work, we propose TNN-C-CCA, a novel deep triplet neural network with cluster canonical correlation analysis, which is an end-to-end supervised learning architecture with an audio branch and a video branch. We not only consider the matching pairs in the common space but also compute the mismatching pairs when maximizing the correlation. In particular, two significant contributions are made. First, a better representation by constructing a deep triplet neural network with triplet loss for optimal projections can be generated to maximize correlation in the shared subspace. Second, positive examples and negative examples are used in the learning stage to improve the capability of embedding learning between audio and video. Our experiment is run over fivefold cross validation, where average performance is applied to demonstrate the performance of audio-video cross-modal retrieval. The experimental results achieved on two different audio-visual datasets show that the proposed learning architecture with two branches outperforms existing six canonical correlation analysis–based methods and four state-of-the-art-based cross-modal retrieval methods.

A novel strategy to balance the results of cross-modal hashing

Hashing methods for cross-modal retrieval has drawn increasing research interests and has been widely studied in recent years due to the explosive growth of multimedia big data. However, a significant phenomenon which has been ignored is that there is a large gap between the results of cross-modal hashing in most cases. For example, the results of Text-to-Image frequently outperform that of Image-to-Text with a large margin. In this paper, we propose a strategy named semantic augmentation to improve and balance the results of cross-modal hashing. An intermediate semantic space is constructed to re-align the feature representations that embedded with weak semantic information. By using the intermediate semantic space, the semantic information of visual features can be further augmented before being sent to cross-modal hashing algorithms. Extensive experiments are carried out on four datasets via seven state-of-the-art cross-modal hashing methods. Compared against the results without semantic augmentation, the Image-to-Text results of these methods with semantic augmentation are improved considerably, which demonstrates the effectiveness of the proposed semantic augmentation strategy in bridging the gap between the results of cross-modal retrieval. Additional experiments are conducted on the real-valued, semi-supervised, semi-paired, partial-paired, and unpaired cross-modal retrieval methods, the results further indicates the effectiveness of our strategy in improving performance of cross-modal retrieval.

Learning Feature Representation and Partial Correlation for Multimodal Multi-Label Data

User-provided annotations in existing multimodal datasets sometimes are inappropriate for model learning and can hinder the task of cross-modal retrieval. To handle this issue, we propose a discriminative and noise-robust cross-modal retrieval method, called FLPCL, which consists of deep feature learning and partial correlation learning. Deep feature learning is implemented by utilizing label supervised information to guide the training of deep neural network for each modality, which aims to find modality-specific deep feature representations that preserve the similarity and discrimination information among multimodal data. Based on deep feature learning, partial correlation learning is proposed to infer direct association between different modalities by removing the effect of common underlying semantics from each modality. It is achieved by maximizing the canonical correlation of the feature representations of different modalities conditioned on the label modality. Different from existing works that build indirect association between modalities via incorporating semantic labels, our FLPCL method can learn more effective and robust multimodal latent representations by explicitly preserving both intra-modal and inter-modal relationship among multimodal data. Extensive experiments on three cross-modal datasets show that our method outperforms state-of-the-art methods on cross-modal retrieval tasks.

Real-world Cross-modal Retrieval via Sequential Learning

Cross-modal retrieval is playing an increasingly important role in our daily life with the explosive growth of multimedia data. However, its learning paradigm under real-life environments is less studied, and most existing approaches are developed in the pre-desired settings (e.g., unchanging modalities and explicitly modal-aligned samples). Inspired by the recent achievement in the field of cognition mechanism on how the human brain acquires knowledge, we present a new sequential learning method for real-world cross-modal retrieval. In this method, a unified model is maintained to capture the common knowledge of various modalities but are learned in a sequential manner such that it behaves adaptively according to the evolving distribution of different modalities, and needs no laborious alignment operations among multimodal data before learning. Furthermore, we reformulate the objective of optimization-based meta-learning and propose a novel meta-learning method to overcome the catastrophic forgetting encountered in sequential learning. Extensive experiments are conducted on four popular image-text multimodal datasets and a five-modal dataset, showing that our method achieves state-of-the-art cross-modal retrieval performance without explicit modal-alignment.

Representation separation adversarial networks for cross-modal retrieval

Cross-modal retrieval aims to search the semantically similar instances from the other modalities by giving a query from one modality. Recently, generative adversarial networks (GANs) has been proposed to model the joint distribution over the data from different modalities and to learn the common representations for cross-modal retrieval. However, most of existing GANs-based methods simply project original representations of different modalities into a common representation space, and ignore the fact that different modalities share the common characteristics and on the other side each modality has the individual characteristics. To address this problem, in this paper, we propose a novel cross-modal retrieval method, called representation separation adversarial networks, which explicitly separates the original representations into common latent representations and private representations. Specifically, we minimize the correlation between the common representations and private representations to ensure independence of them. Then, we reconstruct the original representations via exchanging the common representations of different modalities to encourage the information swap. Finally, the labels are utilized to increase the discriminant of common representations. Comprehensive experimental results on two widely used datasets show that the proposed method achieved better performance than many existing GANs-based methods, and demonstrate that explicitly modeling the private representation for each modality can improve the model to extract common latent representations.

CMRDF: A Real-Time Food Alerting System Based on Multimodal Data

A healthy diet is a major concern for everyone, especially for those with specific diseases, such as diabetes. Meanwhile, with the rapid development of new technologies, it is feasible for us to detect the deep latent relationship between daily meals and wellbeing. Advanced Internet of Things devices, such as smart bracelets and wearable cameras, make it possible for people to know how food is related to health at any time. However, it is still arduous for individuals to memorize all the health information and utilize them to regulate their diet. To deal with such problems, we propose a novel system called cross-modal retrieval on diabetogenic food (CMRDF) which realizes a real-time dietary notice based on multimodal data captured from wearable devices. In this system, we propose a new graph-based cross-modal retrieval method named graph correlation analysis with ranking loss that finds the latent information in multimodal data. We use graph convolutional networks to dig the deep latent information in modalities and represent the data in finer granularity. It uses visual and physiological information to estimate whether the food that a user tries to obtain is diabetogenic or not, and feeds back the reasons in detail. Extensive experiments on the MSCOCO data set and the new proposed multimodal diabetogenic food database real-life diabetogenic show that the proposed cross-modal retrieval method outperforms state-of-the-art methods and CMRDF can achieve reliable results on preventing diabetic patients from inappropriate food.

Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

Cross-modal retrieval aims to find relevant data of different modalities, such as images and text. In order to bridge the modality gap, most existing methods require a lot of coupled sample pairs as training data. To reduce the demands for training data, we propose a cross-modal retrieval framework that utilizes both coupled and uncoupled samples. The framework consists of two parts: Abstraction that aims to provide high-level single-modal representations with uncoupled samples; then, Association links different modalities through a few coupled training samples. Moreover, under this framework, we implement a cross-modal retrieval method based on the consistency between the semantic structure of multiple modalities. First, both images and text are represented with the semantic structure-based representation, which represents each sample as its similarity from the reference points that are generated from single-modal clustering. Then, the reference points of different modalities are aligned through an active learning strategy. Finally, the cross-modal similarity can be measured with the consistency between the semantic structures. The experiment results demonstrate that given proper abstraction of single-modal data, the relationship between different modalities can be simplified, and even limited coupled cross-modal training data are sufficient for satisfactory retrieval accuracy.

Augmented Adversarial Training for Cross-Modal Retrieval

Cross-modal retrieval has received considerable attention in recent years. The core of cross-modal retrieval is to find a representation space to align data from different modalities according to their semantics. In this paper, we propose a cross-modal retrieval method that aligns data from different modalities by transferring one source modality to another target modality with augmented adversarial training. To preserve the semantic meaning in the modality transfer process, we employ the idea of conditional GANs and augment it. The key idea is to incorporate semantic information from the label space into the adversarial training process by sampling more semantic relevant and irrelevant source-target sample pairs. The augmented sample pairs improve the alignment from two aspects. First, relevant source-target sample pairs provide more training samples, leading to a better guidance of the alignment of fake targets and true paired targets. Second, relevant and irrelevant source-target sample pairs teach the discriminator to better distinguish true relevant pairs from fake relevant pairs, which guides the generator to better transfer from the source modality to the target modality. Extensive experiments compared with state-of-the-art methods show the promising power of our approach.

Combination subspace graph learning for cross-modal retrieval

In this paper, a novel supervised cross-modal retrieval method, combination subspace graph learning (CSGL), is proposed, which primarily concentrates on the cross retrieval between images and texts, i.e., image retrieving texts (I2T) and text retrieving images (T2I). To project multimodal data from a low-level feature space into a latent common subspace, most classical methods would learn the projective matrix separately for each mode, ignoring the consistency between different modes. Graph regularization is added to our objective function to preserve the structure of the original data in the projective space. Furthermore, to avoid the suboptimal solution during optimization, we use the collaborative learning strategy to obtain the projective matrix directly, which unites all the modes for better projection. Generally, the CSGL method takes the advantage of the semantic information and the original distribution of the image and text to obtain a more discriminative projection, which is learned in combination, rather than learning individually. Experimental results on three benchmark datasets, Wikipedia, Pascal Sentence, and INRIA-Websearch, show that the proposed method outperforms the state-of-the-art methods.

Cross-modal Hashing Retrieval Based on Density Clustering

Cross-modal hashing retrieval methods have attracted much attention for their effectiveness and efficiency. However, most of the existing hashing methods have the problem of how to precisely learn potential correlations between different modalities from binary codes with minimal loss. In addition, solving binary codes in different modalities is an NP-hard problem. To overcome these challenges, we initially propose a novel adaptive fast cross-modal hashing retrieval method under the inspiration of DBSCAN clustering algorithm, named Cross-modal Hashing Retrieval Based on Density Clustering(DCCH). DCCH utilizes the global density correlation between different modalities to select representative instances to replace the entire data precisely. Furthermore, DCCH excludes the adverse effects of noise points and leverages the discrete optimization process to obtain hash functions. The extensive experiments show that DCCH is superior to other state-of-the-art cross-modal methods on three benchmark bimodal datasets, i.e., Wiki, MIRFlickr and NUS-WIDE. Therefore, the experimental results also prove that our method DCCH is comparatively usable and efficient.

MESH: A Flexible Manifold-Embedded Semantic Hashing for Cross-Modal Retrieval

Hashing based methods for cross-modal retrieval has been widely explored in recent years. However, most of them mainly focus on the preservation of neighborhood relationship and label consistency, while ignore the proximity of neighbors and proximity of classes, which degrades the discrimination of hash codes. And most of them learn hash codes and hashing functions simultaneously, which limits the flexibility of algorithms. To address these issues, in this article, we propose a two-step cross-modal retrieval method named Manifold-Embedded Semantic Hashing (MESH). It exploits Local Linear Embedding to model the neighborhood proximity and uses class semantic embeddings to consider the proximity of classes. By so doing, MESH can not only extract the manifold structure in different modalities, but also can embed the class semantic information into hash codes to further improve the discrimination of learned hash codes. Moreover, the two-step scheme makes MESH flexible to various hashing functions. Extensive experimental results on three datasets show that MESH is superior to 10 state-of-the-art cross-modal hashing methods. Moreover, MESH also demonstrates superiority on deep features compared with the deep cross-modal hashing method.

Cross-modal retrieval by an end to end way

Cross-modal retrieval has attracted most attention in the recent years. For the image and text, how to measure the semantic similarity among them is still a challenging problem in cross-modal retrieval task. In our paper, we propose a cross-modal retrieval method that uses a CNN to obtain the semantic similarity between image and text. Most methods to solve this problem employ two separate parts for each modality to obtain the semantic similarity between them. However, in our work we just use a CNN to get the semantic similarity without having to use separate networks. In addition, we are aiming to solve the problem between long text and image, so we use the topic model to process the text. We evaluate our approach on Wikepedia dataset.

Semantic Consistency Cross-Modal Retrieval With Semi-Supervised Graph Regularization

Most of the existing cross-modal retrieval methods make use of labeled data to learn projection matrices for different modal data. These methods usually learn the original semantic space to bridge the heterogeneous gap, ignoring the rich semantic information contained in unlabeled data. Accordingly, a semantic consistency cross-modal retrieval with semi-supervised graph regularization (SCCMR) algorithm is proposed, which integrates the prediction of labels and the optimization of projection matrices into a unified framework to ensure that the solution obtained is globally optimal. At the same time, the method uses graph embedding to consider the nearest neighbors in the potential subspace of paired images and texts as well as images and texts with the same semantics. ${l_{21}}$ -norm constraint is applied to the projection matrices to select the discriminative features for different modal data. The results show that our method outperforms several advanced methods on four commonly used cross-modal retrieval datasets.

A Deep Hashing Technique for Remote Sensing Image-Sound Retrieval

With the rapid progress of remote sensing (RS) observation technologies, cross-modal RS image-sound retrieval has attracted some attention in recent years. However, these methods perform cross-modal image-sound retrieval by leveraging high-dimensional real-valued features, which can require more storage than low-dimensional binary features (i.e., hash codes). Moreover, these methods cannot directly encode relative semantic similarity relationships. To tackle these issues, we propose a new, deep, cross-modal RS image-sound hashing approach, called deep triplet-based hashing (DTBH), to integrate hash code learning and relative semantic similarity relationship learning into an end-to-end network. Specially, the proposed DTBH method designs a triplet selection strategy to select effective triplets. Moreover, in order to encode relative semantic similarity relationships, we propose the objective function, which makes sure that that the anchor images are more similar to the positive sounds than the negative sounds. In addition, a triplet regularized loss term leverages approximate l1-norm of hash-like codes and hash codes and can effectively reduce the information loss between hash-like codes and hash codes. Extensive experimental results showed that the DTBH method could achieve a superior performance to other state-of-the-art cross-modal image-sound retrieval methods. For a sound query RS image task, the proposed approach achieved a mean average precision (mAP) of up to 60.13% on the UCM dataset, 87.49% on the Sydney dataset, and 22.72% on the RSICD dataset. For RS image query sound task, the proposed approach achieved a mAP of 64.27% on the UCM dataset, 92.45% on the Sydney dataset, and 23.46% on the RSICD dataset. Future work will focus on how to consider the balance property of hash codes to improve image-sound retrieval performance.

Efficient discrete supervised hashing for large-scale cross-modal retrieval

Supervised cross-modal hashing has gained increasing research interest on large-scale retrieval task owning to its satisfactory performance and efficiency. However, there are still some issues to be further addressed: (1) most of them fail to capture the inherent data structure effectively due to the complex correlations among heterogeneous data points; (2) most of them obtain continuous solutions firstly and then quantize the continuous solutions to generate hash codes directly, which causes large quantization error and consequent suboptimal retrieval performance; (3) most of them suffer from relatively high memory cost and computational complexity during training procedure, which makes them unscalable. In this paper, to address above issues, we propose a supervised hashing method for cross-modal retrieval dubbed Efficient Discrete Supervised Hashing (EDSH). Specifically, the sharing space learning with collective matrix factorization and semantic embedding with class labels are seamlessly integrated to learn hash codes. Therefore, the feature based similarities and semantic correlations are both preserved in hash codes, which makes the learned hash codes more discriminative. Then an efficient discrete optimal scheme is designed to handle the scalable issue. Instead of learning hash codes bit-by-bit, hash codes matrix can be obtained directly which is more efficient. Extensive experimental results on three public datasets show that our EDSH produces a superior performance in both accuracy and scalability over several existing cross-modal hashing approaches.

A cross-modal multimedia retrieval method using depth correlation mining in big data environment

Cross-media retrieval is a technology aimed at breaking through the shackles of single-mode retrieval technology, which is limited to the same multimedia form. It is also hoped to be able to search each other across the media form. Comprehensive processing of different multimedia morphological data is an urgent problem to be solved in cross-media retrieval area, in other words, the semantic relationship between potential features should be mined, which will improve their similarity. To solve the above problems, a deep correlation mining method is proposed, which trains different media features by deep learning, and then fuses the correlation between the trained features to solve the heterogeneity between different features, which will make the features of different multimedia data comparable. On this basis, Levenberg-Marquart method is applied to solve the problem that deep learning is easy to fall into local minimum solution in gradient training. Experiments on different databases show that the proposed method is effective in the field of cross-media retrieval. Compared with other advanced multimedia retrieval methods, the proposed method has achieved better retrieval results.

Ternary Adversarial Networks With Self-Supervision for Zero-Shot Cross-Modal Retrieval.

Given a query instance from one modality (e.g., image), cross-modal retrieval aims to find semantically similar instances from another modality (e.g., text). To perform cross-modal retrieval, existing approaches typically learn a common semantic space from a labeled source set and directly produce common representations in the learned space for the instances in a target set. These methods commonly require that the instances of both two sets share the same classes. Intuitively, they may not generalize well on a more practical scenario of zero-shot cross-modal retrieval, that is, the instances of the target set contain unseen classes that have inconsistent semantics with the seen classes in the source set. Inspired by zero-shot learning, we propose a novel model called ternary adversarial networks with self-supervision (TANSS) in this paper, to overcome the limitation of the existing methods on this challenging task. Our TANSS approach consists of three paralleled subnetworks: 1) two semantic feature learning subnetworks that capture the intrinsic data structures of different modalities and preserve the modality relationships via semantic features in the common semantic space; 2) a self-supervised semantic subnetwork that leverages the word vectors of both seen and unseen labels as guidance to supervise the semantic feature learning and enhances the knowledge transfer to unseen labels; and 3) we also utilize the adversarial learning scheme in our TANSS to maximize the consistency and correlation of the semantic features between different modalities. The three subnetworks are integrated in our TANSS to formulate an end-to-end network architecture which enables efficient iterative parameter optimization. Comprehensive experiments on three cross-modal datasets show the effectiveness of our TANSS approach compared with the state-of-the-art methods for zero-shot cross-modal retrieval.

Self-supervised learning-based weight adaptive hashing for fast cross-modal retrieval

Due to the low storage cost and fast search speed, hashing is widely used in cross-modal retrieval. However, there still remain some crucial bottlenecks: Firstly, there are not suitable big datasets for multimodal data. Secondly, imbalance instances will affect the accuracy of the retrieval system. In this paper, we propose an end-to-end self-supervised learning-based weight adaptive hashing method for cross-modal retrieval. For the restriction of datasets, we use the self-supervised fashion to directly extract fine-grained features from labels and use them to supervise the hashing learning of other modalities. To overcome the problem of imbalance instances, we design an adaptive weight loss to flexibly adjust the weight of training samples according to their proportions. Besides these, we also use a binary approximation regularization term to reduce the regularization error. Experiments on MIRFLICKR-25K and NUS-WIDE datasets show that our method can improve 3% performance compared to other methods.

How much do cross-modal related semantics benefit image captioning by weighting attributes and re-ranking sentences?

Although image description generated from attributes achieves big progress, there are still two main problems to be solved further: (1) How to obtain more accurate attributes? (2) How to mitigate the difference between sentence generation and evaluation. To address these issues, we propose a new method to incorporate the cross-modal related semantics into the encoder-decoder structure for image captioning. In the encoding stage, we utilize the salient words derived from cross-modal retrieval to improve the accuracy of attributes. In the decoding stage, we explore two ways to re-rank the sentences generated through beam search with the guidance of semantics acquired through a modified cross-modal retrieval method. The evaluation results on the benchmark dataset MS-COCO in both offline and online prove the benefits of cross-modal related semantics on image captioning.

SCRATCH: A Scalable Discrete Matrix Factorization Hashing Framework for Cross-Modal Retrieval

In this paper, we present a novel supervised cross-modal hashing framework, namely Scalable disCRete mATrix faCtorization Hashing (SCRATCH). First, it utilizes collective matrix factorization on original features together with label semantic embedding, to learn the latent representations in a shared latent space. Thereafter, it generates binary hash codes based on the latent representations. During optimization, it avoids using a large $n\times n$ similarity matrix and generates hash codes discretely. Besides, based on different objective functions, learning strategy, and features, we further present three models in this framework, i.e., SCRATCH-o, SCRATCH-t, and SCRATCH-d. The first one is a one-step method, learning the hash functions and the binary codes in the same optimization problem. The second is a two-step method, which first generates the binary codes and then learns the hash functions based on the learned hash codes. The third one is a deep version of SCRATCH-t, which utilizes deep neural networks as hash functions. The extensive experiments on two widely used benchmark datasets demonstrate that SCRATCH-o and SCRATCH-t outperform some state-of-the-art shallow hashing methods for cross-modal retrieval. The SCRATCH-d also outperforms some state-of-the-art deep hashing models.