Cross-modal Similarity Search Research Articles

Two-Stage Asymmetric Similarity Preserving Hashing for Cross-Modal Retrieval

Hashing-based techniques present appealing solutions for cross-modal retrieval due to its low storage requirements and excellent query efficiency. The majority of cross-modal hashing methods typically adopt equal-length encoding scheme to represent multimodal data and achieve cross-modal similarity search. However, such scheme can be regarded as a relatively strict limitation, because it sacrifices the flexible representation of multimodal data in reality and cannot always guarantee the optimal retrieval performance. To address the challenge, this paper focuses on encoding heterogeneous data with varying hash lengths. To achieve this purpose, we propose a flexible cross-modal hashing approach, named Two-stage Asymmetric Similarity Preserving Hashing, TASPH for short, which can be applied to both unequal-length and equal-length retrieval scenarios. Specifically, in the first stage, TASPH designs a novel discrete asymmetric strategy to learn the modality-specific hash codes with varying lengths, enabling a flexible representation of heterogeneous data. Simultaneously, TASPH utilizes two semantic transformation matrices to establish the semantic correlations between varying hash codes. Different from most of the existing approaches that employ relaxation solutions, TASPH satisfies the discrete constraints without any relaxation. In the second stage, the learned semantic transformation matrices are employed to alleviate cross-modal heterogeneity, which guarantees that TASPH can learn more powerful hash functions to improve the discriminative ability of hash codes. Abundant experiments conducted on three benchmark datasets demonstrate encouraging results compared with the state-of-the-art approaches under different retrieval scenarios.

Robust and discrete matrix factorization hashing for cross-modal retrieval

Hashing based methods have gained great success for cross-modal similarity search, due to its fast query speed and low storage cost. However, there are some challenging problems that need to be further solved: 1) Many approaches are sensitive to noises and outliers, because ℓ2 norm is utilized in the objective function, the error may be amplified. 2) Most existing methods take relaxation or rounding scheme to generate binary codes, causing a large quantization loss. 3) Many supervised cross-media algorithms usually take a large n×n matrix to preserve the similarity relationship, leading to large calculation and making them unscalable. To mitigate these challenges, we develop a novel cross-media search algorithm, i.e., robust and discrete matrix factorization hashing, dubbed RDMH. The method takes a two-step strategy. In the first phase, the ℓ2,1 norm is utilized to improve the robustness, which makes our model not sensitive to noises and outliers. We can learn the hash codes directly by the proposed discrete optimization method instead of relaxation scheme, avoiding the large quantization loss. Moreover, RDMH correlates the hash codes and semantic labels directly instead of manipulating the large similarity matrix. In the second phase, we propose an autoencoder strategy to learn the hash functions, more valuable information can be preserved and making the hash functions more powerful. Comprehensive experiments on several databases demonstrate the superior performance and efficacy of the developed RDMH.

Asymmetric Supervised Consistent and Specific Hashing for Cross-Modal Retrieval.

Hashing-based techniques have provided attractive solutions to cross-modal similarity search when addressing vast quantities of multimedia data. However, existing cross-modal hashing (CMH) methods face two critical limitations: 1) there is no previous work that simultaneously exploits the consistent or modality-specific information of multi-modal data; 2) the discriminative capabilities of pairwise similarity is usually neglected due to the computational cost and storage overhead. Moreover, to tackle the discrete constraints, relaxation-based strategy is typically adopted to relax the discrete problem to the continuous one, which severely suffers from large quantization errors and leads to sub-optimal solutions. To overcome the above limitations, in this article, we present a novel supervised CMH method, namely Asymmetric Supervised Consistent and Specific Hashing (ASCSH). Specifically, we explicitly decompose the mapping matrices into the consistent and modality-specific ones to sufficiently exploit the intrinsic correlation between different modalities. Meanwhile, a novel discrete asymmetric framework is proposed to fully explore the supervised information, in which the pairwise similarity and semantic labels are jointly formulated to guide the hash code learning process. Unlike existing asymmetric methods, the discrete asymmetric structure developed is capable of solving the binary constraint problem discretely and efficiently without any relaxation. To validate the effectiveness of the proposed approach, extensive experiments on three widely used datasets are conducted and encouraging results demonstrate the superiority of ASCSH over other state-of-the-art CMH methods.

Cluster-wise unsupervised hashing for cross-modal similarity search

Cross-modal hashing similarity retrieval plays dual roles across various applications including search engines and autopilot systems. More generally, these methods also known to reduce the computation and memory storage in a training scheme. The key limitation of current methods are that: (i) they relax the discrete constrains to solve the optimization problem which may defeat the model purpose, (ii) projecting heterogenous data into a latent space may encourage to loss the diverse representations in such data, (iii) transforming real-valued data point to the binary codes always resulting in a loss of information and producing the suboptimal continuous latent space. In this paper, we propose a novel framework to project the original data points from different modalities into its own low-dimensional latent space and finds the cluster centroid points in its a low-dimensional space, using Cluster-wise Unsupervised Hashing (CUH). In particular, the proposed clustering scheme aims to jointly learns the compact hash codes and the corresponding linear hash functions. A discrete optimization framework is developed to learn the unified binary codes across modalities under of the guidance cluster-wise code-prototypes. Extensive experiments over multiple datasets demonstrate the effectiveness of our proposed model in comparison with the state-of-the-art in unsupervised cross-modal hashing tasks.

Discriminative Supervised Hashing for Cross-Modal Similarity Search

With the advantages of low storage cost and high retrieval efficiency, hashing techniques have recently been an emerging topic in cross-modal similarity search. As multiple modal data reflect similar semantic content, many works aim at learning unified binary codes. However, discriminative hashing features learned by these methods are not adequate. This results in lower accuracy and robustness. We propose a novel hashing learning framework which jointly performs classifier learning, subspace learning, and matrix factorization to preserve class-specific semantic content, termed Discriminative Supervised Hashing (DSH), to learn the discriminative unified binary codes for multi-modal data. Besides, reducing the loss of information and preserving the non-linear structure of data, DSH non-linearly projects different modalities into the common space in which the similarity among heterogeneous data points can be measured. Extensive experiments conducted on the three publicly available datasets demonstrate that the framework proposed in this paper outperforms several state-of-the-art methods.

Discrete Latent Factor Model for Cross-Modal Hashing.

Due to its storage and retrieval efficiency, cross-modal hashing (CMH) has been widely used for cross-modal similarity search in many multimedia applications. According to the training strategy, existing CMH methods can be mainly divided into two categories: relaxation-based continuous methods and discrete methods. In general, the training of relaxation-based continuous methods is faster than that of discrete methods, but the accuracy of relaxation-based continuous methods is not satisfactory. On the contrary, the accuracy of discrete methods is typically better than that of the relaxation-based continuous methods, but the training of discrete methods is very time-consuming. In this paper, we propose a novel CMH method, called Discrete Latent Factor model-based cross-modal Hashing (DLFH), for cross modal similarity search. DLFH is a discrete method which can directly learn the binary hash codes for CMH. At the same time, the training of DLFH is efficient. Experiments show that the DLFH can achieve significantly better accuracy than existing methods, and the training time of DLFH is comparable to that of the relaxation-based continuous methods which are much faster than the existing discrete methods.

Deep Semantic-Preserving Ordinal Hashing for Cross-Modal Similarity Search.

Cross-modal hashing has attracted increasing research attention due to its efficiency for large-scale multimedia retrieval. With simultaneous feature representation and hash function learning, deep cross-modal hashing (DCMH) methods have shown superior performance. However, most existing methods on DCMH adopt binary quantization functions (e.g., [Formula: see text]) to generate hash codes, which limit the retrieval performance since binary quantization functions are sensitive to the variations of numeric values. Toward this end, we propose a novel end-to-end ranking-based hashing framework, in this paper, termed as deep semantic-preserving ordinal hashing (DSPOH), to learn hash functions with deep neural networks by exploring the ranking structure of feature dimensions. In DSPOH, the ordinal representation, which encodes the relative rank ordering of feature dimensions, is explored to generate hash codes. Such ordinal embedding benefits from the numeric stability of rank correlation measures. To make the hash codes discriminative, the ordinal representation is expected to well predict the class labels so that the ranking-based hash function learning is optimally compatible with the label predicting. Meanwhile, the intermodality similarity is preserved to guarantee that the hash codes of different modalities are consistent. Importantly, DSPOH can be effectively integrated with different types of network architectures, which demonstrates the flexibility and scalability of our proposed hashing framework. Extensive experiments on three widely used multimodal data sets show that DSPOH outperforms state of the art for cross-modal retrieval tasks.

Robust and Flexible Discrete Hashing for Cross-Modal Similarity Search

Multimodal hashing approaches have gained great success on large-scale cross-modal similarity search applications, due to their appealing computation and storage efficiency. However, it is still a challenge work to design binary codes to represent the original features with good performance in an unsupervised manner. We argue that there are some limitations that need to be further considered for unsupervised multimodal hashing: 1) most existing methods drop the discrete constraints to simplify the optimization, which will cause large quantization error; 2) many methods are sensitive to outliers and noises since they use $\ell _{2}$ -norm in their objective functions which can amplify the errors; and 3) the weight of each modality, which greatly influences the retrieval performance, is manually or empirically determined and may not fully fit the specific training set. The above limitations may significantly degrade the retrieval accuracy of unsupervised multimodal hashing methods. To address these problems, in this paper, a novel hashing model is proposed to efficiently learn robust discrete binary codes, which is referred as Robust and Flexible Discrete Hashing (RFDH). In the proposed RFDH model, binary codes are directly learned based on discrete matrix decomposition, so that the large quantization error caused by relaxation is avoided. Moreover, the $\ell _{2,1}$ -norm is used in the objective function to improve the robustness, such that the learned model is not sensitive to data outliers and noises. In addition, the weight of each modality is adaptively adjusted according to training data. Hence the important modality will get large weights during the hash learning procedure. Owing to above merits of RFDH, it can generate more effective hash codes. Besides, we introduce two kinds of hash function learning methods to project unseen instances into hash codes. Extensive experiments on several well-known large databases demonstrate superior performance of the proposed hash model over most state-of-the-art unsupervised multimodal hashing methods.

Label Consistent Matrix Factorization Hashing for Large-Scale Cross-Modal Similarity Search.

Multimodal hashing has attracted much interest for cross-modal similarity search on large-scale multimedia data sets because of its efficiency and effectiveness. Recently, supervised multimodal hashing, which tries to preserve the semantic information obtained from the labels of training data, has received considerable attention for its higher search accuracy compared with unsupervised multimodal hashing. Although these algorithms are promising, they are mainly designed to preserve pairwise similarities. When semantic labels of training data are given, the algorithms often transform the labels into pairwise similarities, which gives rise to the following problems: (1) constructing pairwise similarity matrix requires enormous storage space and a large amount of calculation, making these methods unscalable to large-scale data sets; (2) transforming labels into pairwise similarities loses the category information of the training data. Therefore, these methods do not enable the hash codes to preserve the discriminative information reflected by labels and, hence, the retrieval accuracies of these methods are affected. To address these challenges, this paper introduces a simple yet effective supervised multimodal hashing method, called label consistent matrix factorization hashing (LCMFH), which focuses on directly utilizing semantic labels to guide the hashing learning procedure. Considering that relevant data from different modalities have semantic correlations, LCMFH transforms heterogeneous data into latent semantic spaces in which multimodal data from the same category share the same representation. Therefore, hash codes quantified by the obtained representations are consistent with the semantic labels of the original data and, thus, can have more discriminative power for cross-modal similarity search tasks. Thorough experiments on standard databases show that the proposed algorithm outperforms several state-of-the-art methods.

Shared Predictive Cross-Modal Deep Quantization.

With explosive growth of data volume and ever-increasing diversity of data modalities, cross-modal similarity search, which conducts nearest neighbor search across different modalities, has been attracting increasing interest. This paper presents a deep compact code learning solution for efficient cross-modal similarity search. Many recent studies have proven that quantization-based approaches perform generally better than hashing-based approaches on single-modal similarity search. In this paper, we propose a deep quantization approach, which is among the early attempts of leveraging deep neural networks into quantization-based cross-modal similarity search. Our approach, dubbed shared predictive deep quantization (SPDQ), explicitly formulates a shared subspace across different modalities and two private subspaces for individual modalities, and representations in the shared subspace and the private subspaces are learned simultaneously by embedding them to a reproducing kernel Hilbert space, where the mean embedding of different modality distributions can be explicitly compared. In addition, in the shared subspace, a quantizer is learned to produce the semantics preserving compact codes with the help of label alignment. Thanks to this novel network architecture in cooperation with supervised quantization training, SPDQ can preserve intramodal and intermodal similarities as much as possible and greatly reduce quantization error. Experiments on two popular benchmarks corroborate that our approach outperforms state-of-the-art methods.

Cross-Modal Hashing via Rank-Order Preserving

Due to the query effectiveness and efficiency, cross-modal similarity search based on hashing has acquired extensive attention in the multimedia community. Most existing methods do not explicitly employ the ranking information when learning hash functions, which is quite important for building practical retrieval systems. To solve this issue, this paper proposes a rank-order preserving hashing (RoPH) method with a novel regression-based rank-order preserving loss that has provable large margin property and is easy to optimize. Moreover, we jointly learn the binary codes and hash functions instead of using any relaxation trick. To solve the induced optimization problem, the alternating descent technique is adopted and each subproblem can be solved conveniently. Specifically, we show that the involved binary quadratic programming subproblem with respect to an introduced auxiliary binary variable satisfies submodularity, enabling us to use the off-the-shelf graph-cut algorithms to solve it exactly and efficiently. Extensive experiments on three benchmarks demonstrate that RoPH significantly improves the ranking quality over the state of the arts.

Cross-Modal Similarity Learning via Pairs, Preferences, and Active Supervision

We present a probabilistic framework for learning pairwise similarities between objects belonging to different modalities, such as drugs and proteins, or text and images. Our framework is based on learning a binary code based representation for objects in each modality, and has the following key properties: (i) it can leverage both pairwise as well as easy-to-obtain relative preference based cross-modal constraints, (ii) the probabilistic framework naturally allows querying for the most useful/informative constraints, facilitating an active learning setting (existing methods for cross-modal similarity learning do not have such a mechanism), and (iii) the binary code length is learned from the data. We demonstrate the effectiveness of the proposed approach on two problems that require computing pairwise similarities between cross-modal object pairs: cross-modal link prediction in bipartite graphs, and hashing based cross-modal similarity search.