Cross-media Search Research Articles

A survey on cross-media search based on user intention understanding in social networks

With the increasing popularity of online social networks, more and more people are posting information, updating their statuses, and searching for topics there. Massive cross-media big data has been gathered by online social networks, with high dynamics, context-sparsity, and cross-media semantic gaps. In addition, it can be challenging to understand users' search intentions in the setting of social networks. The above problems have brought severe challenges and obstacles to cross-media searches, which also attracted more and more attention on social networks. As a relatively new research topic and interest, the concept, methodology, and overall research idea of cross-media search based on user search intention understanding are not evident in the literature. The research also lacks a unified paradigm and relatively complete research ideas on social networks. To solve these problems, we reviewed more than 100 references based on our preliminary exploration and research experience in this field from the whole process involved. We also detailed methodology, datasets, evaluation indicators, experiment evaluation, and research trends and analyzed the challenges. These works will help beginners quickly establish research ideas and processes in this field, and enable them to focus on algorithm design without paying too much attention to datasets, evaluation metrics, and research frameworks. We believe this review will attract more researchers to focus on social network cross-media search based on user search intention understanding and benefit their work.

Cross‐media search method based on complementary attention and generative adversarial network for social networks

The rapid development of the social network has brought great convenience to people's lives. A large amount of cross-media big data, such as text, image, and video data, has been accumulated. A cross-media search can facilitate a quick query of information so that users can obtain helpful content for social networks. However, cross-media data suffer from semantic gaps and sparsity in social networks, which bring challenges to cross-media searches. To alleviate the semantic gaps and sparsity, we propose a cross-media search method based on complementary attention and generative adversarial networks (CAGS). To obtain high-quality feature representations, we build a complementary attention mechanism containing the focused and unfocused features of images to realize the consistent association of cross-media data in social networks. By designing the cross-media adversarial learning process, we can obtain a common semantic representation of cross-media data and further alleviate the semantic gap and sparsity issues for social networks. Finally, we perform a similarity calculation to realize an accurate cross-media search. We construct four search tasks utilizing two standard cross-media data sets to verify the search performance of the proposed CAGS.

Robust and discrete matrix factorization hashing for cross-modal retrieval

Hashing based methods have gained great success for cross-modal similarity search, due to its fast query speed and low storage cost. However, there are some challenging problems that need to be further solved: 1) Many approaches are sensitive to noises and outliers, because ℓ2 norm is utilized in the objective function, the error may be amplified. 2) Most existing methods take relaxation or rounding scheme to generate binary codes, causing a large quantization loss. 3) Many supervised cross-media algorithms usually take a large n×n matrix to preserve the similarity relationship, leading to large calculation and making them unscalable. To mitigate these challenges, we develop a novel cross-media search algorithm, i.e., robust and discrete matrix factorization hashing, dubbed RDMH. The method takes a two-step strategy. In the first phase, the ℓ2,1 norm is utilized to improve the robustness, which makes our model not sensitive to noises and outliers. We can learn the hash codes directly by the proposed discrete optimization method instead of relaxation scheme, avoiding the large quantization loss. Moreover, RDMH correlates the hash codes and semantic labels directly instead of manipulating the large similarity matrix. In the second phase, we propose an autoencoder strategy to learn the hash functions, more valuable information can be preserved and making the hash functions more powerful. Comprehensive experiments on several databases demonstrate the superior performance and efficacy of the developed RDMH.

Label Consistent Flexible Matrix Factorization Hashing for Efficient Cross-modal Retrieval

Hashing methods have sparked a great revolution on large-scale cross-media search due to its effectiveness and efficiency. Most existing approaches learn unified hash representation in a common Hamming space to represent all multimodal data. However, the unified hash codes may not characterize the cross-modal data discriminatively, because the data may vary greatly due to its different dimensionalities, physical properties, and statistical information. In addition, most existing supervised cross-modal algorithms preserve the similarity relationship by constructing an n × n pairwise similarity matrix, which requires a large amount of calculation and loses the category information. To mitigate these issues, a novel cross-media hashing approach is proposed in this article, dubbed label flexible matrix factorization hashing (LFMH). Specifically, LFMH jointly learns the modality-specific latent subspace with similar semantic by the flexible matrix factorization. In addition, LFMH guides the hash learning by utilizing the semantic labels directly instead of the large n × n pairwise similarity matrix. LFMH transforms the heterogeneous data into modality-specific latent semantic representation. Therefore, we can obtain the hash codes by quantifying the representations, and the learned hash codes are consistent with the supervised labels of multimodal data. Then, we can obtain the similar binary codes of the corresponding modality, and the binary codes can characterize such samples flexibly. Accordingly, the derived hash codes have more discriminative power for single-modal and cross-modal retrieval tasks. Extensive experiments on eight different databases demonstrate that our model outperforms some competitive approaches.

A cross-media search method for social networks based on adversarial learning and semantic similarity

Social networks contain abundant multimedia information. The research of cross-media information search from social networks has become a hotspot.Single-modal semantic feature extractions and learnings based on deep learning show an appropriate effect on information search from social networks.But features of different modalities cannot be directly compared, therefore the semantic gap between different modalities is a key problem to be resolved for cross-media information search.Focusing on these problems, in this paper, we propose a cross-media search method based on adversarial learning and semantic similarity to accomplish matching and sorting between text and image.This method mainly adopts adversarial learning to train a feature projecting network and a modal specific network.The feature mapping network working as a generator converts raw data from different modalities into a common semantic space based on the corresponding semantic distributions,through which the distance of data from the different modalities is small under the same semantic space while the distance of data from the same modal under different semantics is large.The generator uses semantic distributions and similarities as the basis for training the feature mapping network.The discriminator aims at distinguishing the modality of the data from different media projected into the common semantic space.The designed method with two networks is alternately trained until convergence when the features obtained by the feature mapping network are consistent with the original data semantics,and the modal feature is eliminated. Finally, the search results are obtained by similarity ranking based on the raw data from the common semantic space.The efficiency results show that the evaluation value of text and image search is higher than other similar methods.The efficiency of the method is verified on social network data on security topics.

Cross-Media Semantic Correlation Learning Based on Deep Hash Network and Semantic Expansion for Social Network Cross-Media Search.

Cross-media search from large-scale social network big data has become increasingly valuable in our daily life because it can support querying different data modalities. Deep hash networks have shown high potential in achieving efficient and effective cross-media search performance. However, due to the fact that social network data often exhibit text sparsity, diversity, and noise characteristics, the search performance of existing methods often degrades when dealing with this data. In order to address this problem, this article proposes a novel end-to-end cross-media semantic correlation learning model based on a deep hash network and semantic expansion for social network cross-media search (DHNS). The approach combines deep network feature learning and hash-code quantization learning for multimodal data into a unified optimization architecture, which successfully preserves both intramedia similarity and intermedia correlation, by minimizing both cross-media correlation loss and binary hash quantization loss. In addition, our approach realizes semantic relationship expansion by constructing the image-word relation graph and mining the potential semantic relationship between images and words, and obtaining the semantic embedding based on both internal graph deep walk and an external knowledge base. Experimental results demonstrate that DHNS yields better cross-media search performance on standard benchmarks.

Joint Image-Text Hashing for Fast Large-Scale Cross-Media Retrieval Using Self-Supervised Deep Learning

Recent years have witnessed the promising future of hashing in the industrial applications for fast similarity retrieval. In this paper, we propose a novel supervised hashing method for large-scale cross-media search, termed self-supervised deep multimodal hashing (SSDMH), which learns unified hash codes as well as deep hash functions for different modalities in a self-supervised manner. With the proposed regularized binary latent model, unified binary codes can be solved directly without relaxation strategy while retaining the neighborhood structures by the graph regularization term. Moreover, we propose a new discrete optimization solution, termed as binary gradient descent , which aims at improving the optimization efficiency toward real-time operation. Extensive experiments on three benchmark data sets demonstrate the superiority of SSDMH over state-of-the-art cross-media hashing approaches.