Image Search Reranking Research Articles

Boosting Diversity in Visual Search with Pareto Non-Dominated Re-Ranking

The field of visual search has gained significant attention recently, particularly in the context of web search engines and e-commerce product search platforms. However, the abundance of web images presents a challenge for modern image retrieval systems, as they need to find both relevant and diverse images that maximize users’ satisfaction. In response to this challenge, we propose a non-dominated visual diversity re-ranking (NDVDR) method based on the concept of Pareto optimality. To begin with, we employ a fast binary hashing method as a coarse-grained retrieval procedure. This allows us to efficiently obtain a subset of candidate images for subsequent re-ranking. Fed with this initial retrieved image results, the NDVDR performs a fine-grained re-ranking procedure for boosting both relevance and visual diversity among the top-ranked images. Recognizing the inherent conflict nature between the objectives of relevance and diversity, the re-ranking procedure is simulated as the analytical stage of a multi-criteria decision-making process, seeking the optimal tradeoff between the two conflicting objectives within the initial retrieved images. In particular, a non-dominated sorting mechanism is devised that produces Pareto non-dominated hierarchies among images based on the Pareto dominance relation. Additionally, two novel measures are introduced for the effective characterization of the relevance and diversity scores among different images. We conduct experiments on three popular real-world image datasets and compare our re-ranking method with several state-of-the-art image search re-ranking methods. The experimental results validate that our re-ranking approach guarantees retrieval accuracy while simultaneously boosting diversity among the top-ranked images.

Social image retrieval based on topic diversity

Image search re-ranking is one of the most important approaches to enhance the text-based image search results. Extensive efforts have been dedicated to improve the accuracy and diversity of tag-based image retrieval. However, how to make the top-ranked results relevant and diverse is still a challenging problem. In this paper, we propose a novel method to diversify the retrieval results by latent topic analysis. We first employ NMF (Non-negative Matrix Factorization) Lee and Seung (Nature 401(6755):788–791, 1999) to estimate the initial relevance score to the query q. Then, the initial relevance score is fed into an adaptive multi-feature fusion model to learn the final relevance score. Next, the diversification process is conducted. We group all the images by semantic clustering and estimate the topic distribution of each cluster by topic analysis. The clusters are ranked based on the topic distribution vector and the final retrieval image list is obtained by a greedy selection mechanism based on the estimated relevances. Experimental results on the NUS-Wide dataset show the effectiveness of the proposed approach.

Hypergraph-based image search reranking with elastic net regularized regression

Image search reranking is emerging as an effective technique to refine the text-based image search results using visual information. In this paper, we introduce a novel hypergraph-based image search reranking method that accounts for both relevance and diversity of search results. Namely, the text-based image search results are taken as vertices in a probabilistic regression-based hypergraph model and reranking is formulated as a hypergraph ranking problem with absorbing nodes. More specifically, to discover related samples and characterize the relationships among them, we bring the Elastic Net regularized regression model into the hypergraph construction. Exceeding the conventional hypergraph construction schemes, our scheme is able to describe the high-order relationships and the local manifold structure among visual samples while ensuring the datum-adaptiveness. Afterward, we apply a hypergraph-based ranking with absorbing nodes to ensure a diversified reranking. That is, during the reranking process, previously-ranked samples are transformed into absorbing nodes at each iteration, thereby redundant ones are prevented from receiving high ranking scores. Extensive experiments on real-world data from Flickr suggest our proposed reranking method achieves promising results compared to existing reranking methods.

Hypergraph learning with collaborative representation for image search reranking

Image search reranking has received considerable attention in recent years. It aims at refining the text-based image search results by boosting the rank of relevant images. Hypergraph has been widely used for relevance estimation, where textual results are taken as vertices and the hypergraph ranking is performed to learn their relevance scores. Rather than using the K-nearest neighbor method, recent works have adopted the sparse representation to effectively construct an informative hypergraph. The sparse representation is insensitive to noise and can capture the real neighborhood structure. However, it suffers from a heavy computational cost. Motivated by this observation, in this paper, we leveraged the ridge regression for hypergraph construction. By imposing an $$\ell _2$$-regularizer on the size of their regression coefficients, the ridge regression enforces the training samples to collaborate to represent one query. The so-called collaborative representation exhibits more discriminative power and robustness while being computationally efficient. Thereafter, based on the obtained collaborative representation vectors, we measured the pairwise similarities among samples and generated hyperedges. Extensive experiments on the public MediaEval benchmarks demonstrated the effectiveness and superiority of our method over the state-of-the-art reranking methods.

Aggregation of Multiple Pseudo Relevance Feedbacks for Image Search Re-Ranking

Image retrieval effectiveness can be improved by pseudo relevance feedback (PRF), which automatically uses top-$k$ images of the initial retrieval result as the pseudo feedback. Since there are several different strategies for performing PRF leading to different search results, in this paper we focus on image search re-ranking by search result aggregation as a hybrid approach. In particular, different combinations of the original retrieval result with the result of PRF and the result of PRF by pseudo positive and negative feedbacks, using a strategy based on the Borda count, are compared. Our experiments, carried out on the NUS-WIDE-LIT and Caltech 256 datasets, demonstrate that search result aggregation can provide better retrieval performance than PRF. Specifically, the combination of the original result and the result of PRF by pseudo positive feedback performs the best.

Web Image Search Re-Ranking With Click-Based Similarity and Typicality.

In image search re-ranking, besides the well-known semantic gap, intent gap, which is the gap between the representation of users' query/demand and the real intent of the users, is becoming a major problem restricting the development of image retrieval. To reduce human effects, in this paper, we use image click-through data, which can be viewed as the implicit feedback from users, to help overcome the intention gap, and further improve the image search performance. Generally, the hypothesis-visually similar images should be close in a ranking list-and the strategy-images with higher relevance should be ranked higher than others-are widely accepted. To obtain satisfying search results, thus, image similarity and the level of relevance typicality are determinate factors correspondingly. However, when measuring image similarity and typicality, conventional re-ranking approaches only consider visual information and initial ranks of images, while overlooking the influence of click-through data. This paper presents a novel re-ranking approach, named spectral clustering re-ranking with click-based similarity and typicality. First, to learn an appropriate similarity measurement, we propose click-based multi-feature similarity learning algorithm, which conducts metric learning based on click-based triplets selection, and integrates multiple features into a unified similarity space via multiple kernel learning. Then, based on the learnt click-based image similarity measure, we conduct spectral clustering to group visually and semantically similar images into same clusters, and get the final re-rank list by calculating click-based clusters typicality and within-clusters click-based image typicality in descending order. Our experiments conducted on two real-world query-image data sets with diverse representative queries show that our proposed re-ranking approach can significantly improve initial search results, and outperform several existing re-ranking approaches.

HyperSSR: A hypergraph based semi-supervised ranking method for visual search reranking

Abstract Recently, considerable efforts have been made in visual search reranking towards refining initial text-based image search results. In this paper, we propose a hypergraph based semi-supervised ranking method called HyperSSR for image search reranking. According to the basic visual consistency principle that visually similar images should have similar ranking scores, we introduce the hypergraph to capture the intrinsic geometrical structure of the data distribution. To build a robust hypergraph, a novel hypergraph construction approach is developed to incorporate relevance and pseudo relevance degree information from labeled and unlabeled samples, respectively. Based on the premise that a ranking model should work better with the prior pairwise preferences, we jointly incorporate the hypergraph regularizer and the prior pairwise preferences information into a unified ranking learning framework. Experimental results on MSRA-MM 1.0 dataset suggest our proposed approach produces superior performances compared with several state-of-the-art methods.

Relevance Preserving Projection and Ranking for Web Image Search Reranking With Hierarchical Topic Awareness

Relevance Preserving Projection and Ranking for Web Image Search Reranking With Hierarchical Topic Awareness

Relevance and irrelevance graph based marginal Fisher analysis for image search reranking

Learning-to-rank techniques have shown promising results in the domain of image ranking recently, where dimensionality reduction is a critical step to overcome the “curse of dimensionality”. However, conventional dimensionality reduction approaches cannot guarantee the satisfying performance because the important ranking information is ignored. This paper presents a novel “Ranking Dimensionality Reduction” scheme specifically designed for learning-to-rank based image ranking, which aims at not only discovering the intrinsic structure of data but also keeping the ordinal information. Within this scheme, a new dimensionality reduction algorithm called Relevance Marginal Fisher Analysis (RMFA) is proposed. RMFA models the proposed pairwise constraints of relevance-link and irrelevance-link into the relevance graph and the irrelevance graph, and applies the graphs to build the objective function with the idea of Marginal Fisher Analysis (MFA). Further, a semi-supervised RMFA algorithm called Semi-RMFA is developed to offer a more general solution for the real-world application. Extensive experiments are carried on two popular, real-world image search reranking datasets. The promising results demonstrate the robustness and effectiveness of the proposed scheme and methods.

Relevance Preserving Projection and Ranking for Web Image Search Reranking.

An image search reranking (ISR) technique aims at refining text-based search results by mining images' visual content. Feature extraction and ranking function design are two key steps in ISR. Inspired by the idea of hypersphere in one-class classification, this paper proposes a feature extraction algorithm named hypersphere-based relevance preserving projection (HRPP) and a ranking function called hypersphere-based rank (H-Rank). Specifically, an HRPP is a spectral embedding algorithm to transform an original high-dimensional feature space into an intrinsically low-dimensional hypersphere space by preserving the manifold structure and a relevance relationship among the images. An H-Rank is a simple but effective ranking algorithm to sort the images by their distances to the hypersphere center. Moreover, to capture the user's intent with minimum human interaction, a reversed k-nearest neighbor (KNN) algorithm is proposed, which harvests enough pseudorelevant images by requiring that the user gives only one click on the initially searched images. The HRPP method with reversed KNN is named one-click-based HRPP (OC-HRPP). Finally, an OC-HRPP algorithm and the H-Rank algorithm form a new ISR method, H-reranking. Extensive experimental results on three large real-world data sets show that the proposed algorithms are effective. Moreover, the fact that only one relevant image is required to be labeled makes it has a strong practical significance.

Web Based Image search Using Semantic Signature Re-ranking Technique

Web-based image search re-ranking, as an successfulmethod to get better the results. In a query keyword, the first stair is store the images is first retrieve based on the text-based information. The user to select a query keywordimage, by using this query keyword other images are re-ranked based on their visual properties with images.Now a day to day, people projected to match images in a semantic space which is used attributes or reference classes closely related to the basis of semantic image. though, understanding a worldwide visual semantic space to demonstrate highly different images from the web is difficult and inefficient. The re-ranking images, which automatically offline part learns dissimilar semantic spaces for different query keywords. The features of images are projected into their related semantic spaces to get particular images. At the online stage, images are re-ranked by compare their semantic signatures obtained the semantic precised by the query keyword image. The query-specific semantic signatures extensively improve both the proper and efficiency of image re-ranking. Keyword:Query, keyword, image, re-ranking, semantic, signature __________________________________________________*****_________________________________________________

An attribute-assisted reranking model for web image search.

Image search reranking is an effective approach to refine the text-based image search result. Most existing reranking approaches are based on low-level visual features. In this paper, we propose to exploit semantic attributes for image search reranking. Based on the classifiers for all the predefined attributes, each image is represented by an attribute feature consisting of the responses from these classifiers. A hypergraph is then used to model the relationship between images by integrating low-level visual features and attribute features. Hypergraph ranking is then performed to order the images. Its basic principle is that visually similar images should have similar ranking scores. In this paper, we propose a visual-attribute joint hypergraph learning approach to simultaneously explore two information sources. A hypergraph is constructed to model the relationship of all images. We conduct experiments on more than 1,000 queries in MSRA-MMV2.0 data set. The experimental results demonstrate the effectiveness of our approach.

Image Relevance Prediction Using Query-Context Bag-of-Object Retrieval Model

Image search reranking and image research result summarization are two effective approaches which enhance text-based image search results using visual information. Since the existing approaches optimize search relevance in terms of average performance, they usually cannot achieve satisfactory results for some particular classes of queries, like “object queries,” which is defined as the queries with the intent of searching for some kinds of objects. One possible reason is that the generic approaches such as , , are mostly built based on the global statistics of images as features while ignoring the fact that the relevance between the image and the query sometimes depends on an image patch instead of the whole image. In this paper, we therefore design a novel bag-of-object retrieval model to predict image relevance, which is particularly effective for object queries. First, we construct an object vocabulary containing query-relative objects by mining frequent object patches from the result image collection of the expanded query set. After representing each image as a bag of objects, our retrieval model can be derived from a risk-minimization framework for language modeling. To demonstrate the effectiveness of the proposed model, this paper also present two related applications: for image search reranking, we adopt a supervised framework to combine multiple ranking features from different assumptions; for image search result summarization, we propose a two-step ranking process which optimizes not only representativeness but also image attractiveness. The experimental results show that the proposed methods can significantly outperform the existing approaches.

Joint-Rerank: a novel method for image search reranking

Image search reranking which aims to improve the text-based image search results with the help of other cues has grown into a hot research topic. Most existing reranking methods only focus on image visual cues. However, the visual cues cannot always guarantee to provide enough information for the reranking process. Although, some approaches try to fuse multiple image cues for reranking, they do not or weakly exploit the relationships among multiple image cues. In this paper,we present a novel image reranking framework--Joint-Rerank which considers multiple modalities of images (or multiple cues) jointly as interdependent attributes of an image entity. Joint-Rerank models the images as a multigraph where each image is a node with multimodal attributes (textual and visual cues) and the parallel edges between nodes measure both image intra-modal and inter-modal similarities. Besides, each node has a "self-consistency" that measures how much the multiple modalities of an image may be consistent. To solve the reranking problem, we first degenerate the multigraph into a new complete graph, and then employ a random walk on the degenerated graph to propagate the relevance scores of each node. Finally, the relevance scores of multiple modalities are fused to rank the images. Moreover, in Joint-Rerank, "cross-modal" walk is possible, i.e., a surfer can jump from one image to another following both intra-modal and inter-modal links. In this framework, we propose two methods: Sym-Joint-Rerank and Asym-Joint-Rerank which use different approaches to measure the inter-modal similarities between two nodes. Experimental results on a large web queries dataset which contains 353 image search queries show that both of them are superior or highly competitive to several state-of-the-art reranking algorithms.

Ranking graph embedding for learning to rerank.

Dimensionality reduction is a key step to improving the generalization ability of reranking in image search. However, existing dimensionality reduction methods are typically designed for classification, clustering, and visualization, rather than for the task of learning to rank. Without using of ranking information such as relevance degree labels, direct utilization of conventional dimensionality reduction methods in ranking tasks generally cannot achieve the best performance. In this paper, we show that introducing ranking information into dimensionality reduction significantly increases the performance of image search reranking. The proposed method transforms graph embedding, a general framework of dimensionality reduction, into ranking graph embedding (RANGE) by modeling the global structure and the local relationships in and between different relevance degree sets, respectively. The proposed method also defines three types of edge weight assignment between two nodes: binary, reconstruction, and global. In addition, a novel principal components analysis based similarity calculation method is presented in the stage of global graph construction. Extensive experimental results on the MSRA-MM database demonstrate the effectiveness and superiority of the proposed RANGE method and the image search reranking framework.

Ranking Fisher discriminant analysis

Existing feature dimensionality reduction algorithms are inherently designed for the case of classification, clustering and retrieval, but not for ranking applications such as visual search reranking. This is a serious limitation which restricts the applicability of existing dimensionality reduction methods as well as the generalization ability of ranking applications. Therefore, it is important to design a kind of special methods to be effectively employed for ranking applications. Fisher discriminant analysis (FDA) is one of the most popular dimensionality reduction methods. Thus, we propose a novel dimensionality reduction algorithm based on FDA to solve this kind of problem in this paper. Specifically, relevance degree information—the data label in ranking applications, is introduced to a semi-supervised form of FDA, in which both local information and unlabeled data are employed. We name the proposed method as ranking Fisher discriminant analysis (RFDA). To verify the effectiveness of RFDA, extensive experiments are carried out on image search reranking applications, which show significant performance based on the popular MSRA-MM dataset.

Multimodal Graph-Based Reranking for Web Image Search

This paper introduces a web image search reranking approach that explores multiple modalities in a graph-based learning scheme. Different from the conventional methods that usually adopt a single modality or integrate multiple modalities into a long feature vector, our approach can effectively integrate the learning of relevance scores, weights of modalities, and the distance metric and its scaling for each modality into a unified scheme. In this way, the effects of different modalities can be adaptively modulated and better reranking performance can be achieved. We conduct experiments on a large dataset that contains more than 1000 queries and 1 million images to evaluate our approach. Experimental results demonstrate that the proposed reranking approach is more robust than using each individual modality, and it also performs better than many existing methods.

Prototype-Based Image Search Reranking

The existing methods for image search reranking suffer from the unreliability of the assumptions under which the initial text-based image search result is employed in the reranking process. In this paper, we propose a prototype-based reranking method to address this problem in a supervised, but scalable fashion. The typical assumption that the top-N images in the text-based search result are equally relevant is relaxed by linking the relevance of the images to their initial rank positions. Then, we employ a number of images from the initial search result as the prototypes that serve to visually represent the query and that are subsequently used to construct meta rerankers. By applying different meta rerankers to an image from the initial result, reranking scores are generated, which are then aggregated using a linear model to produce the final relevance score and the new rank position for an image in the reranked search result. Human supervision is introduced to learn the model weights offline, prior to the online reranking process. While model learning requires manual labeling of the results for a few queries, the resulting model is query independent and therefore applicable to any other query. The experimental results on a representative web image search dataset comprising 353 queries demonstrate that the proposed method outperforms the existing supervised and unsupervised reranking approaches. Moreover, it improves the performance over the text-based image search engine by more than 25.48%.

Generating Descriptive Visual Words and Visual Phrases for Large-Scale Image Applications

Bag-of-visual Words (BoWs) representation has been applied for various problems in the fields of multimedia and computer vision. The basic idea is to represent images as visual documents composed of repeatable and distinctive visual elements, which are comparable to the text words. Notwithstanding its great success and wide adoption, visual vocabulary created from single-image local descriptors is often shown to be not as effective as desired. In this paper, descriptive visual words (DVWs) and descriptive visual phrases (DVPs) are proposed as the visual correspondences to text words and phrases, where visual phrases refer to the frequently co-occurring visual word pairs. Since images are the carriers of visual objects and scenes, a descriptive visual element set can be composed by the visual words and their combinations which are effective in representing certain visual objects or scenes. Based on this idea, a general framework is proposed for generating DVWs and DVPs for image applications. In a large-scale image database containing 1506 object and scene categories, the visual words and visual word pairs descriptive to certain objects or scenes are identified and collected as the DVWs and DVPs. Experiments show that the DVWs and DVPs are informative and descriptive and, thus, are more comparable with the text words than the classic visual words. We apply the identified DVWs and DVPs in several applications including large-scale near-duplicated image retrieval, image search re-ranking, and object recognition. The combination of DVW and DVP performs better than the state of the art in large-scale near-duplicated image retrieval in terms of accuracy, efficiency and memory consumption. The proposed image search re-ranking algorithm: DWPRank outperforms the state-of-the-art algorithm by 12.4% in mean average precision and about 11 times faster in efficiency.

Towards a Relevant and Diverse Search of Social Images

Recent years have witnessed the great success of social media websites. Tag-based image search is an important approach to accessing the image content on these websites. However, the existing ranking methods for tag-based image search frequently return results that are irrelevant or not diverse. This paper proposes a diverse relevance ranking scheme that is able to take relevance and diversity into account by exploring the content of images and their associated tags. First, it estimates the relevance scores of images with respect to the query term based on both the visual information of images and the semantic information of associated tags. Then, we estimate the semantic similarities of social images based on their tags. Based on the relevance scores and the similarities, the ranking list is generated by a greedy ordering algorithm which optimizes average diverse precision, a novel measure that is extended from the conventional average precision. Comprehensive experiments and user studies demonstrate the effectiveness of the approach. We also apply the scheme for web image search reranking, and it is shown that the diversity of search results can be enhanced while maintaining a comparable level of relevance.