Image Search Results Research Articles

Survey on Methodologies Used for Web Image Search

The image search has becomes an important feature of multimedia. It plays an important role in daily life. Some image search query results are satisfactory and some are unsatisfactory. The web image search mostly depends on the surrounding text of the image. It is difficult to understand the user intention only by query keywords and this leads to irrelevant image search results. In this paper the methods developed by different researchers in the area of web image search are reviewed. These methods vary from textual information search to user feedback. Also some methods are depend on the visual similarities between the images. To improve the result of web image search, strategies like keyword expansion, active re-ranking is also used. This paper focuses on the methods introduced by different authors for development in the area of web image search. General Terms Image Search, User Intention.

An attribute-assisted reranking model for web image search.

Image search reranking is an effective approach to refine the text-based image search result. Most existing reranking approaches are based on low-level visual features. In this paper, we propose to exploit semantic attributes for image search reranking. Based on the classifiers for all the predefined attributes, each image is represented by an attribute feature consisting of the responses from these classifiers. A hypergraph is then used to model the relationship between images by integrating low-level visual features and attribute features. Hypergraph ranking is then performed to order the images. Its basic principle is that visually similar images should have similar ranking scores. In this paper, we propose a visual-attribute joint hypergraph learning approach to simultaneously explore two information sources. A hypergraph is constructed to model the relationship of all images. We conduct experiments on more than 1,000 queries in MSRA-MMV2.0 data set. The experimental results demonstrate the effectiveness of our approach.

Graph-based clustering and ranking for diversified image search

In this paper, we consider the problem of clustering and re-ranking web image search results so as to improve diversity at high ranks. We propose a novel ranking framework, namely cluster-constrained conditional Markov random walk (CCCMRW), which has two key steps: first, cluster images into topics, and then perform Markov random walk in an image graph conditioned on constraints of image cluster information. In order to cluster the retrieval results of web images, a novel graph clustering model is proposed in this paper. We explore the surrounding text to mine the correlations between words and images and therefore the correlations are used to improve clustering results. Two kinds of correlations, namely word to image and word to word correlations, are mainly considered. As a standard text process technique, tf-idf method cannot measure the correlation of word to image directly. Therefore, we propose to combine tf-idf method with a novel feature of word, namely visibility, to infer the word-to-image correlation. By latent Dirichlet allocation model, we define a topic relevance function to compute the weights of word-to-word correlations. Taking word to image correlations as heterogeneous links and word-to-word correlations as homogeneous links, graph clustering algorithms, such as complex graph clustering and spectral co-clustering, are respectively used to cluster images into topics in this paper. In order to perform CCCMRW, a two-layer image graph is constructed with image cluster nodes as upper layer added to a base image graph. Conditioned on the image cluster information from upper layer, Markov random walk is constrained to incline to walk across different image clusters, so as to give high rank scores to images of different topics and therefore gain the diversity. Encouraging clustering and re-ranking outputs on Google image search results are reported in this paper.

Query difficulty estimation via relevance prediction for image retrieval

Query difficulty estimation (QDE) attempts to automatically predict the performance of the search results returned for a given query. QDE has long been of interest in text retrieval. However, few research works have been conducted in image retrieval. Existing QDE methods in image retrieval mainly explore the statistical characteristics (coherence, specificity, etc.) of the returned images to derive a value for indicating the query difficulty degree. To the best of our knowledge, little research has been done to directly estimate the real search performance, such as average precision. In this paper, we propose a novel query difficulty estimation approach which automatically estimates the average precision of the image search results. Specifically, we first adaptively select a set of query relevant and query irrelevant images for each query via modified pseudo relevance feedback. Then a simple but effective voting scheme and two estimation methods (hard estimation and soft estimation) are proposed to estimate the relevance probability of each image in the search results. Based on the images׳ relevance probabilities, the average precision for each query is derived. The experimental results on two benchmark image search datasets demonstrate the effectiveness of the proposed method.

Learning to rank using user clicks and visual features for image retrieval.

The inconsistency between textual features and visual contents can cause poor image search results. To solve this problem, click features, which are more reliable than textual information in justifying the relevance between a query and clicked images, are adopted in image ranking model. However, the existing ranking model cannot integrate visual features, which are efficient in refining the click-based search results. In this paper, we propose a novel ranking model based on the learning to rank framework. Visual features and click features are simultaneously utilized to obtain the ranking model. Specifically, the proposed approach is based on large margin structured output learning and the visual consistency is integrated with the click features through a hypergraph regularizer term. In accordance with the fast alternating linearization method, we design a novel algorithm to optimize the objective function. This algorithm alternately minimizes two different approximations of the original objective function by keeping one function unchanged and linearizing the other. We conduct experiments on a large-scale dataset collected from the Microsoft Bing image search engine, and the results demonstrate that the proposed learning to rank models based on visual features and user clicks outperforms state-of-the-art algorithms.

Web Image Re-Ranking UsingQuery-Specific Semantic Signatures

Image re-ranking, as an effective way to improve the results of web-based image search, has been adopted by current commercial search engines such as Bing and Google. Given a query keyword, a pool of images are first retrieved based on textual information. By asking the user to select a query image from the pool, the remaining images are re-ranked based on their visual similarities with the query image. A major challenge is that the similarities of visual features do not well correlate with images' semantic meanings which interpret users' search intention. Recently people proposed to match images in a semantic space which used attributes or reference classes closely related to the semantic meanings of images as basis. However, learning a universal visual semantic space to characterize highly diverse images from the web is difficult and inefficient. In this paper, we propose a novel image re-ranking framework, which automatically offline learns different semantic spaces for different query keywords. The visual features of images are projected into their related semantic spaces to get semantic signatures. At the online stage, images are re-ranked by comparing their semantic signatures obtained from the semantic space specified by the query keyword. The proposed query-specific semantic signatures significantly improve both the accuracy and efficiency of image re-ranking. The original visual features of thousands of dimensions can be projected to the semantic signatures as short as 25 dimensions. Experimental results show that 25-40 percent relative improvement has been achieved on re-ranking precisions compared with the state-of-the-art methods.

Automatic image annotation using affective vocabularies: Attribute-based learning approach

To improve image search results, understanding and exploiting the subjective aspects of an image is critical. However, how to effectively extract these subjective aspects (e.g. feeling, emotion, and so on) from an image is a challenging problem. In this paper, we propose a novel approach for predicting affective aspects, one of the most interesting subjective aspects, of concepts in images by learning the semantic attributes of the concept and mining the association between the attributes and affective aspects. The main idea of the proposed approach comes from the assumption that semantic attributes of a concept will influence the user’s affect towards the concept (e.g. an animal with the semantic attributes ‘ small’, ‘ furry’, ‘ white’ can be associated with the affective term ‘ cute’). Based on this assumption, we build a multi-layer affect learning framework that consists of (1) an attribute learning layer that predicts semantic attributes of a concept and (2) an affect learning layer that exploits the outputs from the attribute learning layer for predicting the affective aspects of the concept. Through the experimental results on the Animals with Attributes dataset, we show that the proposed approach outperforms traditional approaches by up to 25% in terms of precision and successfully predicts the affect of concepts in images according to different user preferences.

Employing physical distance in ranking image-search results

In this paper, we enhance web image searching by exploiting the physical distance between an image on the web page and the surrounding text. The rationale is that the closer a term word to an image is, the more strong the relation between that word and image is. The physical distance is measured between the exact positions of a term and an image inside a webpage as it appears when it is completely rendered by a web browser. We show that incorporating physical distance in the ranking process indeed reduces the number of irrelevant images in the returned search results.

Effective browsing of image search results via diversified visual summarization by clustering and refining clusters

The unprecedented growth in applications making use of digital images and multimedia references raised the requirement for image and topic search. Systematic processing of this information is a basic prerequisite for effective analysis of this information, as well as organization and management of it. Likewise, large collections of images are available on the Web, and many search engines provide the possibility of Web image searching based on the keywords. However, there are some problems in finding the desirable image according to the user needs. These problems include inexpressiveness of queries in description of user requirement, large number of irrelevant images to the intended search, lack of summarization, time-consuming review of overall images and lack of diversity. Clustering of image search results can be an efficient solution in solving these problems. In this work, several Folding-based algorithms have been proposed for clustering of image search results. In these algorithms, the efficiency, in comparison with Folding algorithm, is improved through a more effective selection of cluster’s representatives, fuzziness, weight and utilization of hierarchical algorithm preferences. In the represented system, in order to fit clusters with data more appropriately, an algorithm is proposed for refining the clusters. The proposed clustering causes a more convenient task in retrieval process for the user and also causes the efficient retrieval of images. According to the experiences, the proposed method improves the acceptable precision of image clustering.

Clustering results of image searches by annotations and visual features

Image on web has become one of the most important information for browsers; however, the large number of results retrieved from images search engine increases the difficulty in finding the intended images. Image search result clustering (ISRC) is a solution to this problem. Currently, the ISRC-based methods separately utilized textual and visual features to present clustering result. In this paper, we proposed a new ISRC method as called Incremental-Annotations-based image search with clustering (IAISC), which adopted annotation as textual features and category model as visual features. IAISC can provide clustering result based on the semantic meaning and visual trail; further, presented by the iteratively structure, a user can obtain the intended image easily. The experimental result shows our method has high precision that the average precision rate is 73.4%; particularly, the precision rate is 96.5% when the user drills down the intended images till the last round. Regarding efficiency, our system is one and a half times as efficient as the previous studies.

Similarity-based visualization of large image collections

Effective techniques for organizing and visualizing large image collections are in growing demand as visual search gets increasingly popular. Targeting an online astronomy archive with thousands of images, we present our solution for image search and clustering based on the evaluation of image similarity using both visual and textual information. Time-consuming image similarity computation is accelerated using graphics processing unit. To lay out images, we introduce iMap, a treemap-based representation for visualizing and navigating image search and clustering results. iMap not only makes effective use of available display area to arrange images but also maintains stable update when images are inserted or removed during the query. We also develop an embedded visualization that integrates image tags for in-place search refinement. To show the effectiveness of our approach, we demonstrate experimental results, compare our iMap layout with a force-directed layout, and conduct a comparative user study. As a potential tool for astronomy education and outreach, we deploy our iMap to a large tiled display of nearly 50 million pixels.

Learning realistic facial expressions from web images

A large amount of labeled training data is required to develop effective and robust facial expression analysis methods. However, obtaining such data is typically a tedious and time-consuming task. With a rapid advance of the Internet and Web technologies, it has been feasible to collect a large number of images with label information at a low cost of human efforts. In this paper, we propose a search based framework to collect realistic facial expression images from the Web so as to further advance research on robust facial expression recognition. Due to the limitation of current commercial web search engines, a large fraction of returned images is not related to a given query keyword. We present a Support Vector Machine (SVM) based active learning approach for selecting relevant images from noisy image search results. The resulting dataset is more diverse with more sample images per expression compared to other well established facial expression datasets such as CK and JAFFE. In addition, a novel facial expression feature based on the state-of-the-art Weber Local Descriptor (WLD) and histogram contextualization is proposed to handle such a challenging dataset. Comprehensive experimental results demonstrate that our web based dataset is capable of resembling more closely to the real world conditions compared to the CK and JAFFE datasets, and our proposed feature is more effective than the existing widely used features.

Paper] Novel Evaluation Criterion for Visualization of Image Search Results Based on Feature Integration Theory

This paper presents a new evaluation criterion for visualization of image search results based on the feature integration theory. This criterion is derived by combining two elements, visual saliency on visualization and grouping degree of similar images. Visual saliency, which is calculated from the feature integration theory, on visualization of image search results enables representation of users' attention, which is closely related to the effectiveness of finding images. Furthermore, since users perceive similar images that are close to each other as one group, grouping degree of similar images enables evaluation of the effectiveness when users find images similar to a desired image. Therefore, by combining visual saliency on visualization and grouping degree of similar images, we can derive the novel criterion and evaluate the effectiveness of visualization of image search results.

3-태그 기반의 웹 이미지 검색 기법

One of the most popular technologies in Web2.0 is tagging, and it widely applies to Web content as well as multimedia data such as image and video. Web users have expected that tags by themselves would be reused in information search and maximize the search efficiency, but wrong tag by irresponsible Web users really has brought forth a incorrect search results. In past papers, we have gathered various information resources and tags scattered in Web, mapped one tag onto other tags, and clustered these tags according to the corelation between them. A 3-tag based search algorithm which use the clustered tags of past papers, is proposed in this paper. For performance evaluation of the proposed algorithm, our algorithm is compared with image search result of Flickr, typical tag based site, and is evaluated in accuracy and recall factor.

Re-ranking Content Based Social Image Search Results by Multi Modal Relevance Feedback

Re-ranking Content Based Social Image Search Results by Multi Modal Relevance Feedback

Prototype-Based Image Search Reranking

The existing methods for image search reranking suffer from the unreliability of the assumptions under which the initial text-based image search result is employed in the reranking process. In this paper, we propose a prototype-based reranking method to address this problem in a supervised, but scalable fashion. The typical assumption that the top-N images in the text-based search result are equally relevant is relaxed by linking the relevance of the images to their initial rank positions. Then, we employ a number of images from the initial search result as the prototypes that serve to visually represent the query and that are subsequently used to construct meta rerankers. By applying different meta rerankers to an image from the initial result, reranking scores are generated, which are then aggregated using a linear model to produce the final relevance score and the new rank position for an image in the reranked search result. Human supervision is introduced to learn the model weights offline, prior to the online reranking process. While model learning requires manual labeling of the results for a few queries, the resulting model is query independent and therefore applicable to any other query. The experimental results on a representative web image search dataset comprising 353 queries demonstrate that the proposed method outperforms the existing supervised and unsupervised reranking approaches. Moreover, it improves the performance over the text-based image search engine by more than 25.48%.

Oracle in Image Search

This article studies a novel problem in image search. Given a text query and the image ranking list returned by an image search system, we propose an approach to automatically predict the search performance. We demonstrate that, in order to estimate the mathematical expectations of Average Precision (AP) and Normalized Discounted Cumulative Gain (NDCG), we only need to predict the relevance probability of each image. We accomplish the task with a query-adaptive graph-based learning based on the images’ ranking order and visual content. We validate our approach with a large-scale dataset that contains the image search results of 1,165 queries from 4 popular image search engines. Empirical studies demonstrate that our approach is able to generate predictions that are highly correlated with the real search performance. Based on the proposed image search performance prediction scheme, we introduce three applications: image metasearch, multilingual image search, and Boolean image search. Comprehensive experiments are conducted to validate our approach.

Query difficulty estimation for image retrieval

Query difficulty estimation predicts the performance of the search result of the given query. It is a powerful tool for multimedia retrieval and receives increasing attention. It can guide the pseudo relevance feedback to rerank the image search results and re-write the query by suggesting “easy” alternatives to obtain better search results. Many techniques to estimate the query difficulty have been proposed in the textual information retrieval, but directly employing them for image search will result in poor performance. That is because image query is more complex with spatial or structural information, and the well-known semantic gap induces extra burdens for accurate estimations. In this paper, we propose a query difficulty estimation approach by analyzing the top ranked images obtained by ad hoc retrieval models. Specifically, we seamlessly integrate the language model based clarity score, the spatial consistency of local descriptors and the appearance consistency of global features. Experimental results demonstrate that the query difficulty estimated by the proposed algorithm correlates well with the actual retrieval performance. Two applications of query difficulty estimation, namely guided pseudo relevance feedback (GPRF) and selective query refinement (SQR), are also proposed from both system and user perspectives. Experimental results show that both strategies further boost the retrieval performance.

Integrating bilingual search results for automatic junk image filtering

Keyword-based image search engines are now very popular for accessing large amounts of Web images on the Internet. Most existing keyword-based image search engines may return large amounts of junk images (which are irrelevant to the given query word), because the text terms that are loosely associated with the Web images are also used for image indexing. The objective of the proposed work is to effectively filter out the junk images from image search results. Therefore, bilingual image search results for the same keyword-based query are integrated to identify the clusters of the junk images and the clusters of the relevant images. Within relevant image clusters, the results are further refined by removing the duplications under a coarse-to-fine structure. Experiments for a large number of bilingual keyword-based queries (5,000 query words) are simultaneously performed on two keyword-based image search engines (Google Images in English and Baidu Images in Chinese), and our experimental results have shown that integrating bilingual image search results can filter out the junk images effectively.

Searching of Images Based on Content Using Blobs

Now a day’s image searching is still a challenging problem in content based image retrieval (CBIR) system. Most system operates on all images without pre-sorting the images. The image search result contains many unrelated image. The aim of this research is to propose a new method for content based image indexing and research based on blobs feature extraction and existing edges in the image and classification of image to different type and to search image which are similar the given research.