Traditional Relevance Feedback Research Articles

Deep Learning of Image Representations with Convolutional Neural Networks Autoencoder for Image Retrieval with Relevance Feedback

mage retrieval with traditional relevance feedback encounters problems: (1) ability to represent handcrafted features which is limited, and (2) inefficient withhigh-dimensional data such as image data. In this paper,we propose a framework based on very deep convolutionalneural network autoencoder for image retrieval, called AIR(Autoencoders for Image Retrieval). Our proposed frameworkallows to learn feature vectors directly from the raw imageand in an unsupervised manner. In addition, our frameworkutilizes a hybrid approach of unsupervised and supervisedto improve retrieval performance. The experimental resultsshow that our method gives better results than some existingmethods on the CIFAR-100 image set, which consists of 60,000images.

Block-based pseudo-relevance feedback for image retrieval

ABSTRACT Pseudo-relevance feedback (PRF) is a relevance feedback (RF) technique for information retrieval that treats the top k retrieved images as relevance feedback. PRF is used to avoid the limitations of the traditional RF approach, which is a human-in-the-loop process. Although the pseudo-relevance feedback set contains noise, PRF can perform retrieval reasonably effectively. For implementing PRF, the Rocchio algorithm has been considered reasonably effective and is a well-established baseline method. However, it simply treats all of the top k feedback images as being equally similar to the query. Therefore, we present a block-based PRF approach for improving image retrieval performance. In this approach, images in the positive and negative feedback sets are further divided into predefined blocks, each of which contains one to several images, and blocks containing higher- or lower-ranked images will be assigned higher or lower weights, respectively. Experiments using the NUS-WIDE-LITE and Caltech 256 datasets and two different feature representations consistently show that the proposed approach using 30 blocks outperforms the baseline PRF in terms of P@10, P@20, and P@50. Furthermore, we show that a system that incorporates the user’s feedback allows the 30-block-based PRF approach to perform even better.

On the coupled use of signal and semantic concepts to bridge the semantic and user intention gaps for visual content retrieval

The effectiveness of image indexing and retrieval systems has been negatively impacted by both the semantic and user intention gaps. The first is related to the challenge of characterizing the visual semantic information through low-level extracted features (color, texture...) while the second highlights the difficulty for human users to convey their search intents using the traditional relevance feedback or query-by-example visual query mechanisms. We address both issues by highlighting vocabularies of visual concepts that are mapped to extracted low-level features through an automated learning paradigm. These are then instantiated within a semantic indexing and retrieval framework based on a Bayesian model considering the joint distribution of visual and semantic concepts. To address the user intention gap and enrich the expressiveness of the retrieval module, visual and semantic concepts can be coupled within text-based queries. We are therefore able to process both single-concept queries such as the state-of-the-art solutions but also topic-based queries, i.e. non-trivial queries involving multiple characterizations of the visual content. We evaluate our proposal in a precision/recall based evaluation framework on the IAPR-TC 12 benchmark dataset.

Annotation-retrieval reinforcement by visual cognition modeling on manifold

While content-based image annotation and retrieval have been active research topics over the past decade, their correlation is not well exploited until recently. We argue that offline annotation and online retrieval models should be regarded as a unified long-term learner to reinforce each other. Based on this viewpoint, this paper presents an annotation-retrieval reinforcement framework, in which a dual-structured learning model is presented: the keyword-image association is learned by semantic manifold modeling while the semantic correlation between keywords is learned by Image-Word Net modeling. In keyword-image association modeling level, to effectively model keyword-image manifold, we present a manifold co-training algorithm to address the sample insufficiency problem. In manifold-based image annotation, we view annotation process as semantic feature reduction in keyword space, based on which a Biased Fisher Discriminant Analysis (BFDA) algorithm is presented for Eigen feature (keyword) selection. In semantic correlation modeling level, a novel Image-Word Net is learned from annotation training set and users' retrieval log for (1) irrelevant annotated keyword pruning; (2) semantic-level retrieval enhancement. In retrieval, our framework can effectively reveal user target by improving traditional content-based relevance feedback to linguistic-level interaction using annotation information, based on which BFDA is adopted for keyword selection. Finally, user interaction logs are adopted for manifold model updating and annotation refinement. As presented in our experiments, proposed method outperforms state-of-the-art annotation and retrieval algorithms in COREL Image databases with over 10,000 general-purpose images.

A unified log-based relevance feedback scheme for image retrieval

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.

A novel long-term learning algorithm for relevance feedback in content-based image retrieval

Relevance feedback has proven to be a powerful tool to bridge the semantic gap between low-level features and high-level human concepts in content-based image retrieval (CBIR). However, traditional short-term relevance feedback technologies are confined to using the current feedback record only. Log-based long-term learning captures the semantic relationships among images in a database by analyzing the historical relevance information to boost the retrieval performance effectively. In this paper, we propose an expanded-judging model to analyze the historical log data's semantic information and to expand the feedback sample set from both positive and negative relevant information. The index table is used to facilitate the log analysis. The expanded-judging model is applied in image retrieval by combining with short-term relevance feedback algorithms. Experiments were carried out to evaluate the proposed algorithm based on the Corel image database. The promising experimental results validate the effectiveness of our proposed expanded-judging model.

A learning approach to optimizing exploration–exploitation tradeoff in relevance feedback

Relevance feedback is an effective technique for improving search accuracy in interactive information retrieval. In this paper, we study an interesting optimization problem in interactive feedback that aims at optimizing the tradeoff between presenting search results with the highest immediate utility to a user (but not necessarily most useful for collecting feedback information) and presenting search results with the best potential for collecting useful feedback information (but not necessarily the most useful documents from a user's perspective). Optimizing such an exploration---exploitation tradeoff is key to the optimization of the overall utility of relevance feedback to a user in the entire session of relevance feedback. We formally frame this tradeoff as a problem of optimizing the diversification of search results since relevance judgments on more diversified results have been shown to be more useful for relevance feedback. We propose a machine learning approach to adaptively optimizing the diversification of search results for each query so as to optimize the overall utility in an entire session. Experiment results on three representative retrieval test collections show that the proposed learning approach can effectively optimize the exploration---exploitation tradeoff and outperforms the traditional relevance feedback approach which only does exploitation without exploration.

Improving Relevance Feedback via Using Support Vector Machines

Traditional relevance feedback technique could help improve retrieval performance. It usually utilize the most frequent terms in the relevant documents to enrich the user’s initial query. We re-examine this method and find that many expansion terms identified in traditional approaches are indeed unrelated to the query and harmful to the retrieval. This paper introduces a Support Vector Machines Based method to improve the retrieval results. Firstly, the classifier is trained on the feedback documents. Then, we can utilize this classifier to classify the rest of the documents and move relevant documents to the front of irrelevant documents. This new approach avoids modifying the initial query, so it’s a new direction for the relevance feedback techniques. Our Experiments on TREC dataset demonstrate that retrieval effectiveness can be improved more than 24.37% when our proposed approach is used.

Client-Side Relevance Feedback Approach for Image Retrieval in Mobile Environment

During the last decade, high quality (i.e. over 1 megapixel) built-in cameras have become standard features of handheld devices. Users can take high-resolution pictures and share with friends via the internet. At the same time, the demand of multimedia information retrieval using those pictures on mobile devices has become an urgent problem to solve, and therefore attracts attention. A relevance feedback information retrieval process includes several rounds of query refinement, which incurs exchange of images between the mobile device and the server. With limited wireless bandwidth, this process can incur substantial delay, making the system unfriendly to use. This issue is addressed by considering a Client-side Relevance Feedback (CRF) technique. In the CRF system, Relevance Feedback (RF) is done on client side along. Mobile devices’ battery power is saved from exchanging images between server and client and system response is instantaneous, which significantly enhances system usability. Furthermore, because the server is not involved in RF processing, it is able to support more users simultaneously. The experiment indicates that the system outperforms the traditional server-client relevance feedback systems on the aspects of system response time, mobile battery power saving, and retrieval result.

A Log-Based 3D Model Retrieval Relevance Feedback Scheme Using Biased SVMs

Retrieval relevance feedback is an iterative search technique to bridge the semantic gap between the high level user intention and low level data representation. This technique interactively determines a user's desired output or query concept by asking the user whether certain proposed 3D models are relevant or not. In the past, most research efforts in 3D model retrieval field have focused on designing algorithms for traditional relevance feedback. Given a 3D model retrieval system, it can collect and store users’ relevance feedback information in a history log, 3D model retrieval system can take advantage of the log data of users’ feedback to enhance its retrieval performance. In this paper, we propose a unified 3D model retrieval relevance feedback framework that integrates the log data into the traditional relevance feedback schemes to learn effectively the correlation between low-level 3D model features and high-level concepts. In this 3D model retrieval relevance feedback scheme, we use a learning technique for relevance feedback, named biased support vector machine based relevance feedback. Experimental results show that this log-based scheme can achieves higher search accuracy than traditional query refinement schemes.

A Stochastic Approach to Image Retrieval Using Relevance Feedback and Particle Swarm Optimization

Understanding the subjective meaning of a visual query, by converting it into numerical parameters that can be extracted and compared by a computer, is the paramount challenge in the field of intelligent image retrieval, also referred to as the ?semantic gap? problem. In this paper, an innovative approach is proposed that combines a relevance feedback (RF) approach with an evolutionary stochastic algorithm, called particle swarm optimizer (PSO), as a way to grasp user's semantics through optimized iterative learning. The retrieval uses human interaction to achieve a twofold goal: 1) to guide the swarm particles in the exploration of the solution space towards the cluster of relevant images; 2) to dynamically modify the feature space by appropriately weighting the descriptive features according to the users' perception of relevance. Extensive simulations showed that the proposed technique outperforms traditional deterministic RF approaches of the same class, thanks to its stochastic nature, which allows a better exploration of complex, nonlinear, and highly-dimensional solution spaces.

Relevance feedback revisited: dealing with content and structure in XML documents

Relevance feedback (RF) is a technique that allows to enrich an initial query according to the user feedback. The goal is to express more precisely the user’s needs. Some open issues arise when considering semi-structured documents like XML documents. They are mainly related to the form of XML documents which mix content and structure information and to the new granularity of information. Indeed, the main objective of XML retrieval is to select relevant elements in XML documents instead of whole documents. Most of the RF approaches proposed in XML retrieval are simple adaptation of traditional RF to the new granularity of information. They usually enrich queries by adding terms extracted from relevant elements instead of terms extracted from whole documents. In this article, we describe a new approach of RF that takes advantage of two sources of evidence: the content and the structure. We propose to use the query term proximity to select terms to be added to the initial query and to use generic structures to express structural constraints. Both sources of evidence are used in different combined forms. Experiments were carried out within the INEX evaluation campaign and results show the effectiveness of our approaches.

A framework for a feedback process to analyze and personalize a document vector space in a feature extraction model

In this paper, we present a framework for a feedback process to implement a highly accurate document retrieval system. In the system, a document vector space is created dynamically to implement retrieval processing. The retrieval accuracy of the system depends on the vector space. When the vector space is created based on a specific purpose and interest of a user, highly accurate retrieval results can be obtained. In this paper, we present a method for analyzing and personalizing the vector space according to the purposes and interests of users. In order to optimize the document vector space, we defined and implemented functions for the operations of adding, deleting and weighting the terms that were used to create the vector space. By exploiting effectively and dynamically the classified-document information related to the queries, our methods allow users to retrieve relevant documents for their interests and purposes. Even if the search results of the initial retrieval space are not appropriate, by applying the proposed feedback operations, our proposed method effectively improves the search results. We also implemented an experimental search system for semantic document retrieval. Several experimental results including comparisons of our method with the traditional relevance feedback method is presented to clarify how retrieval accuracy was improved by the feedback process and how accurately documents that satisfied the purpose and interests of users were extracted.

A Unified Relevance Feedback Framework for Web Image Retrieval

Although relevance feedback (RF) has been extensively studied in the content-based image retrieval community, no commercial Web image search engines support RF because of scalability, efficiency, and effectiveness issues. In this paper, we propose a unified relevance feedback framework for Web image retrieval. Our framework shows advantage over traditional RF mechanisms in the following three aspects. First, during the RF process, both textual feature and visual feature are used in a sequential way. To seamlessly combine textual feature-based RF and visual feature-based RF, a query concept-dependent fusion strategy is automatically learned. Second, the textual feature-based RF mechanism employs an effective search result clustering (SRC) algorithm to obtain salient phrases, based on which we could construct an accurate and low-dimensional textual space for the resulting Web images. Thus, we could integrate RF into Web image retrieval in a practical way. Last, a new user interface (UI) is proposed to support implicit RF. On the one hand, unlike traditional RF UI which enforces users to make explicit judgment on the results, the new UI regards the users' click-through data as implicit relevance feedback in order to release burden from the users. On the other hand, unlike traditional RF UI which hardily substitutes subsequent results for previous ones, a recommendation scheme is used to help the users better understand the feedback process and to mitigate the possible waiting caused by RF. Experimental results on a database consisting of nearly three million Web images show that the proposed framework is wieldy, scalable, and effective.

Long-Term Cross-Session Relevance Feedback Using Virtual Features

Relevance feedback (RF) is an iterative process, which refines the retrievals by utilizing the user's feedback on previously retrieved results. Traditional RF techniques solely use the short-term learning experience and do not exploit the knowledge created during cross sessions with multiple users. In this paper, we propose a novel RF framework, which facilitates the combination of short-term and long-term learning processes by integrating the traditional methods with a new technique called the virtual feature. The feedback history with all the users is digested by the system and is represented in a very efficient form as a virtual feature of the images. As such, the dissimilarity measure can dynamically be adapted, depending on the estimate of the semantic relevance derived from the virtual features. In addition, with a dynamic database, the user's subject concepts may transit from one to another. By monitoring the changes in retrieval performance, the proposed system can automatically adapt the concepts according to the new subject concepts. The experiments are conducted on a real image database. The results manifest that the proposed framework outperforms the traditional within-session and log-based long-term RF techniques.

KERNEL GBDA FOR RELEVANCE FEEDBACK IN IMAGE RETRIEVAL

Relevance feedback, as a user-in-the-loop mechanism, has been widely employed to improve the performance of content-based image retrieval. Generally, in a relevance feedback algorithm, two key components are: (1) how to select a subset of effective features from a large-scale feature pool and, (2) correspondingly, how to construct a suitable dissimilarity measure. In previous work, the biased discriminant analysis (BDA) has been proposed to address these two problems during the feedback iterations. However, BDA encounters the so called small samples size problem because it has a lack of training samples. In this paper, we utilize the generalized singular value decomposition (GSVD) to significantly reduce the small samples size problem in BDA. The developed algorithm is named GSVD for BDA (GBDA). We then kernelize the GBDA to nonlinear kernel feature space. A large amount of experiments were carried out upon a large scale database, which contains 17800 images. From the experimental results, GBDA and its kernelization are demonstrated to outperform the traditional BDA-based relevance feedback approaches and their kernel extensions, respectively.

A two-level relevance feedback mechanism for image retrieval

Content-based image retrieval (CBIR) is a group of techniques that analyzes the visual features (such as color, shape, texture) of an example image or image subregion to find similar images in an image database. Relevance feedback is often used in a CBIR system to help users express their preference and improve query results. Traditional relevance feedback relies on positive and negative examples to reformulate the query. Furthermore, if the system employs several visual features for a query, the weight of each feature is adjusted manually by the user or system predetermined and fixed by the system. In this paper we propose a new relevance feedback model suitable for medical image retrieval. The proposed method enables the user to rank the results in relevance order. According to the ranking, the system can automatically determine the importance ranking of features, and use this ranking to automatically adjust the weight of each feature. The experimental results show that the new relevance feedback mechanism outperforms previous relevance feedback models.

Evaluating a workspace’s usefulness for image retrieval

Image searching is a creative process. We have proposed a novel image retrieval system that supports creative search sessions by allowing the user to organise their search results on a workspace. The workspace's usefulness is evaluated in a task-oriented and user-centred comparative experiment, involving design professionals and several types of realistic search tasks. In particular, we focus on its effect on task conceptualisation and query formulation. A traditional relevance feedback system serves as a baseline. The results of this study show that the workspace is more useful in terms of both of the above aspects and that the proposed approach leads to a more effective and enjoyable search experience. This paper also highlights the influence of tasks on the users' search and organisation strategy.

Content-based image retrieval using association rule mining with soft relevance feedback

With the rapid development of internet technology, the transmission and access of image items have become easier and the volume of image repository is exploding. An efficient and effective query reformulation is needed for finding the relevant images from the database. Relevance feedback (RF) is an interactive process which refines the retrieval results to a particular query by utilizing the user’s feedback on previously retrieved images. Most of the existing approaches deal with hard feedback (relevant and nonrelevant) and focus on individual experience only. We propose to facilitate the use of soft feedback (involving excellent, fair, don’t care, and bad) to better capture user’s intention. To add this feature, all of the traditional RF techniques should be modified accordingly. Further, the meta-knowledge exploited from multiple users’ experiences can improve the performance of future retrieval results. We propose a soft association rule mining algorithm to infer image relevance from the collective feedback. The number of association rules is kept minimum based on confidence quantization and redundancy detection. Also, binary search and best-first search techniques are implemented to expedite the process of relevance inference from the association rules. The proposed model provides a more flexible interface for relevance feedback and the experimental results manifest that the retrieval performance of the proposed model is better than that of traditional methods.

A unified log-based relevance feedback scheme for image retrieval

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.