User Relevance Feedback Research Articles

An experiment in building profiles in information filtering: the role of context of user relevance feedback

An experiment was conducted to see how relevance feedback could be used to build and adjust profiles to improve the performance of filtering systems. Data was collected during the system interaction of 18 graduate students with SIFTER (Smart Information Filtering Technology for Electronic Resources), a filtering system that ranks incoming information based on users' profiles. The data set came from a collection of 6000 records concerning consumer health. In the first phase of the study, three different modes of profile acquisition were compared. The explicit mode allowed users to directly specify the profile; the implicit mode utilized relevance feedback to create and refine the profile; and the combined mode allowed users to initialize the profile and to continuously refine it using relevance feedback. Filtering performance, measured in terms of Normalized Precision, showed that the three approaches were significantly different ( α=0.05 and p=0.012). The explicit mode of profile acquisition consistently produced superior results. Exclusive reliance on relevance feedback in the implicit mode resulted in inferior performance. The low performance obtained by the implicit acquisition mode motivated the second phase of the study, which aimed to clarify the role of context in relevance feedback judgments. An inductive content analysis of thinking aloud protocols showed dimensions that were highly situational, establishing the importance context plays in feedback relevance assessments. Results suggest the need for better representation of documents, profiles, and relevance feedback mechanisms that incorporate dimensions identified in this research.

An information retrieval model based on vector space method by supervised learning

This paper proposes a method to improve retrieval performance of the vector space model (VSM) in part by utilizing user-supplied information of those documents that are relevant to the query in question. In addition to the user's relevance feedback information, information such as original document similarities is incorporated into the retrieval model, which is built by using a sequence of linear transformations. High-dimensional and sparse vectors are then reduced by singular value decomposition (SVD) and transformed into a low-dimensional vector space, namely the space representing the latent semantic meanings of words. The method has been tested with two test collections, the Medline collection and the Cranfield collection. In order to train the model, multiple partitions are created for each collection. Improvement of average precision of the averages over all partitions, compared with the latent semantic indexing (LSI) model, are 20.57% (Medline) and 22.23% (Cranfield) for the two training data sets, and 0.47% (Medline) and 4.78% (Cranfield) for the test data, respectively. The proposed method provides an approach that makes it possible to preserve user-supplied relevance information for the long term in the system in order to use it later.

Discovering Text Databases with Neural Nets

Since documents on the Web are naturally partitioned into many text databases, the efficient document retrieval process requires identifying the text databases that are most likely to provide relevant documents to the query and then searching for the identified text databases. In this paper, we propose a neural net based approach to such an efficient document retrieval. First, we present a neural net agent that learns about underlying text databases from the user's relevance feedback. For a given query, the neural net agent, which is sufficiently trained on the basis of the BPN learning mechanism, discovers the text databases associated with the relevant documents and retrieves those documents effectively. In order to scale our approach with the large number of text databases, we also propose the hierarchical organization of neural net agents which reduces the total training cost at the acceptable level. Finally, we evaluate the performance of our approach by comparing it to those of the conventional well-known approaches.

Personalized web-document filtering using reinforcement learning

Document filtering is increasingly deployed in Web environments to reduce information overload of users. We formulate online information filtering as a reinforcement learning problem, i.e., TD(0). The goal is to learn user profiles that best represent information needs and thus maximize the expected value of user relevance feedback. A method is then presented that acquires reinforcement signals automatically by estimating user's implicit feedback from direct observations of browsing behaviors. This "learning by observation" approach is contrasted with conventional relevance feedback methods which require explicit user feedbacks. Field tests have been performed that involved 10 users reading a total of 18,750 HTML documents during 45 days. Compared to the existing document filtering techniques, the proposed learning method showed superior performance in information quality and adaptation speed to user preferences in online filtering.

Interactive image retrieval using fuzzy sets

We present an image retrieval system which permits the user to submit a coarse initial query and continuously refine it. The user's relevance feedbacks is modeled by fuzzy sets, and is used to discover and use the more discriminatory features for the given query. The proposed system uses a dissimilarity measure based on the fuzzy integral.

Text Database Discovery on the Web: Neural Net Based Approach

As large numbers of text databases have become available on the Web, many efforts have been made to solve the text database discovery problem: finding which text databases (out of many candidates) are most likely to provide relevant documents to a given query. In this paper, we propose a neural net based approach to this problem. First, we present a neural net agent that learns about underlying text databases from the user's relevance feedback. For a given query, the neural net agent, which is sufficiently trained on the basis of the backpropagation learning mechanism, discovers the text databases associated with the relevant documents and retrieves those documents effectively. In order to scale our approach with the large number of text databases, we also propose the hierarchical organization of neural net agents which reduces the total training cost at the acceptable level. Finally, we evaluate the performance of our approach by comparing it to those of the conventional well-known statistical approaches.

Maxi-Adjustment and Possibilistic Deduction for Adaptive Information Agents

The expressive power of logic is believed to be able to model most of the fundamental aspects of information retrieval (IR). However, it is also understood that classical logic is ineffective for handling partiality and uncertainty in IR. Applying non-classical logics such as the AGM belief revision logic and the possibilistic logic to adaptive information retrieval is appealing since they provide a powerful and rigorous framework to model partiality and uncertainty inherent in any IR processes. The maxi-adjustment method, which is an effective computational apparatus of the AGM paradigm, is applied to develop the learning components of the adaptive information agents. Essentially, maxi-adjustment allows the partial representation K of a user's information needs N to be refined gradually based on the user's relevance feedback t. Generally speaking, learning in adaptive information agents is characterised by the AGM belief revision Kt*. On the other hand, possibilistic logic supports a gradated assessment of the uncertainty arising from matching K with the imperfect characterisation d of an information object D. Information matching in adaptive information agents is underpinned by K ƕ d, where ƕ is the possibilistic inference relation. This paper illustrates how maxi-adjustment and possibilistic deduction can be applied to develop the adaptive information agents. Their impact on the agents learning autonomy and explanatory power is also discussed.

Improving Perceptron Convergence Algorithm for retrieval systems

Based on the premises of relevance feedback, we investigate a well-known learning procedure called the Perceptron Convergence Algorithm (PCA) ( Wong et al., 1988) for information retrieval (IR) systems. Although PCA is one of the simplest methods with the least overhead, it has its disadvantages. One of the main disadvantages is that the rate of convergence of the algorithm can be slow. In this paper, we propose and evaluate an algorithm that can be used in IR systems, called the Fast Perceptron, which improves the speed of convergence of the PCA. By considering a prototype image retrieval system called Web-IDBS (Image Database System) ( Guivada et al., 1994) that supports content based adaptive retrieval using semantic attributes ( Guivada and Raghavan, 1995), we compared the performance of the fast perceptron algorithm to that of the PCA. The objective of this experiment was to compare the impact of user relevance feedback on improving retrieval effectiveness with these query reformulating algorithms. The metrics used for comparison were the number of iterations required in each algorithm to get the optimal query, the final ranking of the target image, and the number of images for which the user provides feedback. The results demonstrated that significant improvement is achieved in the metrics considered.

An inferential approach to Information Retrieval and its implementation using a manual thesaurus

Most inferential approaches to Information Retrieval (IR) have been investigated within the probabilistic framework. Although these approaches allow one to cope with the underlying uncertainty of inference in IR, the strict formalism of probability theory often confines our use of knowledge to statistical knowledge alone (e.g. connections between terms based on their co-occurrences). Human-defined knowledge (e.g. manual thesauri) can only be incorporated with difficulty. In this paper, based on a general idea proposed by van Rijsbergen, we first develop an inferential approach within a fuzzy modal logic framework. Differing from previous approaches, the logical component is emphasized and considered as the pillar in our approach. In addition, the flexibility of a fuzzy modal logic framework offers the possibility of incorporating human-defined knowledge in the inference process. After defining the model, we describe a method to incorporate a human-defined thesaurus into inference by taking user relevance feedback into consideration. Experiments on the CACM corpus using a general thesaurus of English, Wordnet, indicate a significant improvement in the system's performance.

A network approach to probabilistic information retrieval

In this article we show how probabilistic information retrieval based on document components may be implemented as a feedforward (feedbackward) artificial neural network. The network supports adaptation of connection weights as well as the growing of new edges between queries and terms based on user relevance feedback data for training, and it reflects query modification and expansion in information retrieval. A learning rule is applied that can also be viewed as supporting sequential learning using a harmonic sequence learning rate. Experimental results with four standard small collections and a large Wall Street Journal collection (173,219 documents) show that performance of feedback improves substantially over no feedback, and further gains are obtained when queries are expanded with terms from the feedback documents. The effect is much more pronounced in small collections than in the large collection. Query expansion may be considered as a tool for both precision and recall enhancement. In particular, small query expansion levels of about 30 terms can achieve most of the gains at the low-recall high-precision region, while larger expansion levels continue to provide gains at the high-recall low-precision region of a precision recall curve.