Effectiveness Of Information Retrieval Systems Research Articles

Regions and levels: Measuring and mapping users' relevance judgments

The dichotomous bipolar approach to relevance has produced an abundance of information retrieval (IR) research. However, relevance studies that include consideration of users' partial relevance judgments are moving to a greater relevance clarity and congruity to impact the design of more effective IR systems. The study reported in this paper investigates the various regions of across a distribution of users' relevance judgments, including how these regions may be categorized, measured, and evaluated. An instrument was designed using four scales for collecting, measuring, and describing end-user relevance judgments. The instrument was administered to 21 end-users who conducted searches on their own information problems and made relevance judgments on a total of 1059 retrieved items. Findings include: (1) overlapping regions of relevance were found to impact the usefulness of precision ratios as a measure of IR system effectiveness, (2) both positive and negative levels of relevance are important to users as they make relevance judgments, (3) topicality was used more to reject rather than accept items as highly relevant, (4) utility was more used to judge items highly relevant, and (5) the nature of relevance judgment distribution suggested a new IR evaluation measure—median effect. Findings suggest that the middle region of a distribution of relevance judgments, also called “partial relevance,” represents a key avenue for ongoing study. The findings provide implications for relevance theory, and the evaluation of IR systems.

Query modification based on relevance back-propagation in an ad hoc environment

It is well-known that relevance feedback is a method significant in improving the effectiveness of information retrieval systems. Improving effectiveness is important since these information retrieval systems must gain access to large document collections distributed over different distant sites. As a consequence, efforts to retrieve relevant documents have become significantly greater. Relevance feedback can be viewed as an aid to the information retrieval task. In this paper, a relevance feedback strategy is presented. The strategy is based on back-propagation of the relevance of retrieved documents using an algorithm developed in a neural approach. This paper describes a neural information retrieval model and emphasizes the results obtained with the associated relevance back-propagation algorithm in three different environments: manual ad hoc, automatic ad hoc and mixed ad hoc strategy (automatic plus manual ad hoc).

Disambiguating highly ambiguous words

A word sense disambiguator that is able to distinguish among the many senses of common words that are found in general-purpose, broad-coverage lexicons would be useful. For example, experiments have shown that, given accurate sense disambiguation, the lexical relations encoded in lexicons such as WordNet can be exploited to improve the effectiveness of information retrieval systems. This paper describes a classifier whose accuracy may be sufficient for such a purpose. The classifier combines the output of a neural network that learns topical context with the output of a network that learns local context to distinguish among the senses of highly ambiguous words.The accuracy of the classifier is tested on three words, the noun line, the verb serve, and the adjective hard; the classifier has an average accuracy of 87%, 90%, and 81%, respectively, when forced to choose a sense for all test cases. When the classifier is not forced to choose a sense and is trained on a subset of the available senses, it rejects test cases containing unknown senses as well as test cases it would misclassify if forced to select a sense. Finally, when there are few labeled training examples available, we describe an extension of our training method that uses information extracted from unlabeled examples to improve classification accuracy.

Relevance judgements for assessing recall

Recall and Precision have become the principle measures of the effectiveness of information retrieval systems. Inherent in these measures of performance is the idea of a relevant document. Although recall and precision are easily and unambiguously defined, selecting the documents relevant to a query has long been recognized as problematic. To compare performance of different systems, standard collections of documents, queries, and relevance judgments have been used. Unfortunately the standard collections, such as SMART and TREC, have locked in a particular approach to relevance that is suitable for assessing precision but not recall. The problem is demonstrated by comparing two information retrieval methods over several queries, and showing how a new method of forming relevance judgments that a suitable for assessing recall gives different results. Recall is an interesting and practical issue, but current test procedures are inadequate for measuring it.

Evaluating the effectiveness of information retrieval systems using simulated queries

Simulation is a widely used investigative tool for exploring proposed systems without incurring the costs of actually building them. Studying the effectiveness of document retrieval systems by means of simulation, however, has remained elusive because of the guesswork involved in deciding whether a document with a given (simulated) description will be relevant to the information need of an inquirer expressed by a given (simulated) query. In this article, a simulation method is described for estimating recall and fallout in a document retrieval system. Examples are presented to illustrate the method, and it is justified probabilistically. By using this method, we may compare the effectiveness of employing various indexing procedures, compare the effectiveness of employing various matching functions, or obtain absolute effectiveness measures for a proposed system. © 1990 John Wiley & Sons, Inc.

A critical investigation of recall and precision as measures of retrieval system performance

Recall and precision are often used to evaluate the effectiveness of information retrieval systems. They are easy to define if there is a single query and if the retrieval result generated for the query is a linear ordering. However, when the retrieval results are weakly ordered, in the sense that several documents have an identical retrieval status value with respect to a query, some probabilistic notion of precision has to be introduced. Relevance probability, expected precision, and so forth, are some alternatives mentioned in the literature for this purpose. Furthermore, when many queries are to be evaluated and the retrieval results averaged over these queries, some method of interpolation of precision values at certain preselected recall levels is needed. The currently popular approaches for handling both a weak ordering and interpolation are found to be inconsistent, and the results obtained are not easy to interpret. Moreover, in cases where some alternatives are available, no comparative analysis that would facilitate the selection of a particular strategy has been provided. In this paper, we systematically investigate the various problems and issues associated with the use of recall and precision as measures of retrieval system performance. Our motivation is to provide a comparative analysis of methods available for defining precision in a probabilistic sense and to promote a better understanding of the various issues involved in retrieval performance evaluation.

Architecture of an expert system for composite document analysis, representation, and retrieval

The CODER (COmposite Document Expert/extended/effective Retrieval) project is a multi-yeare effort to investigate how best to apply artificial intelligence methods to increase the effectiveness of information retrieval systems handling collections of composite documents. To ensure system adaptability and to allow controlled experimentation, CODER has been designed as a distributed expert system. The use of individually tailored specialist experts, coupled with standardized blackboard modules for communication and control and external knowledge bases for maintenance of factual world knowledge, allows for quick prototyping, incremental development, and flexibility under change. The system as a whole is being implemented under UNIX as a set of MU-Prolog and C modules communicating through pipes and TCP/IP sockets.

Development of the coder system: A testbed for artificial intelligence methods in information retrieval

The CODER ( Co mposite D ocument E xpert/Extended/Effective R etrieval) system is a testbed for investigating the application of artificial intelligence methods to increase the effectiveness of information retrieval systems. Particular attention is being given to analysis and representation of heterogeneous documents, such as electronic mail digests or messages, which vary widely in style, length, topic, and structure. Since handling passages of various types in these collections is difficult even for experimental systems like SMART, it is necessary to turn to other techniques being explored by information retrieval and artificial intelligence researchers. The CODER system architecture involves communities of experts around active blackboards, accessing knowledge bases that describe users, documents, and lexical items of various types. The initial lexical knowledge base construction work is now complete, and experts for search and time/date handling can perform a variety of processing tasks. User information and queries are being gathered, and a simple distributed skeletal system is operational. It appears that a number of artificial intelligence techniques are needed to best handle such common but complex document analysis and retrieval tasks.

The effectiveness of information retrieval systems: the need for improved explanatory frameworks

There is a critical lacuna in information retrieval research between the theoretical framework employed in laboratory tests to explain the performance of information retrieval systems, and the sorts of factors which bear on their operational effectiveness. The influence of assumptions as to which factors it is important to take into account in evaluations of the effectiveness of information retrieval systems, derived from the laboratory testing approach in the form of an implicit model of the retrieval situation, has had an inhibiting effect on information retrieval research. Evaluations of the operational effectiveness of information retrieval systems are particularly hampered by the absence of a suitable conceptual framework of explanation. The argument of this paper is that there is a need for more microevaluation of the activities and environments of the users of information systems in order to develop an understanding of the relation of information services to those activities and environments. This sort of understanding should provide more in the way of guidance for the design of more effective systems than is available from current information retrieval theory.

THE RELATIONSHIP BETWEEN DOCUMENTS READ AND RELEVANT REFERENCES RETRIEVED AS EFFECTIVENESS MEASURES FOR INFORMATION RETRIEVAL SYSTEMS

The nature and relative merits of two measures of information retrieval system effectiveness, documents read and relevant references retrieved, are discussed. Using data on 2,380 searches completed at London University's Central Information Service, the author presents a mathematical model for the relationship between these two variables. Data suggest that this is best modelled by a monotonically asymptotic function, i.e. by a function increasing to a limit. A negative exponential function provides the best fit to the data, though a simpler hyperbolic function also provides a good fit. The suggested relationship between documents read and relevant references retrieved is further confirmed when the data is analysed by subject area of the user group. The major distinction between groups was the limiting value for mean number of documents read. Medical scientists read considerably more than the average, while engineers read considerably less. Biological, physical and social scientists were indistinguishable from the population as a whole within the statistical limitations of the data. Potential areas for further research are suggested.

Automatic Information Retrieval

Advances such as specialized parallel hardware and new algorithms for text searching will improve the effectiveness of information retrieval systems.

An associative search strategy for information retrieval

This paper extends Goffman's Indirect Method of Information Retrieval by suggesting a more flexible search strategy. The suggested strategy in its simplified version is more effective than the Boolean strategy used in existing information retrieval systems and can be implemented at comparable costs. Some of its features might further increase the effectiveness of information retrieval systems.

Relevance assessments and retrieval system evaluation

Two widely used criteria for evaluating the effectiveness of information retrieval systems are, respectively, the recall and the precision. Since the determination of these measures is dependent on a distinction between documents which are relevant to a given query and documents which are not relevant to that query, it has sometimes been claimed that an accurate, generally valid evaluation cannot be based on recall and precision measures. A study was made to determine the effect of variations in relevance assessments on the average recall and precision values used to measure retrieval effectiveness. Using a collection of 1200 documents in information science for test purposes, it is found that large scale differences in the relevance assessments do not produce significant variations in average recall and precision. It thus appears that properly computed recall and precision data may represent effectiveness indicators which are generally valid for many distinct user classes.

OPENING THE BLACK BOX OF ‘RELEVANCE’

The purpose of this project was to identify variables thought to affect relevance judgments and conduct a series of laboratory studies to determine the effects of these variables on relevance judgments. This paper discusses the variable of ‘implicit use orientations’—the particular attitude taken by a subject (judge) about the intended use of a document. One hundred and forty judges rated each of nine abstracts for relevance to several short information requirement statements. The same judges then repeated the ratings, each adopting (assuming) one of fourteen use orientations described to them. It was found that the particular use orientation assumed by the judge has a marked effect on relevance judgments. It was also found that implicit use orientations can be analysed in terms of their underlying structure and that their study offers the possibility of discovering the conditions under which one person can accurately simulate and use the implicit use orientations of another. Relevance judgments have been used as a basis for measures designed to evaluate the effectiveness of information retrieval systems. These judgments have usually been accepted at face value and have not been subjected to critical scrutiny. There is reason to believe, however, that as ordinarily obtained, they may be unreliable and sensitive to a number of conditions of measurement that have not been carefully controlled in previous evaluation studies.