Performance Of Information Retrieval Systems Research Articles

Glean: using syntactic information in document filtering

In this paper, we describe a system called Glean, which is based on the idea that coherent text contains significant latent information, such as syntactic structure and patterns of language use, which can be used to enhance the performance of information retrieval systems. We propose an approach to increase the precision of information retrieval that makes use of syntactic information obtained using a supertagger. In this approach, patterns based on local syntactic context are induced from training material. These patterns are used to refine the set of documents retrieved by a standard Web search engine or an information retrieval system, by selecting relevant information and filtering out irrelevant items. We show that syntactic information does improve the effectiveness of filtering irrelevant documents, and that supertagging is more effective than part of speech tagging in filtering documents. Further, we also show how the extent of syntactic context affects filtering performance. We discuss the relationship between Glean and other attempts at improving information retrieval performance.

The use of genetic programming to build Boolean queries for text retrieval through relevance feedback

We report here on preliminary work on the use of evolutionary computing techniques which aims to improve Boolean information retrieval system performance through relevance feedback. There are many evolutionary techniques in computing, such as neural networks and genetic algorithms. One specific form of genetic algorithm technique has been used in our study: that of genetic programming. Terms from relevant documents are used to randomly create Boolean queries. Boolean queries are thought of as genetic programming organisms and, as such, are used for breeding to produce new organisms. Organisms which perform well, in terms of how good they are at retrieval, are given a better chance of being selected for breeding, with the result being that the overall fitness of the organisms improve to some extent. The aim is to develop the best Boolean query for an information need, given a small corpus of test documents, and then to use that query on the full collection to retrieve yet more relevant documents.

Comparing Boolean and probabilistic information retrieval systems across queries and disciplines

Whether using Boolean queries or ranking documents using document and term weights will result in better retrieval performance has been the subject of considerable discussion among document retrieval system users and researchers. We suggest a method that allows one to analytically compare the two approaches to retrieval and examine their relative merits. The performance of information retrieval systems may be determined either by using experimental simulation, or through the application of analytic techniques that directly estimate the retrieval performance, given values for query and database characteristics. Using these performance predicting techniques, sample performance figures are provided for queries using the Boolean and and or, as well as for probabilistic systems assuming statistical term independence or term dependence. The variation of performance across sublanguages (used in different academic disciplines) and queries is examined. The performance of models failing to meet statistical and other assumptions is examined.

Inverted file partitioning schemes in multiple disk systems

Multiple-disk I/O systems (disk arrays) have been an attractive approach to meet high performance I/O demands in data intensive applications such as information retrieval systems. When we partition and distribute files across multiple disks to exploit the potential for I/O parallelism, a balanced I/O workload distribution becomes important for good performance. Naturally, the performance of a parallel information retrieval system using an inverted file structure is affected by the partitioning scheme of the inverted file. In this paper, we propose two different partitioning schemes for an inverted file system for a shared-everything multiprocessor machine with multiple disks. We study the performance of these schemes by simulation under a number of workloads where the term frequencies in the documents are varied, the term frequencies in the queries are varied, the number of disks are varied and the multiprogramming level is varied. >

Term dependence: Truncating the Bahadur Lazarsfeld expansion

The performance of probabilistic information retrieval systems is studied where differing statistical dependence assumptions are used when estimating the probabilities inherent in the retrieval model. Experimental results using the Bahadur Lazarsfeld expansion suggest that the greatest degree of performance increase is achieved by incorporating term dependence information in estimating Pr( d¦ rel). It is suggested that incorporating dependence in Pr( d¦ rel) to degree 3 be used; incorporating more dependence information results in relatively little increase in performance. Experiments examine the span of dependence in natural language text, the window of terms in which dependencies are computed, and their effect on information retrieval performance. Results provide additional support for the notion of a window of ± 3 to ± 5 terms in width; terms in this window may be most useful when computing dependence.

Variations in relevance judgments and the evaluation of retrieval performance

The relevance judgments used to evaluate the performance of information retrieval systems are known to vary among judges and to vary under certain conditions extraneous to the relevance relationship between queries and documents. The study reported here investigated the degree to which variations in relevance judgments affect the evaluation of retrieval performance. Four sets of relevance judgments were used to test the retrieval effectiveness of six document representations. In no case was there a noticeable or material difference in retrieval performance due to variations in relevance judgment. Additionally, for each set of relevance judgments, the relative performance of the six different document representations was the same. Reasons why variations in relevance judgments may not affect recall and precision results were examined in further detail.

An analytic measure predicting information retrieval system performance

The performance of information retrieval, hypertext linkage, and text filtering systems may be measured by using historical data or by estimating performance using Bayesian probabilistic or artificial intelligence methods. The measurement of performance is necessary to evaluate document retrieval systems, electronic mail filters, office information systems, and, in general, retrieval from databases when the searcher has incomplete information about the characteristics of the records to be retrieved. We provide a method for estimating precision or retrieval quality without examining individual database documents. This method requires knowledge of only the query or expressed information need and a set of database parameters constant for all queries. The concepts of historic and expected precision are examined. The analytic expected precision measure is used to examine the performance of a system using relevance feedback to increase the accuracy of parameter estimates. Use of a precision-document graph instead of the commonly used precision-recall graph is examined, and several uses of the precision-document graph for a computer-human interface are suggested, including its use as a graphic aid assisting users in deciding when to stop searching. An economic model of information retrieval and the stopping problem is provided, and the conditions are examined under which the user should stop searching when either using or not using relevance feedback.

A sensitivity analysis of a probabilistic information retrieval system

Results of a set of exploratory simulations to test the effects of errors in estimation of individual term probabilities on the performance of a probabilistic information retrieval system are presented. Searches were executed with various levels of term error on a test collection of probabilistically indexed information sources. The amount of error in the final probability of relevance used to rank sources introduced by these errors was analytically determined. The first simulation analyzed simulated rankings obtained by simulating probabilities of relevance according to the error distribution; in the second simulation measures of rank correlation between simulated rankings and the system's ranking were calculated; in the third, the effect of the error on retrieval performance using the measure expected search length was determined. It was found that substantial error was introduced into final probabilities of relevance, but that for low levels of term error the impact on ranking and retrieval performance was moderate, while even with high levels the actual ranking performed significantly better than a random ranking of retrieved sources. © 1990 John Wiley & Sons, Inc.

INDEX: The statistical basis for an automatic conceptual phrase-indexing system

In recent years researchers have become increasingly convinced that the performance of information retrieval systems can be greatly enhanced by the use of key phrases for automatic conceptual document indexing and retrieval. In this article we describe two programs, INDEX and INDEXD, which locate repeated phrases in a document, gather statistical information about them, and rank them according to their value as index phrases. The programs show promise as the basis for a sophisticated conceptual indexing system. The simpler program, INDEX, ranks phrases in such a way that frequently occurring phrases which contain several frequently occurring words are given a high ranking. INDEXD is an extension of INDEX which incorporates a dictionary for stemming, weighting of words and validation of syntax of output phrases. Sample output of both programs is included, and we discuss plans to combine INDEXD with linguistic and artificial intelligence techniques to provide a general conceptual phrase-indexing system that can incorporate expert knowledge about a given application area. © 1990 John Wiley & Sons, Inc.

The effectiveness of information retrieval systems: the need for improved explanatory frameworks

There is a critical lacuna in information retrieval research between the theoretical framework employed in laboratory tests to explain the performance of information retrieval systems, and the sorts of factors which bear on their operational effectiveness. The influence of assumptions as to which factors it is important to take into account in evaluations of the effectiveness of information retrieval systems, derived from the laboratory testing approach in the form of an implicit model of the retrieval situation, has had an inhibiting effect on information retrieval research. Evaluations of the operational effectiveness of information retrieval systems are particularly hampered by the absence of a suitable conceptual framework of explanation. The argument of this paper is that there is a need for more microevaluation of the activities and environments of the users of information systems in order to develop an understanding of the relation of information services to those activities and environments. This sort of understanding should provide more in the way of guidance for the design of more effective systems than is available from current information retrieval theory.

Mathematical models of information system use

This report presents the results from a study of mathematical models relating to the usage of information systems. For each of four models, the papers developed during the study provide three types of analyses: reviews of the literature relevant to the model, analytical studies, and tests of the models with data drawn from specific operational situations. (1) The Cobb-Douglas model: x 0 = ax 1 b x 2 (1− b) . This classic production model, normally interpreted as applying to the relationship between production, labor, and capital, is applied to a number of information related contexts. These include specifically the performance of libraries, both public and academic, and the use of information resources by the nation's industry. The results confirm not only the utility of the Cobb-Douglas model in evaluation of the use of information resources, but demonstrate the extent to which those resources currently are being used at significantly less than optimum levels. (2) Mixture of Poissons: χ 0 = ∑ i=0 n i ∑ j=0 p n je mj(m j)′/i! where x 0 is the usage and ( n j, m j), j = 0 to p, are the p + 1 components of the distribution. This model of heterogeneity is applied to the usage of library materials and of thesaurus terms. In each case, both the applicability and the analytical value of the model are demonstrated. (3) Inverse effects of distance: x = a e − md if c(d) = rd x = ad − m if c(d) = r log( d). These two models reflect different inverse effects of distance, the choice depending upon the cost of transportation. If the cost, c(d), is linear, the usage is inverse exponential; if logarithmic, the usage is inverse power. The literature that discusses the relationship between usage of facilities and the distance from them is reviewed. The models are tested with data from the usage of the Los Angeles Public Library, both Central Library and branches, based on a survey of 3662 users. (4) Weighted entropy: S(x 1,x 2,...,x n)= - ∑ i=1 n r(x iP(x i) log(p(x i)). This generalization of the “entropy measure of information” is designed to accommodate the effects of “relevancy”, as measured by r(x), upon the performance of information retrieval systems. The relevant literature is reviewed and the application to retrieval systems is considered.

Spatial representations of knowledge

This research examined the relationship between subjects' knowledge of an area and their similarity ratings of pairs of concepts. Knowledge was operationalized as (1) test score performance, and (2) correspondence with an expert's similarity judgments. Subjects (n=55) were exposed to a three-week module on social psychology as part of their introductory psychology course. At the end of this course segment they were administered an examination and, later, a term similarity questionnaire. Students' similarity ratings related to both exam score and similarity ratings of the expert. Furthermore, as expected, test performance was strongly related (r=+.55) to the correlation with the expert's similarity judgments. A model based on term similarity judgments was proposed as a theoretical explanation of how term dependence models influence information retrieval system performance.

A threshold rule applied to the retrieval decision model

AbstractThe retrieval decision problem is considered from the viewpoint of a decision theory approach. A threshold rule based on a suggested rule for indexing is considered and analyzed for retrieval decisions. The threshold rule is seen as a good descriptive measure of what a reasonable retrieval system should be able to accomplish. A retrieval mechanism of randomly drawing documents is analyzed to determine the relative strength of the threshold rule. The Neyman‐Pearson lemma is shown to yield a better a priori decision rule for retrieval; maximize precision subject to a fixed level of recall, instead of setting a lower limit upon precision, as does the threshold rule. The threshold rule is seen as a necessary, but not sufficient, condition for effective retrieval. A sufficient condition for the threshold rule illustrates the relationship between it and the rule derived from the Neyman‐Pearson lemma. Finally, a new measure of information retrieval system performance is presented, based on the threshold rule.

Effective information retrieval using term accuracy

The performance of information retrieval systems can be evaluated in a number of different ways. Much of the published evaluation work is based on measuring the retrieval performance of an average user query. Unfortunately, formal proofs are difficult to construct for the average case. In the present study, retrieval evaluation is based on optimizing the performance of a specific user query. The concept of query term accuracy is introduced as the probability of occurrence of a query term in the documents relevant to that query. By relating term accuracy to the frequency of occurrence of the term in the documents of a collection it is possible to give formal proofs of the effectiveness with respect to a given user query of a number of automatic indexing systems that have been used successfully in experimental situations. Among these are inverse document frequency weighting, thesaurus construction, and phrase generation.

Grammatically-based automatic word class formation

Most previous attempts at producing word classes (thesauri) by statistical analysis have used very limited distributional information such as word co-occurrence in a document or a sentence. This paper describes an automatic procedure which uses the syntactic relations as the basis for grouping words into classes. It forms classes by grouping together nouns that occur as subject (or object) of the same verbs, and similarly by grouping together verbs occurring with the same subject or object. The program was applied to a small corpus of sentences in a subfield of pharmacology. This procedure yielded the word classes for the subfield, in good agreement with the word classes recognized by pharmacologists. The word classes can be used to describe the informational patterns that occur in texts of the subfield, to disambiguate parses of a sentence, and perhaps to improve the performance of current information retrieval systems.