Glean: using syntactic information in document filtering

Abstract

In this paper, we describe a system called Glean, which is based on the idea that coherent text contains significant latent information, such as syntactic structure and patterns of language use, which can be used to enhance the performance of information retrieval systems. We propose an approach to increase the precision of information retrieval that makes use of syntactic information obtained using a supertagger. In this approach, patterns based on local syntactic context are induced from training material. These patterns are used to refine the set of documents retrieved by a standard Web search engine or an information retrieval system, by selecting relevant information and filtering out irrelevant items. We show that syntactic information does improve the effectiveness of filtering irrelevant documents, and that supertagging is more effective than part of speech tagging in filtering documents. Further, we also show how the extent of syntactic context affects filtering performance. We discuss the relationship between Glean and other attempts at improving information retrieval performance.