Set Of Index Terms Research Articles

Set-based vector model

This work presents a new approach for ranking documents in the vector space model. The novelty lies in two fronts. First, patterns of term co-occurrence are taken into account and are processed efficiently. Second, term weights are generated using a data mining technique called association rules. This leads to a new ranking mechanism called the set-based vector model . The components of our model are no longer index terms but index termsets, where a termset is a set of index terms. Termsets capture the intuition that semantically related terms appear close to each other in a document. They can be efficiently obtained by limiting the computation to small passages of text. Once termsets have been computed, the ranking is calculated as a function of the termset frequency in the document and its scarcity in the document collection. Experimental results show that the set-based vector model improves average precision for all collections and query types evaluated, while keeping computational costs small. For the 2-gigabyte TREC-8 collection, the set-based vector model leads to a gain in average precision figures of 14.7% and 16.4% for disjunctive and conjunctive queries, respectively, with respect to the standard vector space model. These gains increase to 24.9% and 30.0%, respectively, when proximity information is taken into account. Query processing times are larger but, on average, still comparable to those obtained with the standard vector model (increases in processing time varied from 30% to 300%). Our results suggest that the set-based vector model provides a correlation-based ranking formula that is effective with general collections and computationally practical.

A text mining approach for automatic construction of hypertexts

The research on automatic hypertext construction emerges rapidly in the last decade because there exists a urgent need to translate the gigantic amount of legacy documents into web pages. Unlike traditional 'flat' texts, a hypertext contains a number of navigational hyperlinks that point to some related hypertexts or locations of the same hypertext. Traditionally, these hyperlinks were constructed by the creators of the web pages with or without the help of some authoring tools. However, the gigantic amount of documents produced each day prevent from such manual construction. Thus an automatic hypertext construction method is necessary for content providers to efficiently produce adequate information that can be used by web surfers. Although most of the web pages contain a number of non-textual data such as images, sounds, and video clips, text data still contribute the major part of information about the pages. Therefore, it is not surprising that most of automatic hypertext construction methods inherit from traditional information retrieval research. In this work, we will propose a new automatic hypertext construction method based on a text mining approach. Our method applies the self-organizing map algorithm to cluster some at text documents in a training corpus and generate two maps. We then use these maps to identify the sources and destinations of some important hyperlinks within these training documents. The constructed hyperlinks are then inserted into the training documents to translate them into hypertext form. Such translated documents will form the new corpus. Incoming documents can also be translated into hypertext form and added to the corpus through the same approach. Our method had been tested on a set of at text documents collected from a newswire site. Although we only use Chinese text documents, our approach can be applied to any documents that can be transformed to a set of index terms.

Navigation via similarity: Automatic linking based on semantic closeness

Notions of similarity underlie a wide variety of access methods in hypermedia and information retrieval. This paper describes a research project, in which similarity measures have been extended to include imprecise matching over different dimensions of structured classification schemes (subject, space, time). The semantic similarity of information units forms the basis for the automatic construction of links and is integrated into hypermedia navigation. A semantic hypermedia architecture is outlined, and a prototype museum social history application is described. Illustrative navigation scenarios are presented which make use of a navigation via similarity tool. Three different measures of semantic closeness underpin the similarity tool. The temporal measure takes account of periods as well as time points. The most general measure is based on a traversal of a semantic net, taking into account relationship type and level of specialization. It is based on a notion of closeness rather than absolute distance, and returns a set of semantically close terms. A method of calculating semantic similarity between sets of index terms, based on the maximal closeness values achieved by each term is discussed.

On the creation of hypertext links in full‐text documents: Measurement of retrieval effectiveness

An important stage in the process of retrieval of objects from a hypertext database is the creation of a set of internodal links that are intended to represent the relationships existing between objects; this operation is often undertaken manually, just as index terms are often manually assigned to documents in a conventional retrieval system. In an earlier article (Ellis, D., Furner-Hines, J., & Willett, P., 1994b), the results were published of a study in which several different sets of links were inserted, each by a different person, between the paragraphs of each of a number of full-text documents. These results showed little similarity between the link-sets, a finding that was comparable with those of studies of inter-indexer consistency, which suggest that there is generally only a low level of agreement between the sets of index terms assigned to a document by different indexers. In this article, a description is provided of an investigation into the nature of the relationship existing between (i) the levels of inter-linker consistency obtaining among the group of hypertext databases used in our earlier experiments, and (ii) the levels of effectiveness of a number of searches carried out in those databases. An account is given of the implementation of the searches and of the methods used in the calculation of numerical values expressing their effectiveness. Analysis of the results of a comparison between recorded levels of consistency and those of effectiveness does not allow us to draw conclusions about the consistency-effectiveness relationship that are equivalent to those drawn in comparable studies of inter-indexer consistency. © 1996 John Wiley & Sons, Inc.

All in the mind: concept analysis in indexing

The indexing process consists of the comprehension of the document to be indexed, followed by the production of a set of index terms. Differences between academic indexing and back-of-book indexing are discussed. Text comprehension is a branch of human information processing, and it is argued that the model of text comprehension and production developed by van Dijk and Kintsch can form the basis for a cognitive process model of indexing. Strategies for testing such a model are suggested.

ON THE CREATION OF HYPERTEXT LINKS IN FULL‐TEXT DOCUMENTS: MEASUREMENT OF INTER‐LINKER CONSISTENCY

An important stage in the process of retrieval of objects from a hypertext database is the creation of a set of inter‐nodal links that are intended to represent the relationships existing between objects; this operation is often undertaken manually, just as index terms are often manually assigned to documents in a conventional retrieval system. Studies of conventional systems have suggested that a degree of consistency in the terms assigned to documents by indexers is positively associated with retrieval effectiveness. It is thus of interest to investigate the consistency of assignment of links in separate hypertext versions of the same full‐text document, since a measure of agreement may be related to the subsequent utility of the resulting hypertext databases. The calculation of values indicating the degree of similarity between objects is a technique that has been widely used in the fields of textual and chemical information retrieval; in this paper, we describe the application of arithmetic coefficients and topological indices to the measurement of the degree of similarity between the sets of inter‐nodal links in hypertext databases. We publish the results of a study in which several different sets of links are inserted, by different people, between the paragraphs of each of a number of full‐text documents. Our results show little similarity between the sets of links identified by different people; this finding is comparable with those of studies of inter‐indexer consistency, where it has been found that there is generally only a low level of agreement between the sets of index terms assigned to a document by different indexers.

A probability distribution model for information retrieval

A probability distribution model for information retrieval is proposed in this article. In contrast to other approaches, each document in the new model is characterized by a simple probability measure on the set of index terms. Based on this interpretation of document representation, two different retrieval strategies are discussed. One is based on utility theory and the other is derived from information theory. The new model not only enhances retrieval effectiveness as demonstrated by experiments, but also provides valuable insight into many fundamental concepts introduced over the years in a variety of retrieval models.

Trends in research on information retrieval — The potential for improvements in conventional Boolean retrieval systems

Operational retrieval systems are firmly embedded within the pure Boolean framework, and the theoretical model underlying these systems is based on the implicit assumption that documents and user information needs can be precisely and completely characterized by sets of index terms and Boolean search request formulations, respectively. However, this assumption must be considered grossly inaccurate since uncertainty is intrinsic to the document retrieval process. The inability of the standard Boolean model to deal effectively with the inherent fallibility of retrieval decisions is the main reason for a number of serious deficiencies exhibited by present-day operational retrieval systems. This article reviews recent advances in information retrieval research and examines their practical potential for overcoming these deficiencies. The primary source for this review is the subsequent articles that comprise this special issue of Information Processing & Management, although earlier results published elsewhere have also been considered.

Probabilistic methods for ranking output documents in conventional Boolean retrieval systems

Current operational information retrieval systems based on Boolean searching could be radically improved through the incorporation of a weighting mechanism for ranking output documents. However, a number of previous attempts to refine conventional Boolean retrieval systems along these lines have not been fully successful because of their inherent inconsistencies and ambiguities. A detailed account of the research aimed at obtaining a more rigorous methodology is given in this article. The presented approach has been developed by extending well-known probabilistic output ranking methods that are applicable in retrieval systems in which document representations as well as search request formulations are simply sets of index terms. A series of experiments, carried out in recent years to verify some of these methods, have particularly demonstrated the value of a systematic statistical use of relevance feedback information. It is therefore expected that the application of the extended probabilistic document ranking methodology in conventional Boolean systems will also prove to be useful, and that considerable improvements in retrieval performance of these systems will be obtained. A theoretical framework used to derive the proposed output ranking scheme is described in detail. A simple illustrative example is included, followed by a thorough discussion of the suggested approach.

Towards a theory of document learning

AbstractIn a bibliographic information system the database consists of a set of document representatives. Part of each document representative is a list of index terms whose function is to identify the content of the document represented. These index terms can be derived using an automated process. It is argued that automatic methods do not result in the best set of index terms being assigned to a document representative. A mathematical model is proposed which describes an information system which includes a method, document learning, of improving the set of index terms assigned to a document representative.

Dynamic dictionary updating

A method for updating the dictionary in a dynamic information retrieval system is presented. It is shown that as a collection changes through addition and deletion of documents, the appropriate set of index terms may be determined without complete periodic regeneration of the dictionary. Results are presented for experiments involving a complete change in collection membership, with the dynamic dictionary updating methods shown to be effective.

Computer assisted indexing

The paper describes a technique which enables an on line computer based information retrieval system to aid the indexers by selecting possible index terms to be assigned to a new document entering the system. The method analyses the index terms assigned to the references cited by the new article. This produces a list of index terms weighted according to their expected correlation with the subject matter of the new article. Thus, the indexer is presented with a weighted list of proposed indexing terms as an aid. An evaluation is made of the results produced by the technique for a trial set of documents. These documents have already been indexed for the MEDLARS system. The list of actual index terms for this trial set of documents is compared with the set of index terms proposed by the technique for each document. The results of this comparison are encouraging, and they are discussed in the paper. The economic aspects of implementing the technique in a working information retrieval system is considered. This includes the expected benefits, and an estimate of the actual cost of using the technique as an aid in terms of computer time and indexer time.

Source Indexing of IEEE Publications

In his editorial in this issue of the Transactions, Dr. Schoenfeld introduces the IEEE catalog card program to members of the Bio-Medical Engineering Group. In future issues each paper will be assigned a catalog card which collects into one place the conventional bibliographic information such as author and title, the abstract, and a brief set of index terms describing the most significant topics in the papers. The index terms are intended to expedite efficient storage and retrieval of papers by subject matter rather than only by author, title and general subject category. The purpose of this paper is to illuminate the considerations that led to the introduction of the use of index terms and to indicate the scope of the overall IEEE information system program.