On the Measurement of Inter-Linker Consistency and Retrieval Effectiveness in Hypertext Databases

Abstract

An important stage in the process of retrieval of objects from a hypertext database is the creation of a set of inter-nodal links that are intended to represent the relationships existing between objects; this operation is often undertaken manually, just as index terms are often manually assigned to documents in a conventional retrieval system. In this paper, a study is reported in which several different sets of links were inserted, each by a different person, between the paragraphs of each of a number of full-text documents. The degree of similarity between the members of each pair of link-sets (i.e., the degree of inter-linker consistency) was then evaluated. The results indicated that little similarity existed amongst the link-sets, a finding that is comparable with those of studies of inter-indexer consistency, which suggest that there is generally only a low level of agreement between the sets of index terms assigned to a document by different indexers. These latter studies have historically been considered significant on account of their common assumption that there exists a positive relationship between recorded levels of inter-indexer consistency and the levels of retrieval effectiveness that may be achieved by the systems studied. In order to test the validity of making a similar assumption in the context of link-assignment, the paper continues with a description of an investigation into the nature of the relationship existing between (i) the levels of inter-linker consistency obtaining among the group of hypertext databases used in our earlier experiments and (ii) the levels of effectiveness of a number of searches carried out in those databases. An account is given of the implementation of the searches and of the methods used in the calculation of numerical values expressing their effectiveness, and conclusions are drawn regarding the consistency-effectiveness relationship.