The cost-performance of an on-line, free-text bibliographic retrieval system

Abstract

The cost-performance evaluation of the SUPARS system is reported. SUPARS was an on-line, free-text bibliographic retrieval system; cost-effectivness data of such systems are not readily available. In our evaluation, two measures of cost were employed : a computer processing charge expressed in dollars, and the number of documents retrieved (a measure of work that must be expended to review the retrieved items). The measure of performance was an estimate of the recall ratio. To obtain the requisite measures an experimental plan was developed in which experts searched the data base of Psychological Abstracts forming their queries from written statements of information needs. These statements (along with a list of documents relevant to them) were produced by people with information problems. Tallies were kept of the number of documents retrieved before each of the designated relevant items were found. The major findings are noted below. (1) Queries to the system employing simple Boolean operators (AND, OR) have better cost-performance characteristics than queries using more elegant searching operators. (2) On-demand access to the index or dictionary contributes sizeably to improving the cost-performance of the system. (3) The argument is raised that human factors, such as the differences among users of a system, probably should be a major factor in the design, operation and evaluation of retrieval systems. It appears that consideration of these factors will improve system cost-performance.