Retrieval expectations, cluster-based effectiveness, and performance standards in the CF database

Abstract

Retrieval performance of controlled and uncontrolled subject representations is investigated as a function of retrieval expectations in the cystic fibrosis (CF) database. Controlled subject representations are based on Medical Subject Headings (MeSH), and uncontrolled subject representations are based on word stems derived from titles and abstracts. Retrieval expectations are varied systematically from comprehensive (in which marginally relevant documents are acceptable) to specific (in which only highly relevant documents are sought). A maximally effective cluster-based search strategy selects the most appropriate cluster from the set of available clusters for each query and isolates retrieval effectiveness from the uncertainties of an operational search strategy and the competence of a searcher. A maximally effective feedback strategy optimizes the exhaustivity of subject representations for each query. Retrieval performance measures reveal the capacity of each representation to associate documents relevant to the same query and disassociate those that are not relevant in single-link hierarchies. Controlled and uncontrolled subject representations produce equivalent levels of performance at all levels of retrieval expectation. Combinations of retrieval outcomes are consistently more effective than outcomes of either subject representation. Low performance standards, derived from random structures, distinguish between unacceptably low and questionable levels of performance and between questionable and acceptable levels of performance in the CF database. A distinction between acceptable and high levels of performance is based on normative considerations. The retrieval effectiveness associated with each query is characterized as unacceptable, questionable, acceptable, or high. Results suggest that the low performance standards constitute a severe test of the utility of search strategies or document representations in the CF database.