Laboratory experiments with Okapi: participation in the TREC programme

Abstract

A brief review of the history of laboratory testing of information retrieval systems focuses on the idea of a general‐purpose test collection of documents, queries and relevance judgements. The TREC programme is introduced in this context, and an overview is given of the methods used in TREC. The Okapi team’s participation in TREC is then discussed. The team has made use of TREC to improve some of the automatic techniques used in Okapi, specifically the term weighting function and the algorithms for term selection for query expansion. The consequence of this process has been a very good showing for Okapi in terms of the TREC evaluation results. Some of the issues around the much more difficult problem of interactive evaluation in TREC are then discussed. Although some interesting interactive experiments have been performed at TREC, the problems of reconciling the requirements of the laboratory context with the concerns of interactive retrieval are still largely unresolved.