Information retrieval on the connection machine: 1 to 8192 gigabytes

Abstract

This paper describes algorithms and data structures for applying a parallel computer to information retrieval. Previous work has described an implementation based on overlap encoded signatures. That system was limited by (a) the necessity of keeping the signatures in primary memory and (b) the difficulties involved in implementing document-term weighting. Overcoming these limitations required adapting the inverted index techniques used on serial machines. The most obvious adaptation, also previously described, suffers from the fact that data must be sent between processors at query time. Since interprocessor communication is generally slower than local computation, this suggests that an algorithm which does not perform such communication might be faster. This paper presents a data structure, called a partitioned posting file, in which the interprocessor communication takes place at database-construction time, so that no data movement is needed at query-time. Performance characteristics and storage overhead are established by benchmarking against a synthetic database. Based on these figures, it appears that currently available hardware can deliver interactive document ranking on databases containing between 1 and 8192 Gigabytes of text.