Employing multiple representations for Chinese information retrieval

Abstract

For information retrieval in the Chinese language, three representation methods for texts are popular, namely: 1-gram or character, bigram, and short-word. Each has its advantages as well as drawbacks. Employing more than one method may combine advantages from them and enhance retrieval effectiveness. We investigated two ways of using them simultaneously: mixing representations in documents and queries, and combining retrieval lists obtained via different representations. The experiments were done with the 170 MB evaluated Chinese corpora and 54 long and short queries available from the (TREC) program and using our Probabilistic Indexing and Retrieval Components System (PIRCS retrieval system). Experiments show that good retrieval need not depend on accurate word segmentation; approximate segmentation into short-words will do. Results also show and confirm that bigram representation alone works well; mixing characters with bigram representation boosts effectiveness further, but it is preferable to mix characters with short-word indexing which is more efficient, needs less resource, and gives better retrieval more often. Combining retrieval lists from short-word with character representation and from bigram indexing provides the best retrieval results but also at a substantial cost.