Direct comparison of commercial and academic retrieval system

Abstract

Searching for information has increasingly prevailed in people’s life. A new trend in the study of information retrieval is the amplified interest in search over corporation and personal information collections. Recently released commercial desktop search engines are the achievements due to this trend. Comparing to the Web search engines, these desktop search engines are working on collections with relatively small size, and often rely on the machine power of a single desktop computer. These new developments provide an interesting opportunity for the evaluation of academic retrieval systems. Academic retrieval systems are systems developed in academic environment for research use. Many of them are open source systems. For many years, due to various reasons like collection size and machine resources, commercial search engines and academic retrieval systems are studied and evaluated in non-interrelated environments. Now, since commercial desktop search engines and academic retrieval systems are working on collections with similar size and on comparable machine power, it becomes possible to directly compare their performance. In this paper, we want to compare two off-shelf retrieval systems. They are Google Desktop (GDS) v20051208 and Indri 2.0. GDS is a popular desktop search engine, which provides full text search on various types of files in a personal computer, including emails. Google didn’t disclose what is the retrieval model used in GDS. Indri 2.0, is a state-of-art retrieval system developed by University of Massachusetts Amherst and Carnegie Mellon University. It combines an