Measuring search-engine quality and query difficulty: Ranking with target and freestyle

Abstract

Instead of using traditional performance measures such as precision and recall, information retrieval performance may be measured by considering the probability that the search engine is optimal and the difficulty associated with retrieving documents with a given query or on a given topic. These measures of desirable characteristics are more easily and more directly interpretable than are traditional measures. The performance of the Target and Freestyle search engines is examined, and is very good. Each query in the CF database is assigned a difficulty number, and these numbers are found to strongly correlate with other measures of retrieval performance such as an E or F value. The query difficulty correlates weakly with query length.