CONCEPT-BASED TERM WEIGHTING FOR WEB INFORMATION RETRIEVAL

Abstract

In this paper we present a novel technique for determining term importance by exploiting concept-based information found in ontologies. Calculating term importance is a significant and fundamental aspect of most information retrieval approaches, and it is traditionally determined through inverse document frequency (IDF). We propose concept-based term weighting (CBW), a technique that is fundamentally different to IDF in that it calculates term importance by intuitively interpreting the conceptual information in ontologies. We show that when CBW is used in an approach for web information retrieval on benchmark data, it performs comparatively to IDF, with only a 3.5% degradation in retrieval accuracy. While this small degradation has been observed, the significance of this technique is that (1) unlike IDF, CBW is independent of document collection statistics, (2) it presents a new way of interpreting ontologies for retrieval, and (3) it introduces an additional source of term importance information that can be used for term weighting.