Detecting context-differentiating terms using competitive learning

Abstract

Personal information agents monitor ongoing user information accesses in order to provide users with context-relevant information. Providing the needed information requires effective methods for identifying the user's task context, based on available information. For user browsing tasks, one approach to context identification is to extract context-determining terms from the documents that the user consults. The thesis of this article is (1) that term extraction for personal information agents can be done by learning terms whose occurrence frequencies have a large variance over time, (2) that indexing and retrieval based on these terms can be at least as effective as standard information retrieval techniques, and (3) that this information can be learned without comprehensive corpus analysis, making it suitable for use in personal information retrieval.We have developed an unsupervised term extraction algorithm, WordSieve, that learns individualized context-differentiating terms for document indexing and retrieval. This article presents a new version of WordSieve, compares its design and performance to our initial approach, and assesses its effectiveness for a controlled personal information retrieval task, compared to three common indexing techniques requiring statistics about the global corpus. In the experiments, the new version of WordSieve generates task-relevant indices of comparable or better quality to common indexing techniques, using only local information.