Regions and levels: Measuring and mapping users' relevance judgments

Abstract

The dichotomous bipolar approach to relevance has produced an abundance of information retrieval (IR) research. However, relevance studies that include consideration of users' partial relevance judgments are moving to a greater relevance clarity and congruity to impact the design of more effective IR systems. The study reported in this paper investigates the various regions of across a distribution of users' relevance judgments, including how these regions may be categorized, measured, and evaluated. An instrument was designed using four scales for collecting, measuring, and describing end-user relevance judgments. The instrument was administered to 21 end-users who conducted searches on their own information problems and made relevance judgments on a total of 1059 retrieved items. Findings include: (1) overlapping regions of relevance were found to impact the usefulness of precision ratios as a measure of IR system effectiveness, (2) both positive and negative levels of relevance are important to users as they make relevance judgments, (3) topicality was used more to reject rather than accept items as highly relevant, (4) utility was more used to judge items highly relevant, and (5) the nature of relevance judgment distribution suggested a new IR evaluation measure—median effect. Findings suggest that the middle region of a distribution of relevance judgments, also called “partial relevance,” represents a key avenue for ongoing study. The findings provide implications for relevance theory, and the evaluation of IR systems.