What eyes can tell about the use of relevance criteria during predictive relevance judgment?

Abstract

This paper reports on the preliminary findings of a user study that explored how searchers fixate on information associated with different relevance criteria during the process of predictive relevance judgment. In order to address this objective a user study was conducted that involved the completion of questionnaires, use of eye tracking technology, talk aloud protocols and post-search interviews. As opposed to previous studies, the present research asked participants to search for real information needs that represented different search contexts (e.g. from searches about personal interest to academic related searches). This permitted the identification of several relevance criteria that naturally occur across different search contexts and the emergence of some fixation patterns, not observed before, associated to the use of these criteria. The paper concludes with a discussion of the impact and implication of this study in the wider context of relevance judgment and information seeking in context research.