The use of relevance criteria during predictive judgment: An eye tracking approach

Abstract

This paper reports on the findings of a user study that explored how searchers tend to fixate on information associated with different relevance criteria in the web during the relevance judgment process. The user study involved the completion of questionnaires, use of eye tracking technology, talk aloud protocols and post-search interviews. As opposed to previous studies, the present research asked participants to search for real information needs that represented different search contexts (e.g. from searches about personal interest to academic related searches). This permitted the identification of several relevance criteria that naturally occur across different search contexts and the emergence of some fixation patterns, not observed before, associated to the use of these criteria. Although the study examined participants' eye movements for both predictive and evaluative relevance judgments, this paper is focused on the findings of the predictive relevance judgment process and specifically participants' evaluation of the results presented in the search result interface of the Google search engine. It is anticipated that the findings reported in this paper could shed light on the process of predictive relevance judgment and especially on the problem of relevance criteria use through the use of eye tracking.