USERS' PARTIAL RELEVANCE JUDGEMENTS DURING ONLINE SEARCHING

Abstract

Users' relevance judgements are central to both the systems and user‐oriented approaches to information retrieval (IR). A basic assumption of IR and online searching research has been that users always require the most ‘highly’ relevant items when using an IR system. This paper reports the results from research that sought to examine users conducting their initial online search on a particular information problem. Findings from three separate studies of relevance judgements by 44 initial search users were examined, including two studies of 13 end‐users and a study of 18 users engaged in mediated online searches. Results show that the number of items judged ‘partially’ relevant (on the scale: relevant; partially relevant; not relevant) was found to correlate positively with changes in users': (1) criteria for making relevance judgements; (2) information problem definition; and (3) personal knowledge due to the search interaction; and also (4) search intermediaries' perceptions that a user's question and information problem has changed during the mediated search interaction. Items judged ‘highly’ relevant were not correlated with these factors. Results of the three studies suggest that: (1) a relationship exists between partially relevant items retrieved and changes in the users' information problem or question during an information seeking process; (2) partial relevance judgements play an important role for users in the early stages of seeking information on a particidar information problem; and (3) ‘highly’ relevant items may or may not be the only items useful at the early stages of users' information seeking processes. Implications for the development of IR systems, relevance research and searching practice are also examined.