Relevance criteria dynamics: A study of online news selection on SERPs

Abstract

ABSTRACTA lab‐based experiment was conducted to understand the multidimensional and dynamic changes in the use of relevance criteria on online news on search engine result pages (SERPs). Data from retrospective think‐aloud (RTA) was first analyzed to investigate the criteria used when participants selected a search result on SERPs. The number and frequency of criteria used for relevance judgments were calculated. The criteria used on subsequent SERPs visits and on results at different ranks on SERPs were then analyzed to understand the differences in relevance criteria that were applied. The results showed that among the top six identified criteria, specific source and topicality were the two most often used relevance criteria. 59.6% of the times partic‐ ipants used a single criterion to make relevance judgment. Considering how criteria changed on subsequent visits to SERPs and on search results at different ranks, specific source was the most often used criterion initially but was then surpassed by topicality on subsequent visits to SERPs and on results at lower ranks. This could be explained by users using specific source as the primary cue in information evaluation and by users' expectation that Google's search engine algorithm shows more relevant information on the first SERP and at the top result ranks.