How do users handle inconsistent information?

Abstract

While search engines sometimes return different documents containing contradictory answers, little is known about how users handle inconsistent information. This paper investigates the effect of search expertise (defined as specialized knowledge on the internal workings of search engines) on search behavior and satisfaction criteria of users. We selected four tasks comprising factoid questions with inconsistent answers, extracted answers that 30 study participants had found in these tasks, and analyzed their answer-finding behavior in terms of the presence or absence of search expertise. Our main findings are as follows: (1) finding inconsistent answers causes users with search expertise (search experts) to feel dissatisfied, while effort in searching for answers is the dominant factor in task satisfaction for those without search expertise (search non-experts); (2) search experts tend to spend longer completing tasks than search non-experts even after finding possible answers; and (3) search experts narrow down the scope of searches to promising answers as time passes as opposed to search non-experts, who search for any answers even in the closing stage of task sessions. These findings suggest that search non-experts tend to be less concerned about the consistency in their found answers, on the basis of which we discuss the design implications for making search non-experts aware of the existence of inconsistent answers and helping them to search for supporting evidence for answers.