Interactive retrieval using simulated versus real work task situations: Differences in sub‐facets of tasks and interaction performance

Abstract

AbstractTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on users' interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty‐two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre‐ and a post‐search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that simulated and real work task situations are significantly different in some sub‐facets of task, but not all sub‐facets examined in this study. However, the influence of the two tasks on users' interactive information search behavior and interaction performance is not significant, though the sub‐facets of the real and simulated work task situation are correlated to interaction performance differently. Therefore, simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub‐facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.