Interactive Information Retrieval Research Research Articles

Reflection on future directions: a systematic review of reported limitations and solutions in interactive information retrieval user studies

PurposeThe purpose of this paper is to understand how users behave and evaluate how systems with users are essential for interactive information retrieval (IIR) research. User study methodology serves as a primary approach to answering IIR research questions. In addition to designing user study procedures, understanding the limitations of varying study designs and discussing solutions to the limitations is also critical for improving the methods and advancing the knowledge in IIR.Design/methodology/approachGiven this unresolved gap, we apply the faceted framework developed by Liu and Shah (2019) in systematically reviewing 131 IIR user studies recently published (2016–2021) on multiple IR and information science venues.FindingsOur study achieve three goals: (1) extracting and synthesizing the reported limitations on multiple aspects of user study (e.g. recruitment, tasks, study procedures, system interfaces, data analysis methods) under associated facets; (2) summarizing the reported solutions to the limitations; (3) clarifying the connections between types of limitations and types of solutions.Practical implicationsThe bibliography of user studies can be used by students and junior researchers who are new to user-centered IR studies as references for study design. Our results can facilitate the reflection and improvement on IR research methodology and serve as a checklist for evaluating customized IIR user studies in varying problem spaces.Originality/valueTo our knowledge, this work is the first study that systematically reviews the study limitations and solutions reported by IIR researchers in articles and empirically examines their connections to different study components.

Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation

Re-using research resources is essential for advancing knowledge and developing repeatable, empirically solid experiments in scientific fields, including interactive information retrieval (IIR). Despite recent efforts on standardizing research re-use and documentation, how to quantitatively measure the reusability of IIR resources still remains an open challenge. Inspired by the reusability evaluations on Cranfield experiments, our work proactively explores the problem of measuring IIR test collection reusability and makes threefold contributions: (1) constructing a novel usefulness-oriented framework with specific analytical methods for evaluating the reusability of IIR test collections consisting of query sets, document/page sets, and sets of task-document usefulness (tuse); (2) explaining the potential impacts of varying IIR-specific factors (e.g. search tasks, sessions, user characteristics) on test collection reusability; (3) proposing actionable methods for building reusable test collections in IIR and thereby amortizing the true cost of user-oriented evaluations. The Cranfield-inspired reusability assessment framework serves as an initial step towards accurately evaluating the reusability of IIR research resources and measuring the reproducibility of IIR evaluation results. It also demonstrates an innovative approach to integrating the insights from individual heterogeneous user studies with the evaluation techniques developed in standardized ad hoc retrieval experiments, which will facilitate the maturation of IIR fields and eventually benefits both sides of research.

Interactive retrieval using simulated versus real work task situations: Differences in sub‐facets of tasks and interaction performance

AbstractTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on users' interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty‐two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre‐ and a post‐search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that simulated and real work task situations are significantly different in some sub‐facets of task, but not all sub‐facets examined in this study. However, the influence of the two tasks on users' interactive information search behavior and interaction performance is not significant, though the sub‐facets of the real and simulated work task situation are correlated to interaction performance differently. Therefore, simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub‐facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.