Interactive Information Retrieval Evaluation Research Articles

Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluation

Re-using research resources is essential for advancing knowledge and developing repeatable, empirically solid experiments in scientific fields, including interactive information retrieval (IIR). Despite recent efforts on standardizing research re-use and documentation, how to quantitatively measure the reusability of IIR resources still remains an open challenge. Inspired by the reusability evaluations on Cranfield experiments, our work proactively explores the problem of measuring IIR test collection reusability and makes threefold contributions: (1) constructing a novel usefulness-oriented framework with specific analytical methods for evaluating the reusability of IIR test collections consisting of query sets, document/page sets, and sets of task-document usefulness (tuse); (2) explaining the potential impacts of varying IIR-specific factors (e.g. search tasks, sessions, user characteristics) on test collection reusability; (3) proposing actionable methods for building reusable test collections in IIR and thereby amortizing the true cost of user-oriented evaluations. The Cranfield-inspired reusability assessment framework serves as an initial step towards accurately evaluating the reusability of IIR research resources and measuring the reproducibility of IIR evaluation results. It also demonstrates an innovative approach to integrating the insights from individual heterogeneous user studies with the evaluation techniques developed in standardized ad hoc retrieval experiments, which will facilitate the maturation of IIR fields and eventually benefits both sides of research.

Professor Pia Borlund

UKeiG would like to inform you that the 5th Tony Kent Strix Annual Memorial Lecture 2019 is to be delivered by Professor Pia Borlund, Department of Archivistics, Library and Information Science at Oslo Metropolitan University and will take place on the afternoon of Friday, 29th November 2019 at The Geological Society, Burlington House, Piccadilly, London. Professor Borlund was nominated for the award by Diane H. Sonnenwald, Emerita Professor of Library & Information Studies at University College Dublin and received unanimous praise and support from the Strix judging panel. “Similar to the memorable Dr Tony Kent, the work by Borlund is original and innovative, and has had a significant impact on information retrieval (IR) research and applications. Her approaches are analytical and practical, and her devotion and dedication to users and interactive information retrieval (IIR) are outstanding. Within the IR community, Borlund is known for her innovative contributions to IR user studies, evaluations and test design, including strong, novel methodological contributions to IIR evaluation. In particular, she is recognised for the development of her IIR evaluation model which uniquely employs simulated work task situations. Borlund developed the IIR evaluation model as a doctoral student by taking up the challenges of the calls put forward by Professor Stephen Robertson (recipient of the first Tony Kent Strix Award in 1998) and Micheline Hancock-Beaulieu, in their iconic 1992 paper on IR systems evaluation.
 Keywords: Information Retrieval, Search, Tony Kent Strix, Awards

Order effect in interactive information retrieval evaluation: an empirical study

Purpose – The purpose of this paper is to report a study of order effect in interactive information retrieval (IIR) studies. The phenomenon of order effect is well-known, and it is the main reason why searches are permuted (counter-balanced) between test participants in IIR studies. However, the phenomenon is not yet fully understood or investigated in relation to IIR; hence the objective is to increase the knowledge of this phenomenon in the context of IIR as it has implications for test design of IIR studies. Design/methodology/approach – Order effect is studied via partly a literature review and partly an empirical IIR study. The empirical IIR study is designed as a classic between-groups design. The IIR search behaviour was logged and complementary post-search interviews were conducted. Findings – The order effect between groups and within search tasks were measured against nine classic IIR performance parameters of search interaction behaviour. Order effect is seen with respect to three performance parameters (website changes, visit of webpages, and formulation of queries) shown by an increase in activity on the last performed search. Further the theories with respect to motivation, fatigue, and the good-subject effect shed light on how and why order effect may affect test participants’ IR system interaction and search behaviour. Research limitations/implications – Insight about order effect has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Due to the limited sample of 20 test participants (Library and Information Science (LIS) students) inference statistics is not applicable; hence conclusions can be drawn from this sample of test participants only. Originality/value – Only few studies in LIS focus on order effect and none from the perspective of IIR.

Interactive Information Retrieval: An Introduction

The paper introduces the research area of interactive information retrieval (IIR) from a historical point of view. Further, the focus here is on evaluation, because much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction and satisfaction with retrieved information. In order to position IIR evaluation, the Cranfield model and the series of tests that led to the Cranfield model are out lined. Three iconic user-oriented studies and projects that all have contributed to how IIR is perceived and under stood today are presented: The MEDLARS test, the Book House fiction retrieval system, and the OKAPI project. On this basis the call for alternative IIR evaluation approaches motivated by the three revolutions (the cognitive, the relevance, and the interactive revolutions) put forward by Robertson & Hancock-Beaulieu (1992) is presented. As a response to this call the ‘IIR evaluation model’ by Borlund (e.g., 2003a) is introduced. The objective of the IIR evaluation model is to facilitate IIR evaluation as close as possible to actual information searching and IR process es, though still in a relatively controlled evaluation environment, in which the test instrument of a simulated work task situation plays a central part.

A systematic review of interactive information retrieval evaluation studies, 1967–2006

With the increasing number and diversity of search tools available, interest in the evaluation of search systems, particularly from a user perspective, has grown among researchers. More researchers are designing and evaluating interactive information retrieval (IIR) systems and beginning to innovate in evaluation methods. Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors. This article presents a historical overview of 40 years ofIIRevaluation studies using the method of systematic review. A total of 2,791 journal and conference units were manually examined and 127 articles were selected for analysis in this study, based on predefined inclusion and exclusion criteria. These articles were systematically coded using features such as author, publication date, sources and references, and properties of the research method used in the articles, such as number of subjects, tasks, corpora, and measures. Results include data describing the growth ofIIRstudies over time, the most frequently occurring and cited authors and sources, and the most common types of corpora and measures used. An additional product of this research is a bibliography ofIIRevaluation research that can be used by students, teachers, and those new to the area. To the authors' knowledge, this is the first historical, systematic characterization of theIIRevaluation literature, including the documentation of methods and measures used by researchers in this specialty.

Interactive retrieval using simulated versus real work task situations: Differences in sub‐facets of tasks and interaction performance

AbstractTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on users' interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty‐two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre‐ and a post‐search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that simulated and real work task situations are significantly different in some sub‐facets of task, but not all sub‐facets examined in this study. However, the influence of the two tasks on users' interactive information search behavior and interaction performance is not significant, though the sub‐facets of the real and simulated work task situation are correlated to interaction performance differently. Therefore, simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub‐facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users' information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

Evaluating advanced search interfaces using established information‐seeking models

AbstractWhen users have poorly defined or complex goals, search interfaces that offer only keyword‐searching facilities provide inadequate support to help them reach their information‐seeking objectives. The emergence of interfaces with more advanced capabilities, such as faceted browsing and result clustering, can go some way toward addressing such problems. The evaluation of these interfaces, however, is challenging because they generally offer diverse and versatile search environments that introduce overwhelming amounts of independent variables to user studies; choosing the interface object as the only independent variable in a study would reveal very little about why one design outperforms another. Nonetheless, if we could effectively compare these interfaces, then we would have a way to determine which was best for a given scenario and begin to learn why. In this article, we present a formative inspection framework for the evaluation of advanced search interfaces through the quantification of the strengths and weaknesses of the interfaces in supporting user tactics and varying user conditions. This framework combines established models of users and their needs and behaviors to achieve this. The framework is applied to evaluate three search interfaces and demonstrates the potential value of this approach to interactive information retrieval evaluation.

Evaluation Challenges and Directions for Information-Seeking Support Systems

ISSSs provide an exciting opportunity to extend previous information-seeking and interactive information retrieval evaluation models and create a research community that embraces diverse methods and broader participation.

Evaluating a metadata‐based term suggestion interface for PubMed with real users with real requests

AbstractThis paper reports results of an evaluation study of MAP (Multi‐faceted Access to PubMed), a metadata induced query suggestion interface for PubMed bibliographic search.A novel evaluation methodology was used to address the challenges involved in evaluating an IIR (Interactive Information Retrieval) system such as the MAP interface. The most significant aspect of this methodology is that, instead of using assigned tasks common in traditional IR evaluation, it asks real users with real search requests to search with real systems in an experimental setting. Several performance measures were created based on which comparisons were made between MAP and PubMed baseline. MAP was shown to perform better in several of these measures, especially when the search requests had not been attempted before.The finding pointed to search characteristics as an important intervening variable in IIR evaluation. The advantages of and potential threats to our methodology were also discussed.