Interactive Information Retrieval Systems Research Articles

The effects of work task difficulty on health information searching behaviour

Information needs and information seeking processes depend on contextual factors. Identifying and paying attention to contextual factors such as work tasks can be helpful for personalization of information retrieval. This study aimed to determine the relationship between work task difficulty and interactive health information searching behaviour (HISB). The study was performed through an analysis of transaction logs to identify relation between the work task difficulty and HISB of postgraduate student health ambassadors (PSHB). Thirty participants were purposefully sampled and invited to complete four simulated work tasks. User perception of work task difficulty, satisfaction, and success of search process were measured using questionnaires. A total of 120 log files were analysed. The results showed that with increasing tasks difficulty, PSHB see more results pages, review more information items and finally save more documents. Also, with increasing difficulty, PSHB enter more and longer queries in information retrieval systems and, in addition to changing the query more, they use more keywords and longer queries to search. As the task difficulty increases, the rate of satisfaction and success, as well as the navigational speed decrease. Understanding how work task attributes affect HISB can be effective for designers of interactive information retrieval systems, and developers of personalized health information retrieval systems and recommendation systems based on contextual information.

Eye Gaze Relevance Feedback Indicators for Information Retrieval

There is a growing interest in the research on interactive information retrieval, particularly in the study of eye gaze-enhanced interaction. Feedback generated from user gaze features is important for developing an interactive information retrieval system. Generating these gaze features have become less difficult with the advancement of the eye tracker system over the years. In this work, eye movement as a source of relevant feedback was examined. A controlled user experiment was carried out and a set of documents were given to users to read before an eye tracker and rate the documents according to how relevant they are to a given task. Gaze features such as fixation duration, fixation count and heat maps were captured. The result showed a medium linear relationship between fixation count and user explicit ratings. Further analysis was carried out and three classifiers were compared in terms of predicting document relevance based on gaze features. It was found that the J48 decision tree classifier produced the highest accuracy.

Intellectual structure of information science 2011–2020: an author co-citation analysis

PurposeThis study continues a long history of author co-citation analysis of the intellectual structure of information science into the time period of 2011–2020. It also examines changes in this structure from 2006–2010 through 2011–2015 to 2016–2020. Results will contribute to a better understanding of the information science research field.Design/methodology/approachThe well-established procedures and techniques for author co-citation analysis were followed. Full records of research articles in core information science journals published during 2011–2020 were retrieved and downloaded from the Web of Science database. About 150 most highly cited authors in each of the two five-year time periods were selected from this dataset to represent this field, and their co-citation counts were calculated. Each co-citation matrix was input into SPSS for factor analysis, and results were visualized in Pajek. Factors were interpreted as specialties and labeled upon an examination of articles written by authors who load primarily on each factor.FindingsThe two-camp structure of information science continued to be present clearly. Bibliometric indicators for research evaluation dominated the Knowledge Domain Analysis camp during both fivr-year time periods, whereas interactive information retrieval (IR) dominated the IR camp during 2011–2015 but shared dominance with information behavior during 2016–2020. Bridging between the two camps became increasingly weaker and was only provided by the scholarly communication specialty during 2016–2020. The IR systems specialty drifted further away from the IR camp. The information behavior specialty experienced a deep slump during 2011–2020 in its evolution process. Altmetrics grew to dominate the Webometrics specialty and brought it to a sharp increase during 2016–2020.Originality/valueAuthor co-citation analysis (ACA) is effective in revealing intellectual structures of research fields. Most related studies used term-based methods to identify individual research topics but did not examine the interrelationships between these topics or the overall structure of the field. The few studies that did discuss the overall structure paid little attention to the effect of changes to the source journals on the results. The present study does not have these problems and continues the long history of benchmark contributions to a better understanding of the information science field using ACA.

A comparison of unimodal and multimodal models for implicit detection of relevance in interactive IR

Implicit detection of relevance has been approached by many during the last decade. From the use of individual measures to the use of multiple features from different sources (multimodality), studies have shown the feasibility to automatically detect whether a document is relevant. Despite promising results, it is not clear yet to what extent multimodality constitutes an effective approach compared to unimodality. In this article, we hypothesize that it is possible to build unimodal models capable of outperforming multimodal models in the detection of perceived relevance. To test this hypothesis, we conducted three experiments to compare unimodal and multimodal classification models built using a combination of 24 features. Our classification experiments showed that a univariate unimodal model based on the left‐click feature supports our hypothesis. On the other hand, our prediction experiment suggests that multimodality slightly improves early classification compared to the best unimodal models. Based on our results, we argue that the feasibility for practical applications of state‐of‐the‐art multimodal approaches may be strongly constrained by technology, cultural, ethical, and legal aspects, in which case unimodality may offer a better alternative today for supporting relevance detection in interactive information retrieval systems.

VisIRR

In this article, we present an interactive visual information retrieval and recommendation system, called VisIRR, for large-scale document discovery. VisIRR effectively combines the paradigms of (1) a passive pull through query processes for retrieval and (2) an active push that recommends items of potential interest to users based on their preferences. Equipped with an efficient dynamic query interface against a large-scale corpus, VisIRR organizes the retrieved documents into high-level topics and visualizes them in a 2D space, representing the relationships among the topics along with their keyword summary. In addition, based on interactive personalized preference feedback with regard to documents, VisIRR provides document recommendations from the entire corpus, which are beyond the retrieved sets. Such recommended documents are visualized in the same space as the retrieved documents, so that users can seamlessly analyze both existing and newly recommended ones. This article presents novel computational methods, which make these integrated representations and fast interactions possible for a large-scale document corpus. We illustrate how the system works by providing detailed usage scenarios. Additionally, we present preliminary user study results for evaluating the effectiveness of the system.

Evaluating interactive bibliographic information retrieval systems: A user‐centered approach

ABSTRACTAs an integral part of research activities, scholars frequently interact with bibliographic information retrieval systems to acquire scholarly information. With the dramatic increase of scientific literature, there is the pressing need of building effective and efficient bibliographic information retrieval systems that support more granular and complex bibliographic information needs. Investigating users' interaction with these systems plays an important role in understanding the systems' features and usability issues, which contributes to better system design and implementation. In this paper, we evaluate three interactive bibliographic information retrieval systems proposed in our previous studies using a user‐centered approach, including form‐, natural language‐, and visual graph‐based systems. This study recruits 20 participants to evaluate the three systems from the aspects of success rate, search time, query size, usefulness, ease of use, ease of learning, and satisfaction. Results showed that: the form‐based system needed the least time to formulate simple bibliographic queries and allowed the participants to make the least mistakes; the natural language‐based system needed the least time to formulate high‐complexity bibliographic queries and was rated as the most useful, easy‐to‐use, and easy‐to‐learn system; and the visual graph‐based system's strengths lie in the support of complex queries as shown that it got better ratings of usefulness and satisfaction by the high‐complexity task group.

Combining heterogeneous sources in an interactive multimedia content retrieval model

Interactive multimodal information retrieval systems (IMIR) increase the capabilities of traditional search systems, by adding the ability to retrieve information of different types (modes) and from different sources. This article describes a formal model for interactive multimodal information retrieval. This model includes formal and widespread definitions of each component of an IMIR system. A use case that focuses on information retrieval regarding sports validates the model, by developing a prototype that implements a subset of the features of the model. Adaptive techniques applied to the retrieval functionality of IMIR systems have been defined by analysing past interactions using decision trees, neural networks, and clustering techniques. This model includes a strategy for selecting sources and combining the results obtained from every source. After modifying the strategy of the prototype for selecting sources, the system is re-evaluated using classification techniques. This evaluation compares the normalised discounted cumulative gain (NDCG) measure obtained using two different approaches: the multimodal system using a baseline strategy based on predefined rules as a source selection strategy, and the same multimodal system with the functionality adapted by past user interactions. In the adapted system, a final value of 81,54% was obtained for the NDCG.

On interactive learning-to-rank for IR: Overview, recent advances, challenges, and directions

With the amount and variety of information available on digital repositories, answering complex user needs and personalizing information access became a hard task. Putting the user in the retrieval loop has emerged as a reasonable alternative to enhance search effectiveness and consequently the user experience. Due to the great advances on machine learning techniques, optimizing search engines according to user preferences has attracted great attention from the research and industry communities. Interactively learning-to-rank has greatly evolved over the last decade but it still faces great theoretical and practical obstacles. This paper describes basic concepts and reviews state-of-the-art methods on the several research fields that complementarily support the creation of interactive information retrieval (IIR) systems. By revisiting ground concepts and gathering recent advances, this article also intends to foster new research activities on IIR by highlighting great challenges and promising directions. The aggregated knowledge provided here is intended to work as a comprehensive introduction to those interested in IIR development, while also providing important insights on the vast opportunities of novel research.

The use of a graph‐based system to improve bibliographic information retrieval: System design, implementation, and evaluation

In this article, we propose a graph‐based interactive bibliographic information retrieval system—GIBIR. GIBIR provides an effective way to retrieve bibliographic information. The system represents bibliographic information as networks and provides a form‐based query interface. Users can develop their queries interactively by referencing the system‐generated graph queries. Complex queries such as “papers on information retrieval, which were cited by John's papers that had been presented in SIGIR” can be effectively answered by the system. We evaluate the proposed system by developing another relational database‐based bibliographic information retrieval system with the same interface and functions. Experiment results show that the proposed system executes the same queries much faster than the relational database‐based system, and on average, our system reduced the execution time by 72% (for 3‐node query), 89% (for 4‐node query), and 99% (for 5‐node query).

An appointment with Dr. google: Online searching for health information in times of stress

An appointment with Dr. google: Online searching for health information in times of stress

A systematic review of interactive information retrieval evaluation studies, 1967–2006

With the increasing number and diversity of search tools available, interest in the evaluation of search systems, particularly from a user perspective, has grown among researchers. More researchers are designing and evaluating interactive information retrieval (IIR) systems and beginning to innovate in evaluation methods. Maturation of a research specialty relies on the ability to replicate research, provide standards for measurement and analysis, and understand past endeavors. This article presents a historical overview of 40 years ofIIRevaluation studies using the method of systematic review. A total of 2,791 journal and conference units were manually examined and 127 articles were selected for analysis in this study, based on predefined inclusion and exclusion criteria. These articles were systematically coded using features such as author, publication date, sources and references, and properties of the research method used in the articles, such as number of subjects, tasks, corpora, and measures. Results include data describing the growth ofIIRstudies over time, the most frequently occurring and cited authors and sources, and the most common types of corpora and measures used. An additional product of this research is a bibliography ofIIRevaluation research that can be used by students, teachers, and those new to the area. To the authors' knowledge, this is the first historical, systematic characterization of theIIRevaluation literature, including the documentation of methods and measures used by researchers in this specialty.

Interactive retrieval using simulated versus real work task situations: Differences in sub‐facets of tasks and interaction performance

AbstractTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on users' interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty‐two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre‐ and a post‐search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that simulated and real work task situations are significantly different in some sub‐facets of task, but not all sub‐facets examined in this study. However, the influence of the two tasks on users' interactive information search behavior and interaction performance is not significant, though the sub‐facets of the real and simulated work task situation are correlated to interaction performance differently. Therefore, simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub‐facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users' information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.

Report on the SIGIR 2010 workshop on the simulation of interaction

All search in the real-world is inherently interactive. Information retrieval (IR) has a firm tradition of using simulation to evaluate IR systems as embodied by the Cranfield paradigm. However, to a large extent, such system evaluations ignore user interaction. Simulations provide a way to go beyond this limitation. With an increasing number of researchers using simulation to evaluate interactive IR systems, it is now timely to discuss, develop and advance this powerful methodology within the field of IR. During the SimInt 2010 workshop around 40 participants discussed and presented their views on the simulation of interaction. The main conclusion and general consensus was that simulation offers great potential for the field of IR; and that simulations of user interaction can make explicit the user and the user interface while maintaining the advantages of the Cranfield paradigm.

New measures for the evaluation of interactive information retrieval systems: Normalized task completion time and normalized user effectiveness

User satisfaction, though difficult to measure, is the main goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, user effectiveness also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and user effectiveness. In this paper, we propose two new measures to evaluate IIR systems, the Normalized Task Completion Time (NT) and the Normalized User Effectiveness (NUE). The two measures overcome limitations of existing measures and are efficient to calculate in that they do not need a large pool of search tasks. A user study was conducted to investigate the relationships between the two measures and the user satisfaction and effectiveness of a given IR system. The learning effects described by NT, NUE, and the task completion time were also studied and compared. The results show that NT is strongly correlated with user satisfaction, NUE is a better indicator of system effectiveness than task completion time, and both new measures are superior to task completion time in describing the learning effect of the given IR system.

Evaluating an integrated system supporting multiple information‐seeking strategies

AbstractMany studies have demonstrated that people engage in a variety of different information behaviors when engaging in information seeking. However, standard information retrieval systems such as Web search engines continue to be designed to support mainly one such behavior, specified searching. This situation has led to suggestions that people would be better served by information retrieval systems which support different kinds of information‐seeking strategies. This article reports on an experiment comparing the retrieval effectiveness of an integrated interactive information retrieval (IIR) system which adapts to support different information‐seeking strategies with that of a standard baseline IIR system. The experiment, with 32 participants each searching on eight different topics, indicates that using the integrated IIR system resulted in significantly better user satisfaction with search results, significantly more effective interaction, and significantly better usability than that using the baseline system.

Distribution of cognitive load in Web search

AbstractThe search task and the system both affect the demand on cognitive resources during information search. In some situations the demands may become too high for a person. This article has a three‐fold goal. First, it presents and critiques methods to measure cognitive load. Second, it explores the distribution of load across search task stages. Finally, it seeks to improve our understanding of factors affecting cognitive load levels in information search. To this end, a controlled Web search experiment with 48 participants was conducted. Interaction logs were used to segment search tasks semiautomatically into task stages. Cognitive load was assessed using a new variant of the dual‐task method. Average cognitive load was found to vary by search task stages. It was significantly higher during query formulation and user description of a relevant document as compared to examining search results and viewing individual documents. Semantic information shown next to the search results lists in one of the studied interfaces was found to decrease mental demands during query formulation and examination of the search results list. These findings demonstrate that changes in dynamic cognitive load can be detected within search tasks. Dynamic assessment of cognitive load is of core interest to information science because it enriches our understanding of cognitive demands imposed on people engaged in the search process by a task and the interactive information retrieval system employed.

Investigating information retrieval support techniques for different information‐seeking strategies

AbstractWe report on a study that investigated the efficacy of four different interactive information retrieval (IIR) systems, each designed to support a specific information‐seeking strategy (ISS). These systems were constructed using different combinations of IR techniques (i.e., combinations of different methods of representation, comparison, presentation and navigation), each of which was hypothesized to be well suited to support a specific ISS. We compared the performance of searchers in each such system, designated “experimental,” to an appropriate “baseline” system, which implemented the standard specified query and results list model of current state‐of‐the‐art experimental and operational IR systems. Four within‐subjects experiments were conducted for the purpose of this comparison. Results showed that each of the experimental systems was superior to its baseline system in supporting user performance for the specific ISS (that is, the information problem leading to that ISS) for which the system was designed. These results indicate that an IIR system, which intends to support more than one kind of ISS, should be designed within a framework which allows the use and combination of different IR support techniques for different ISSs.

Evaluating a metadata‐based term suggestion interface for PubMed with real users with real requests

AbstractThis paper reports results of an evaluation study of MAP (Multi‐faceted Access to PubMed), a metadata induced query suggestion interface for PubMed bibliographic search.A novel evaluation methodology was used to address the challenges involved in evaluating an IIR (Interactive Information Retrieval) system such as the MAP interface. The most significant aspect of this methodology is that, instead of using assigned tasks common in traditional IR evaluation, it asks real users with real search requests to search with real systems in an experimental setting. Several performance measures were created based on which comparisons were made between MAP and PubMed baseline. MAP was shown to perform better in several of these measures, especially when the search requests had not been attempted before.The finding pointed to search characteristics as an important intervening variable in IIR evaluation. The advantages of and potential threats to our methodology were also discussed.

CreChainDo: an iterative and interactive Web information retrieval system based on lattices

This paper presents an iterative and interactive information retrieval (IR) system for Web search using formal concept analysis (FCA). FCA provides a natural way to organise objects according to their properties and it has been used in recent work to organise search engine results. The navigation over the lattice helps the user to explore a structured and synthetic result. Such a lattice contains concepts that are relevant and others that are not relevant regarding a given IR task. In this way, lattices are introduced in an interactive and iterative system. The user expresses his negative or positive agreement with some concept of the lattice in respect of his objective of IR. These user choices are converted into operations over the lattice. The lattice is dynamically updated for a better fit to the request.