Simulated Work Task Situations Research Articles

An exploration of search-as-learning in digital archives of an online museum

PurposeRecent studies on search-as-learning (SAL) have recognized the significance of identifying users' learning needs as they evolve for acquiring knowledge during the search process. In this study, the authors clarify the extent to which search behaviors reflect the learning outcome and foster the users' knowledge of Chinese art.Design/methodology/approachThe authors conducted an exploratory-sequential mixed-methods approach using simulated work task situations to collect empirical data. The authors used two types of simulated learning tasks for topics related to painting and antique knowledge. A lot of 25 users participated in this evaluation of digital archives (DAs) at the National Palace Museum (NPM) in Taiwan. For each set of topics, a close-ended task related to lower-level learning goals and an open-ended task related to higher-level learning goals.FindingsThe learning criteria reflect changes in the users' knowledge structure, revealing the SAL process. Furthermore, users achieved better task performance on the higher-level creative-learning task, which suggests that they met more learning criteria, exhibited a greater variety of search patterns when exploring the topics via interaction with various sources. Finally, there is a close relationship between creative-learning tasks, prior knowledge, keyword search actions and learning outcomes.Originality/valueThe authors discuss implications with respect to the design of DAs in practice and contributions to the body of SAL knowledge in DAs of online museums. For future reference, the authors provide implications for the development of learning measures from the perspective of user search behavior with associated learning outcomes in the context of DAs.

Professor Pia Borlund

UKeiG would like to inform you that the 5th Tony Kent Strix Annual Memorial Lecture 2019 is to be delivered by Professor Pia Borlund, Department of Archivistics, Library and Information Science at Oslo Metropolitan University and will take place on the afternoon of Friday, 29th November 2019 at The Geological Society, Burlington House, Piccadilly, London. Professor Borlund was nominated for the award by Diane H. Sonnenwald, Emerita Professor of Library & Information Studies at University College Dublin and received unanimous praise and support from the Strix judging panel. “Similar to the memorable Dr Tony Kent, the work by Borlund is original and innovative, and has had a significant impact on information retrieval (IR) research and applications. Her approaches are analytical and practical, and her devotion and dedication to users and interactive information retrieval (IIR) are outstanding. Within the IR community, Borlund is known for her innovative contributions to IR user studies, evaluations and test design, including strong, novel methodological contributions to IIR evaluation. In particular, she is recognised for the development of her IIR evaluation model which uniquely employs simulated work task situations. Borlund developed the IIR evaluation model as a doctoral student by taking up the challenges of the calls put forward by Professor Stephen Robertson (recipient of the first Tony Kent Strix Award in 1998) and Micheline Hancock-Beaulieu, in their iconic 1992 paper on IR systems evaluation.
 Keywords: Information Retrieval, Search, Tony Kent Strix, Awards

A study of the use of simulated work task situations in interactive information retrieval evaluations

Purpose – The purpose of this paper is to report a study of how the test instrument of a simulated work task situation is used in empirical evaluations of interactive information retrieval (IIR) and reported in the research literature. In particular, the author is interested to learn whether the requirements of how to employ simulated work task situations are followed, and whether these requirements call for further highlighting and refinement. Design/methodology/approach – In order to study how simulated work task situations are used, the research literature in question is identified. This is done partly via citation analysis by use of Web of Science®, and partly by systematic search of online repositories. On this basis, 67 individual publications were identified and they constitute the sample of analysis. Findings – The analysis reveals a need for clarifications of how to use simulated work task situations in IIR evaluations. In particular, with respect to the design and creation of realistic simulated work task situations. There is a lack of tailoring of the simulated work task situations to the test participants. Likewise, the requirement to include the test participants’ personal information needs is neglected. Further, there is a need to add and emphasise a requirement to depict the used simulated work task situations when reporting the IIR studies. Research limitations/implications – Insight about the use of simulated work task situations has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Originality/value – Simulated work task situations are widely used in IIR studies, and the present study is the first comprehensive study of the intended and unintended use of this test instrument since its introduction in the late 1990’s. The paper addresses the need to carefully design and tailor simulated work task situations to suit the test participants in order to obtain the intended authentic and realistic IIR under study.

Framing of different types of information needs within simulated work task situations: An empirical study in the school context

This paper reports a meta-evaluation of how to frame different types of information needs within simulated work task situations. This is done via an empirical study of teenagers and their teachers’ Internet information searching. Two sets of simulated work task situations were carefully designed to reflect verificative, conscious topical and muddled topical information needs of each group of test participants. The study shows that it is challenging to formulate verificative simulated work task situations and to incorporate curiosity in the muddled topical simulated work task situations. The results also show that the search behaviour of the two groups differs across the information needs, as expected, but also between the two groups, owing to the search strategy and attitude of the teenagers. This is seen by how fast they were at searching and assessing relevance, often using Google’s ‘picture search function’, and saving the reading in detail for later.

Interactive Information Retrieval: An Introduction

The paper introduces the research area of interactive information retrieval (IIR) from a historical point of view. Further, the focus here is on evaluation, because much research in IR deals with IR evaluation methodology due to the core research interest in IR performance, system interaction and satisfaction with retrieved information. In order to position IIR evaluation, the Cranfield model and the series of tests that led to the Cranfield model are out lined. Three iconic user-oriented studies and projects that all have contributed to how IIR is perceived and under stood today are presented: The MEDLARS test, the Book House fiction retrieval system, and the OKAPI project. On this basis the call for alternative IIR evaluation approaches motivated by the three revolutions (the cognitive, the relevance, and the interactive revolutions) put forward by Robertson & Hancock-Beaulieu (1992) is presented. As a response to this call the ‘IIR evaluation model’ by Borlund (e.g., 2003a) is introduced. The objective of the IIR evaluation model is to facilitate IIR evaluation as close as possible to actual information searching and IR process es, though still in a relatively controlled evaluation environment, in which the test instrument of a simulated work task situation plays a central part.

Interactive retrieval using simulated versus real work task situations: Differences in sub‐facets of tasks and interaction performance

AbstractTask design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on users' interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty‐two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre‐ and a post‐search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that simulated and real work task situations are significantly different in some sub‐facets of task, but not all sub‐facets examined in this study. However, the influence of the two tasks on users' interactive information search behavior and interaction performance is not significant, though the sub‐facets of the real and simulated work task situation are correlated to interaction performance differently. Therefore, simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub‐facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.

An exploration of the relationships between work tasks and users' interaction performance

This paper reports a study on the relationships between work tasks and users' performance when they interact with information systems, that is, interaction performance. An experiment with 24 participants was conducted to explore the issue. The participants were asked to search for useful information to support the completion of six simulated work task situations from operational information systems. Users' interaction performance is measured based on users' search effectiveness and efficiency. The results indicate: 1) Work tasks and time sufficiency for the search dramatically affect users' interaction performance. 2) Information sufficiency affects users' perception of their interaction performance. 3) The two facet of work tasks, i.e. product and objective work task complexity, significantly and interactively affect users' interaction performance to different degrees. 4) In contrast to product, objective work task complexity is more influential on users' interaction performance. 5) In addition, the study identifies some indicators of work task characteristics, for example, the number of items users select to support their work tasks and the search time they spend in locating useful information for a work task. The results shed light on the role of work task in information search and retrieval. The findings help improve interactive information systems design and personalization of information retrieval.

Experimental components for the evaluation of interactive information retrieval systems

This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental setting consists of three components: (1) the involvement of potential users as test persons; (2) the application of dynamic and individual information needs; and (3) the use of multidimensional and dynamic relevance judgements. Hidden under the information need component is the essential central sub‐component, the simulated work task situation, the tool that triggers the (simulated) dynamic information needs. This paper also reports on the empirical findings of the metaevaluation of the application of this sub‐component, the purpose of which is to discover whether the application of simulated work task situations to future evaluation of IIR systems can be recommended. Investigations are carried out to determine whether any search behavioural differences exist between test persons‘ treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exists one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. The empirical results of the meta‐evaluation provide positive evidence for the application of simulated work task situations to the evaluation of IIR systems. The results also indicate that tailoring work task situations to the group of test persons is important in motivating them. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons’ search treatment.

The development of a method for the evaluation of interactive information retrieval systems

The paper describes the ideas and assumptions underlying the development of a new method for the evaluation and testing of interactive information retrieval (IR) systems, and reports on the initial tests of the proposed method. The method is designed to collect different types of empirical data, i.e. cognitive data as well as traditional systems performance data. The method is based on the novel concept of a ‘simulated work task situation’ or scenario and the involvement of real end users. The method is also based on a mixture of simulated and real information needs, and involves a group of test persons as well as assessments made by individual panel members. The relevance assessments are made with reference to the concepts of topical as well as situational relevance. The method takes into account the dynamic nature of information needs which are assumed to develop over time for the same user, a variability which is presumed to be strongly connected to the processes of relevance assessment.