Abstract

This paper reports on the empirical findings of the meta-evaluation of the application of work tasks in connection with the evaluation of interactive information retrieval (IIR) systems. The purpose of the meta-evaluation is to uncover if it is recommendable to apply work tasks to future evaluation of IIR systems. It is investigated if any search behavioural differences exist between test persons' treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exists one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. We are also interested in learning what defines a 'good' work task situation. The empirical results of the meta-evaluation provide positive evidence of the application of simulated work task situations in connection with evaluation of IIR systems. The results also point to that tuning of work task situations towards the group of test persons is of importance due to motivation of the test persons. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons' search treatment. Finally, it is verified that there exists a general pattern of assessment behaviour for the test persons. This verification makes yet another experimental reason for permutation of work tasks between test persons, in order to avoid bias of the retrieval results.

Highlights

  • The concept of work task is widely applied within the field of Information Science, for instance in relation to systems design and development [e.g. 1-4], studies of user behaviour and task complexities [e.g. 5-7], as well as at a theoretical level when modelling information retrieval (IR) use and interaction [e.g. 8]

  • The test persons comments reveal three characteristics of what makes a good simulated work task situation: (1) the situation has to be of a nature to which the test persons can relate and in which they can identify themselves; (2) the topic of the situation has to be of interest to the group of test persons; and (3) the situation has to provide enough imaginative context in order for the test persons to be able to apply the situation

  • This paper reports on the empirical results of the evaluation of the application of work tasks in connection with the evaluation of information retrieval (IIR) systems

Read more

Summary

Introduction

The concept of work task is widely applied within the field of Information Science, for instance in relation to systems design and development [e.g. 1-4], studies of user behaviour and task complexities [e.g. 5-7], as well as at a theoretical level when modelling information retrieval (IR) use and interaction [e.g. 8]. The concept of work task has been introduced as a core component in relation to the evaluation of interactive information retrieval (IIR) systems [e.g. 9-12]. The demand can be explained by the fact that IR systems have become more interactive, and cannot be evaluated without including the interactive seeking and retrieval processes [13]. This is so because IIR systems are systems where the user dynamically conducts searching tasks and correspondingly reacts to systems responses over session time. The foci of IIR system evaluation include all the user’s activities of interaction with the retrieval and feedback mechanisms as well as the retrieval outcome itself

Methods
Results
Conclusion
Full Text
Paper version not known

Talk to us

Join us for a 30 min session where you can share your feedback and ask us any queries you have

Schedule a call

Disclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.