Abstract
Evaluation of interactive search systems and study of users’ struggling search behaviors require a significant number of search tasks. However, generation of such tasks is inherently difficult, as each task is supposed to trigger struggling search behavior rather than simple search behavior. To the best of our knowledge, there has not been a commonly used task set for research in struggling search. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for evaluation. To deal with this problem, we propose a crowd-powered task generation method and develop a platform to efficiently generate struggling search tasks on basis of online wikis such as Wikipedia. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of “repeated similar queries” and “quick-back clicks”; tasks of diverse topics, high quality and difficulty can be created using this method. For benefit of the community, we publicly released a task generation platform TaskGenie, a task set of 80 topically diverse struggling search tasks with “baselines,” and the corresponding anonymized user behavior logs.
Highlights
Struggling Search, as Hassan et al [11] introduced, describes the searching process wherein users experience difficulty in finding required information
Our findings suggest that Wikipedia is a good source for paraphrased sentences which can potentially serve in the creation of difficult search tasks across diverse topics
Comparing to the previous task generation method that requires experts or professionals of certain areas resulting in small-sized task sets [2, 23, 25], the method we proposed in this paper can generate a large number of struggling search tasks through crowdsourcing which is shown to be cost-effective
Summary
Struggling Search, as Hassan et al [11] introduced, describes the searching process wherein users experience difficulty in finding required information. Generation of struggling search tasks is essential for both the study of users’ struggling search behaviors and the performance evaluation of interactive search systems. Creating such a task generator is not easy. There has not been a unified task set for research in struggling search. This task set can be used to reliably simulate struggling search among users. We provide the basic success rate and task difficulty level that can be used in the development and evaluation of methods to support users while they struggle in search tasks. Our proposed framework can be used to generate struggling search tasks as per the topical/ domain-related needs at hand
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Free) 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a call
