User assessment of search task difficulty: Relationships between reasons and ratings

Abstract

There is an increasing research interest in search task difficulty. A recent study developed a user-perceived task difficulty reason scheme including 21 reasons (two of which arise prior to searching). Interesting questions arose as to whether the many difficulty reasons in the scheme were correlated or whether the difficulty reasons were related to task difficulty levels. Using a dataset collected from 48 participants searching on four tasks, this study examined these questions while also validating the task difficulty reason scheme. Results suggest that the 19 difficulty reasons elicited after a search task are rarely correlated, and are, instead, independent. Several difficulty reasons, such as “task requirements too specific” and “low topic knowledge,” contribute significantly to difficulty levels. The large discrepancy in the frequency of some reasons' appearance in high, mid, and low difficulty rating groups may be explained by task type features. These findings help develop an in-depth explanation of search task difficulty and contribute to task-related information retrieval research and system design.