An extensible personal photograph collection for graded relevance assessments and user simulation

Abstract

In this paper, we present a document collection with graded relevance assessments that has been sampled from real photographers. In order to reflect both vagueness in the commonly used retrieval calculations as well as in the user's query, we argue that current document collections being based on binary relevance judgments have drawbacks -- in particular if user-centered or relevance feedback-related experiments are conducted. In addition, such system-centric collections are based on documents, which do not necessarily reflect a layperson's personal photo collection. To overcome this issue, we suggest a test set of documents that is based on a study of 19 real photographers discriminating it from Flickr downloads or the like. The collection has been categorized on basis of different criteria such as document quality or motif quality plus the aforementioned graded relevance assessments. Reflecting the photograph taking behavior of the investigated photographers, we are also providing an event-based ground-truth in addition to a topic-based one. In total, 130 different topics are available for the collection. In order to provide means to address different photographer and user types, e.g. in user simulations or in usability engineering, we make the demographic information of both photographers and assessors available. Eventually, this links interactive information retrieval evaluation with persona-based interaction design -- a factor that has been neglected in multimedia information retrieval so far.