Human computation: Image metadata acquisition based on a single-player annotation game

Abstract

Effective acquisition of descriptive semantics for images is still an open issue today. Crowd-based human computation represents a family of approaches able to provide large scale metadata with decent quality. Within this field, games with a purpose (GWAP) have become increasingly important, as they have the potential to motivate contributors to the process through entertainment. However, the existing solutions are weak, when specific metadata are needed. In this work, we present a game with a purpose called PexAce, which utilizes manpower to collect tags characterizing a set of given images. Using novel game mechanics, the game is a single-player, less prone to cold-start problems and suitable for deployment in the domain of personal imagery. As our experiments show, the game delivers tags that characterize images with high precision (using a posteriori expert evaluation and evaluation against the gold standard: the extended Corel 5k dataset). We also employ the game in the domain of personal images, where very specific metadata are needed for their proper organization (person names, places, events) and show, that the game is able to collect even these kinds of metadata. We show that the key to higher quality metadata lies in combining the fun factor of the game with motivation for personal gain.