The uncertain representation ranking framework for concept-based video retrieval

Robin Aly,Aiden Doherty,Alan F Smeaton,Djoerd Hiemstra,Franciska De Jong

doi:10.1007/s10791-012-9207-y

Abstract

Concept based video retrieval often relies on imperfect and uncertain concept detectors. We propose a general ranking framework to define effective and robust ranking functions, through explicitly addressing detector uncertainty. It can cope with multiple concept-based representations per video segment and it allows the re-use of effective text retrieval functions which are defined on similar representations. The final ranking status value is a weighted combination of two components: the expected score of the possible scores, which represents the risk-neutral choice, and the scores’ standard deviation, which represents the risk or opportunity that the score for the actual representation is higher. The framework consistently improves the search performance in the shot retrieval task and the segment retrieval task over several baselines in five TRECVid collections and two collections which use simulated detectors of varying performance.

Highlights

Concept-based video retrieval has many advantages over other content-based approaches (Snoek and Worring 2009)
We describe an adaptation of the uncertain representation ranking (URR) framework to shot retrieval in which the expected score component is equivalent to the Probabilistic Framework of Unobservable Binary (PRFUBE), which was originally proposed by Aly et al (2008)
While the framework is independent of the retrieval task, we adapted it to the tasks of retrieving shots and segments

Summary

Introduction

Concept-based video retrieval has many advantages over other content-based approaches (Snoek and Worring 2009). It is more straightforward to define ranking functions on concept-based representations than for most other content-based representations (Naphade et al 2006). The definition of a ranking function for the query ‘‘Find me tigers’’ is intuitively more straightforward based on the concept Animal in a (video-) segment than based on the color distribution in an example image. As the current state-of-the art in automatic concept detection is not mature enough for ranking functions directly using the binary concept labels occurs/absent (Hauptmann et al 2007), conceptbased search engines use the confidence score of a detector that the concept occurs. The uncertainty introduced by the use of confidence scores makes the definition of effective and robust ranking functions again more difficult. This paper presents a general framework for the definition of concept-based ranking functions for video retrieval that fulfill these requirements

Methods

Findings

Discussion

Conclusion