User preference-aware video highlight detection via deep reinforcement learning

Abstract

Video highlight detection is a technique to retrieval short video clips that capture a user’s primary attention or interest within an unedited video. There exists a substantial interest in automatizing highlight detection to facilitate efficient video browsing. Recent research often focuses on objectively finding frames that are visual representative as well as diversity to form highlights. However, the user preferences are relatively subjective and may vary from person to person. Therefore, it is not trivial to find different highlights over a same video for different users. This paper describes a reinforcement learning-based framework that detects different highlights according to different user’s preferences. Under this framework, a novel reward function that accounts for relevance of user preference to candidate highlights is introduced. During training, the framework strives for earning higher rewards by learning to detect more diverse and more preference-aware highlights. The effectiveness of the proposed method is illustrated by applying it to different types of real world movies, and show it achieves state-of-the-art results.