Visual storylines: Semantic visualization of movie sequence

Abstract

This paper presents a video summarization approach that automatically extracts and visualizes movie storylines in a static image for the purposes of efficient representation and quick overview. A new type of video visualization, Visual Storylines, is designed to summarize video storylines in an image composition while preserving the style of the original videos. This is achieved with a series of video analysis, image synthesis, relationship quantification and geometric layout optimization techniques. Specifically, we analyze the video contents and quantify the video story unit relationships automatically through clustering video shots according to both the visual and audio data. A multi-level storyline visualization method then organizes and synthesizes a suitable amount of representative information, including both the locations and interested objects and characters, with the assistance of arrows, according to the relationships between the video story units and the temporal structure of the video sequence. Several results have demonstrated that our approach is able to abstract the main storylines of professionally edited video such as commercial movies and TV series, though some semantic key clues might be missed in the summarization. Preliminary user studies have been performed to evaluate our approach, and the results show that our approach can be used to assist viewers to understand video contents when they are familiar with the context of the video or when a text synopsis is provided.