Visual topic discovering, tracking and summarization from social media streams

Abstract

Nowadays, microblogging has become popular, with hundreds of millions of short messages being posted and shared every minute on a variety of topics in social media such as Facebook, Twitter and Weibo. Many of such messages contain videos that captured particular events or moments in people's life. In this work, we seek to automatically identify the video topics posted in the social media streams on Weibo. While Topic Detection and Tracking (TDT) task has been extensively studied in multimedia retrieval, automatically discovering, tracking and summarizing video topics from social media streams is still challenging due to short and noisy content, diverse and fast changing topics, and large data volume. In this paper, we propose a K-partite graph based approach to address these challenges. We introduce a K-partite graph representation to simultaneously model the relationships among videos contained in the Weibo streams, their textural features and visual features. We propose a novel joint clustering algorithm to capture global structure of the K-partite graph in a "relation cluster network" (RCN) where latent, meta-nodes are added to the network to represent video clusters. Based on this network we propose methods for tracking and summarizing the videos in streams through fusing various types of features and multiple ranking schemes. The experiment results based on a real dataset show the effectiveness of our method with significant improvement over baseline.