Modeling long-term video semantic distribution for temporal action proposal generation

Abstract

Video temporal segmentation plays a vital role in video analysis since many higher-level computer vision tasks rely on it. Some recent efforts have been dedicated to generating temporal action proposals for long and untrimmed videos, which requires methods to generate accurate boundaries for video semantics. In this paper, we propose a novel and efficient Temporal Distribution Network (TDN), to model the long-term distribution of video semantic units (video dictionary). Firstly, we encode the semantics and context relations of video segments with a boundary-specified video embedding method. Then based on temporal convolutional layers, we design a Temporal Distribution Network (TDN) enumerating all the possible temporal locations in one pass and generating proposals that have high action confidence scores by capturing the long-term distributions of video semantics. We validate our method on temporal action proposal generation tasks and action detection tasks. Experimental results on two benchmark datasets, THUMOS14 and ActivityNet-1.3, show that the proposed method can significantly outperform the state-of-the-art approaches. Our model could obtain high-quality action proposals with a much faster speed.