Exploiting Informative Video Segments for Temporal Action Localization

Abstract

We propose a novel method of exploiting informative video segments by learning segment weights for temporal action localization in untrimmed videos. Informative video segments represent the intrinsic motion and appearance of an action, and thus contribute crucially to action localization. The learned segment weights represent the informativeness of video segments to recognizing actions and help infer the boundaries required to temporally localize actions. We build a supervised temporal attention network (STAN) that includes a supervised segment-level attention module to dynamically learn the weights of video segments, and a feature-level attention module to effectively fuse multiple features of segments. Through the cascade of the attention modules, STAN exploits informative video segments and generates descriptive and discriminative video representations. We use a proposal generator and a classifier to estimate the boundaries of actions and classify the classes of actions. Extensive experiments are conducted on two public benchmarks: THUMOS2014 and ActivityNet1.3. The results demonstrate that our proposed method achieves competitive performance compared with the state-of-the-art methods. Moreover, compared with the baseline method that equally treats video segments, STAN achieves significant improvements with the mAP increased from 30.4% to 39.8% on the THUMOS2014 dataset and from 31.4% to 35.9% on the ActivityNet1.3 dataset, demonstrating the effectiveness of learning informative video segments for temporal action localization.