Play and rewind: Context-aware video temporal action proposals

Abstract

In this paper, we investigate the problem of Temporal Action Proposal (TAP) generation, which plays a fundamental role in large-scale untrimmed video analysis but remains largely unsolved. Most of the prior works proposed the temporal actions by predicting the temporal boundaries or actionness scores of video units. Nevertheless, context information among surrounding video units has not been adequately explored, which may result in severe loss of information. In this work, we propose a context-aware temporal action proposal network which makes full use of the contextual information in two aspects: 1) To generate initial proposals, we design a Bi-directional Parallel LSTMs to extract the visual features of a video unit by considering its contextual information. Therefore, the prediction of temporal boundaries and actionness scores will be more accurate because it knows what happened in the past and what will happen in the future; and 2) To refine the initial proposals, we design an action-attention based re-ranking network which considers both surrounding proposal and initial actionness scores to assign true action proposals with high confidence scores. Extensive experiments are conducted on two challenging datasets for both temporal action proposal generation and detection tasks, demonstrating the effectiveness of the proposed approach. In particular, on THUMOS’14 dataset, our method significantly surpasses state-of-the-art methods by 7.73% on AR@50. Our code is released at: https://github.com/Rheelt/TAPG.