ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation

Khoa Vo,Sang Truong,Kashu Yamazaki,Minh-Triet Tran,Ngan Le,Akihiro Sugimoto

doi:10.1109/access.2021.3110973

Abstract

Temporal action proposal generation (TAPG) aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet plays an important role in many tasks of video analysis and understanding. Despite the great achievement in TAPG, most existing works ignore the human perception of interaction between agents and the surrounding environment by applying a deep learning model as a black-box to the untrimmed videos to extract video visual representation. Therefore, it is beneficial and potentially improve the performance of TAPG if we can capture these interactions between agents and the environment. In this paper, we propose a novel framework named Agent-Aware Boundary Network (ABN), which consists of two sub-networks (i) an Agent-Aware Representation Network to obtain both agent-agent and agents-environment relationships in the video representation, and (ii) a Boundary Generation Network to estimate the confidence score of temporal intervals. In the Agent-Aware Representation Network, the interactions between agents are expressed through local pathway, which operates at a local level to focus on the motions of agents whereas the overall perception of the surroundings are expressed through global pathway, which operates at a global level to perceive the effects of agents-environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks (i.e C3D, SlowFast and Two-Stream) show that our proposed ABN robustly outperforms state-of-the-art methods regardless of the employed backbone network on TAPG. We further examine the proposal quality by leveraging proposals generated by our method onto temporal action detection (TAD) frameworks and evaluate their detection performances. The source code can be found in this URL https://github.com/vhvkhoa/TAPG-AgentEnvNetwork.git.

Highlights

Temporal action proposal generation (TAPG) [1]–[14] is one of the most key and fundamental tasks in video understanding i.e. action recognition [15], [16], video summarization [17], [18], video captioning [19], [20], video recommendation [21], video highlight detection [22], and smart surveillance [23], [24]
To address the aforementioned limitations, we leverage human perception process of a temporal action proposal which is a combination of three entities i.e. agent, action, and environment and we propose a novel Contextual Agent-Aware Boundary Network (ABN)
Our ABN contains two components corresponding to Agent-Environment representation network and boundary generation network

Summary

Introduction

Temporal action proposal generation (TAPG) [1]–[14] is one of the most key and fundamental tasks in video understanding i.e. action recognition [15], [16], video summarization [17], [18], video captioning [19], [20], video recommendation [21], video highlight detection [22], and smart surveillance [23], [24]. Most of existing TAPG approaches first detect a set of possible starting and ending timestamps of all actions separately, and a proposal evaluation module is employed to evaluate every possible pair of starting and ending timestamps by predicting its confidence score. A robust TAPG method should be able to (i) generate temporal proposals with actual boundaries to cover action instances precisely and exhaustively; (ii) cover multi-duration actions; (iii) generate reliable confidence scores so that proposals can be retrieved properly [6].

Objectives

Methods

Findings

Conclusion