Abstract
Temporal action proposal generation (TAPG) aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet plays an important role in many tasks of video analysis and understanding. Despite the great achievement in TAPG, most existing works ignore the human perception of interaction between agents and the surrounding environment by applying a deep learning model as a black-box to the untrimmed videos to extract video visual representation. Therefore, it is beneficial and potentially improve the performance of TAPG if we can capture these interactions between agents and the environment. In this paper, we propose a novel framework named Agent-Aware Boundary Network (ABN), which consists of two sub-networks (i) an Agent-Aware Representation Network to obtain both agent-agent and agents-environment relationships in the video representation, and (ii) a Boundary Generation Network to estimate the confidence score of temporal intervals. In the Agent-Aware Representation Network, the interactions between agents are expressed through local pathway, which operates at a local level to focus on the motions of agents whereas the overall perception of the surroundings are expressed through global pathway, which operates at a global level to perceive the effects of agents-environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks (i.e C3D, SlowFast and Two-Stream) show that our proposed ABN robustly outperforms state-of-the-art methods regardless of the employed backbone network on TAPG. We further examine the proposal quality by leveraging proposals generated by our method onto temporal action detection (TAD) frameworks and evaluate their detection performances. The source code can be found in this URL https://github.com/vhvkhoa/TAPG-AgentEnvNetwork.git.
Highlights
Temporal action proposal generation (TAPG) [1]–[14] is one of the most key and fundamental tasks in video understanding i.e. action recognition [15], [16], video summarization [17], [18], video captioning [19], [20], video recommendation [21], video highlight detection [22], and smart surveillance [23], [24]
To address the aforementioned limitations, we leverage human perception process of a temporal action proposal which is a combination of three entities i.e. agent, action, and environment and we propose a novel Contextual Agent-Aware Boundary Network (ABN)
Our ABN contains two components corresponding to Agent-Environment representation network and boundary generation network
Summary
Temporal action proposal generation (TAPG) [1]–[14] is one of the most key and fundamental tasks in video understanding i.e. action recognition [15], [16], video summarization [17], [18], video captioning [19], [20], video recommendation [21], video highlight detection [22], and smart surveillance [23], [24]. Most of existing TAPG approaches first detect a set of possible starting and ending timestamps of all actions separately, and a proposal evaluation module is employed to evaluate every possible pair of starting and ending timestamps by predicting its confidence score. A robust TAPG method should be able to (i) generate temporal proposals with actual boundaries to cover action instances precisely and exhaustively; (ii) cover multi-duration actions; (iii) generate reliable confidence scores so that proposals can be retrieved properly [6].
Sign-in/Register to view highlights & summary Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
