MIFNet: Multiple instances focused temporal action proposal generation

Abstract

Temporal action proposal generation (TAPG) serves as a promising solution for video analysis. However, the performance of existing methods is still far from satisfactory for real-world applications. We attribute it to a crucial issue, i.e., hard multiple instances. In this paper, we investigate why this is the case. We discover that when processing multiple instances videos, mainstream approaches always recognize multiple instances as one instance due to boundary ambiguity or ignoring insignificant backgrounds between these instances. To address this problem, we propose a MultipleInstancesFocusedNetwork(MIFNet) that improves the quality of action proposals by considering boundary correlations and fusing multi-scale proposals. In particular, we first propose a pure boundary embedding module named Boundary Constraint Module (BCM) for suppressing the generation of hard negatives proposal by evaluating boundary correlation. The BCM introduces a boundary contrastive learning strategy that can pull the positive boundary pairs’ representation closer and push the negative pairs’ representation away. Then, a Proposal Blending Module (PBM) is proposed, which augments the proposal-level representation by modeling information among multi-scale proposals so that proposals can be complemented with local details as well as global information. The experimental results on the ActivityNet-v1.3 and THUMOS14 benchmarks demonstrate that MIFNet outperforms the state-of-the-arts.