Dual-branch Cross-scale Feature Interaction for Temporal Action Detection

Abstract

Temporal Action Detection (TAD) is aim to predict action boundary and category simultaneously. Most existing RGB-based methods model temporal dependency using pyramid-style features without interaction among different scales, which usually result in inaccurate prediction for long-term actions. The reason is that features at different scales involve information with different granularity, which is suitable for either prediction of action boundary or category. In this paper, we present a novel Dual-branch Cross-scale Feature Interaction (DCFI) method that directly exchanges different scale information from both temporal and spatial perspective for TAD. To be specific, in one branch, a cross-scale temporal transformer module is devised to enable both semantic and temporal communications among different scale features with a merge-to-split mechanism. While the other branch designs a cross-scale spatial mixer module to mine the most salient spatial difference between consecutive and long-term frames via a scale-mixer. Benefiting from these two modules, DCFI achieves comprehensive temporal as well as spatial interaction across all feature scales, and thus accurately predicts the boundaries of different time-span action instances. Extensive experiments on two challenging benchmarks, i.e.., THUMOS-14 and ActivityNet-1.3, demonstrate that our DCFI achieves new state-of-the-art performance with only RGB.