Mr.CAN: Class-Aware Network with Multi-Relations for Temporal Action Detection

Abstract

Recognizing action patterns and exploring multiple relations are vital for Temporal Action Detection (TAD) task, which aims at locating and classifying action segments in untrimmed videos. However, most existing methods attempt to build a general model to handle diverse actions, ignoring the huge difference between various classes. Besides, the exploration of temporal and semantic relations between different segments remains an ongoing challenge due to complex video content. In this paper, we contend that different action classes should be processed differently and thus design a new Class-Aware Mechanism to achieve accurate detection. Moreover, an effective module named Multi-relations Builder is proposed to establish temporal and semantic relations simultaneously. These two modules are integrated as Class-Aware Network with Multi-relations (MrCAN). In comprehensive experiments conducted on two benchmarks, it out-performs all other current methods and achieves state-of-the-art performance, improving the average mAP from 45.78% to 48.98% on THUMOS-14 and from 35.52% to 35.87% on ActivityNet-1.3 respectively. Furthermore, the well-designed Multi-relations Builder can also be used to boost some other existing methods.