Progressive privileged knowledge distillation for online action detection

Abstract

Online Action Detection (OAD) in videos addresses the problem of real-time analysis for streaming videos, i.e., only the observed historical video frames are available at prediction time. Considering the future frames observable only at the training stage as a form of privileged information, this paper adopts the Learning Using Privileged Information (LUPI) paradigm. Knowledge distillation (KD) is employed to transfer the privileged information from the offline teacher to the online student. Note that this setting is different from conventional KD because the difference between the teacher and student models mostly lies in the input data rather than the network architecture. To relieves the input information gap for the LUPI, we propose a simple but effective Privileged Knowledge Distillation (PKD) method that enforce KD loss to partial hidden features of the student model. Moreover, we also schedules a curriculum learning procedure to gradually distill the privileged information. This approach is named as Progressive Privileged Knowledge Distillation (PPKD). Compared to some OAD methods that explicitly predict future frames or feature, our approach avoids predicting stage and achieves state-of-the-art accuracy on two popular OAD benchmarks, TVSeries and THUMOS14.