Sequence feature generation with temporal unrolling network for zero-shot action recognition

Abstract

Zero-Shot Action Recognition (ZSAR) aims to recognize unseen action classes not included in the training dataset. Existing generative methods for ZSAR synthesize a feature of unseen action from a class embedding to overcome the absence of training data. Specifically, previous methods synthesize a feature which is averaged along a time axis, even though a video is extracted as a sequence of feature vectors. They suffer from the ambiguity of temporal information, which leads to confusion among actions sharing similar subactions. To tackle the problem, we first propose to synthesize not an averaged feature but a sequence consisting of feature vectors along the time axis. Hence, we design Sequence Feature Generative Adversarial Network (SFGAN) with Temporal Unrolling NEtwork (TUNE), which unrolls a class embedding into a set of condition vectors for generating sequences of features. Also, we employ a sequence discriminator as the second teacher. Through extensive experiments on the three benchmarks, HMDB51, UCF101, and Olympic, we validate the efficacy of sequence generation for ZSAR, and our method achieves the state-of-the-art generalized zero-shot learning performances.