Coupling Adversarial Graph Embedding for transductive zero-shot action recognition

Abstract

Zero-shot action recognition (ZSAR) aims to recognize novel actions that have not been seen in the training stage. However, ZSAR always suffers from serious domain shift problem, which causes poor performance. This is because: 1) Videos contain complicated intrinsic structures, including cross-sample visual correlations and cross-category semantic relationships, which make it challenging to generalize domain shift over categories and transfer knowledge across videos. 2) Existing methods do not disentangle unique and shared information underlying unseen videos during embedding. They are always weakly adaptive to novel categories and easily shift unseen videos to irrelevant action prototypes. In this paper, we propose a novel Coupling Adversarial Graph Embedding (CAGE) method for ZSAR, which formulates an effective visual-to-semantic embedding to alleviate the domain shift problem. Our model implements in a transductive setting that assumes accessing to a full set of unseen videos. Firstly, a structured graph is built for expressing both seen and unseen videos, which integrally captures visual and semantic relationships between them. Then, an effective visual-to-semantic embedding is formulated based on graph convolutional network (GCN), which is generalized to disjoint action categories and optimized for label propagation. In addition, a couple of adversarial constraints are proposed to characterize unique information of unseen videos and purify shared information across categories, which further improve the adaptability and discriminability of our model. Experiments on Olympic sports, HMDB51 and UCF101 datasets show that our model achieves impressive performance on ZSAR task.