Class structure‐aware adversarial loss for cross‐domain human action recognition

Abstract

Cross-domain action recognition is a challenging vision task due to the domain shift and the absence of labeled data in the target domain. With only labelled source domain and unlabelled target domain data during training, some existing methods rely on an adversarial framework to align the features from different domains to a common latent space. However, the existing adversarial-based approaches have a major limitation of only attempting to perform the alignment from a holistic view, ignoring the underlying coherence of class structure across domains. A class structure-aware adversarial loss (CSCAL) is presented to address this issue. The CSCAL incorporates the category information into the adversarial learning branch to capture the fine-grained alignment of each class, effectively avoiding the false mixup of samples from different categories in the embedding space. Experiments on HMDB51, UCF101 and Olympic Sports datasets show significant improvement compared to the baseline. Code and trained model can be found at https://github.com/bregmangh/CSCAL.