A Multimodal Pairwise Discrimination Network for Cross-Domain Action Recognition

Abstract

In recent years, action recognition has become a hot research topic in the computer vision and machine learning domain. Despite many well-designed action recognition approaches have been proposed, we point out that some limitations still exist including the separated fusion of different Spatio-temporal features and the reconstruction classification model, and the requirement of similar environmental conditions when capturing the training and testing data. Thus, research interest has shifted from traditional action recognition towards cross-domain action recognition. To solve these limitations, in this work, we propose a novel multimodal pairwise discrimination network (short for MPD) for cross-domain action recognition that is an end-to-end network architecture. In MPD, it can jointly fuse different Spatio-temporal features from the video, learn domain invariant features for different action domains (source and target domains), and build the classification model. To characterize the shift between these domains, subnetwork parameters in corresponding layers of MPD are required to be relevant, but not identical. Besides, the domain invariant feature discrimination needs to be improved. Extensive experimental results on two different public benchmarks including indoor environment and outdoor environment demonstrate that our MPD solution can significantly outperform state-of-the-art methods with a 4% to 20% improvement in average accuracy.