Activity Image-to-Video Retrieval via Domain Adversarial Learning

Abstract

With the rapid development of cross-modal retrieval, Activity Image-to-Video Retrieval (AIVR) aiming to retrieve videos containing similar activity as the query image, has attracted much attention. The core of AIVR is to learn motion information from the video to assist retrieval. In this paper, we present a novel Adversarial Activity Image-to-Video Retrieval (AAIVR) framework that acquires the video motion information by using domain adversarial learning. Domain adversarial learning is performed as an interplay between two processes. Firstly, representation disentanglement through encoders, tries to disentangle the video features into motion features and appearance features. Meanwhile, image appearance features from the query image are obtained. Then, we perform the image-to-video domain transfer to eliminate the modality inconsistency. A binary discriminator is designed to verify the transfer result. During testing, we conduct retrieval in both appearance and video feature spaces simultaneously. Extensive experimental results on two benchmark datasets show that the proposed AAIVR method significantly outperforms the state-of-the-art methods.