Dual frame-level and region-level alignment for unsupervised video domain adaptation

Abstract

Most existing unsupervised video domain adaptation methods focus on extracting frame-level features and using temporal attention to create a domain-invariant feature space. However, this ignores the importance of different image regions and temporal dynamics. In this study, we propose a method that considers both frame-level and image-region level alignment to learn a dual spatial and temporal domain-invariant feature space. This is achieved by: 1) extracting frame-level image features and aligning them into a domain-invariant latent feature space; 2) extracting region-level temporal features and aligning them into a domain-invariant latent feature space. Optimal frame-level correlation is achieved through video-frame level alignment, while optimal image-region level correlation is demonstrated through image-region level alignment between the source and target videos. We use an LSTM network to parameterize this dual feature alignment, which is more efficient than attention mechanisms and results in significantly improved performance and reduced GPU memory cost by over 40% compared to current state-of-the-art methods.