DI-VTR: Dual inter-modal interaction model for video-text retrieval

Abstract

Video-text retrieval is a challenging task for multimodal information processing due to the semantic gap between different modalities. However, most existing methods do not fully mine the intra-modal interactions, as with the temporal correlation of video frames, which results in poor matching performance. Additionally, the imbalanced semantic information between videos and texts also leads to difficulty in the alignment of the two modalities. To this end, we propose a dual inter-modal interaction network for video-text retrieval, i.e., DI-VTR. To learn the intra-modal interaction of video frames, we design a contextual-related video encoder to obtain more fine-grained content-oriented video representations. We also propose a dual inter-modal interaction module to accomplish accurate multilingual alignment between the video and text modalities by introducing multilingual text to improve the representation ability of text semantic features. Extensive experimental results on commonly-used video-text retrieval datasets, including MSR-VTT, MSVD and VATEX, show that the proposed method achieves significantly improved performance compared with state-of-the-art methods.