Real-world Cross-modal Retrieval via Sequential Learning

Abstract

Cross-modal retrieval is playing an increasingly important role in our daily life with the explosive growth of multimedia data. However, its learning paradigm under real-life environments is less studied, and most existing approaches are developed in the pre-desired settings (e.g., unchanging modalities and explicitly modal-aligned samples). Inspired by the recent achievement in the field of cognition mechanism on how the human brain acquires knowledge, we present a new sequential learning method for real-world cross-modal retrieval. In this method, a unified model is maintained to capture the common knowledge of various modalities but are learned in a sequential manner such that it behaves adaptively according to the evolving distribution of different modalities, and needs no laborious alignment operations among multimodal data before learning. Furthermore, we reformulate the objective of optimization-based meta-learning and propose a novel meta-learning method to overcome the catastrophic forgetting encountered in sequential learning. Extensive experiments are conducted on four popular image-text multimodal datasets and a five-modal dataset, showing that our method achieves state-of-the-art cross-modal retrieval performance without explicit modal-alignment.