Cross-Modal Learning to Rank with Adaptive Listwise Constraint

Abstract

Multi-modal data lies on heterogeneous feature spaces, which brings a significant challenge to cross-modal retrieval. Some works have been proposed to cope with this problem by learning a common subspace. However, previous methods often learn the common subspace by enhancing the relation between embedded features and relevant class labels but ignore the relation between embedded features and irrelevant class labels. Additionally, most methods assume that irrelevant samples are of equal importance. Considering this, we propose to train an optimal common embedding space via cross-modal learning to rank with adaptive listwise constraint (CMAL2R) based on two-branch neural networks. The listwise loss function in CMAL2R adaptively assigns larger margins to harder irrelevant samples, strengthening the relation between embedded features and irrelevant class labels. Experiments on Wikipedia and Pascal datasets demonstrate the effectiveness for bi-directional image-text retrieval.