Debiased momentum contrastive learning for multimodal video similarity measures

Abstract

The growing potential of multimodal short videos has contributed to a new type of recommendation. It depends on effectively measuring the similarities between the short video pairs, which consist of video frames and descriptions. Previous studies proposed using contrastive learning with in-batch negative sampling. However, such a strategy would take other semantically similar samples in the batch as negatives, leading to a biased issue. This paper proposes a debiased momentum contrastive learning (DMCL) on the unified multimodal Transformer model (VideoSim) for video similarity measures. The proposed DMCL alleviates the bias issue of negative sampling by introducing implicit knowledge of the model itself as soft labels. Instead of simply taking other samples in the batch as negatives, the proposed DMCL applies soft labels as supervision from the ground truth and the contextual semantic similarities between video-text pairs to alleviate the bias caused by negative sampling. In addition, DMCL expands the negative samples by a momentum queue strategy, which allows storing more multimodal representations to contrast more negatives. The experimental results of measuring multimodal video similarity show that the proposed DMCL outperforms all baselines in terms of Spearman’s rank correlation. Ablation studies and extensive analysis further demonstrate the effectiveness of the proposed method.