Mining negative samples on contrastive learning via curricular weighting strategy

Abstract

Contrastive learning, which pulls positive pairs closer and pushes away negative pairs, has remarkably propelled the development of self-supervised representation learning. Previous studies either neglected negative sample selection, resulting in suboptimal performance, or emphasized hard negative samples from the beginning of training, potentially leading to convergence issues. Drawing inspiration from curriculum learning, we find that learning with negative samples ranging from easy to hard improves both model performance and convergence rate. Therefore, we propose a dynamic negative sample weighting strategy for contrastive learning. Specifically, we design a loss function that adaptively adjusts the weights assigned to negative samples based on the model's performance. Initially, the loss prioritizes easy samples, but as training advances, it shifts focus to hard samples, enabling the model to learn more discriminative representations. Furthermore, to prevent an undue emphasis on false negative samples during later stages, which probably results in trivial solutions, we apply L2 regularization on the weights of hard negative samples. Extensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed weighting strategy. The ablation study confirms both the reasonableness of the curriculum and the effectiveness of the regularization.