Prior Matching Operator in Self-Supervised Learning

Abstract

In self-supervised representation learning (SSL), contrastive learning has achieved remarkable successes in recent years. In contrastive learning, the augmented views of the same image are brought closer (i.e., positive pairs), while views from different images (i.e., negative pairs) are separated apart. To perform well, contrastive methods of ten rely on a large number of negative pairs, which is computationally demanding. Intuitively, the negative samples play a role to scatter the representation of samples in space, which guarantees the uniformity of the representation space, thus making it the key to avoiding representation collapse (i.e., constant features). In this work, we propose an interpretable method, in which prior distribution matching is applied to prevent representation collapse, while no negative pair is required. Empirical experiments show promising results of our method yielded by the priors of Gaussian, uniform, and spherical distribution on the CIFAR-10 and CIFAR-100 datasets.