SCICL: A Sparse Classifiers Induced Contrastive Learning Method

Abstract

As an effective self-supervised learning method for pre-training feature embeddings, contrastive learning aims at capturing the consistent information between augmented views of the input sample. The consistent information is considered to be discriminative, yet the precise semantics it entails remain elusive. Through experimentation, we further discover that consistent information is related to both invariant information between augmented views and partially varying information, but it may not necessarily be discriminative. Furthermore, our theoretical analysis reveals that employing sparse classifiers to ensure the invariance of soft labels can effectively assist neural networks in capturing discriminative features. Building upon the insights, we leverage sparse classifiers in combination with contrastive learning to ensure that the soft labels and the learned representations of different augmented views remain aligned simultaneously and call the method Sparse Classifiers Induced Contrastive Learning (SCICL). Extensive experimental results on various datasets and backbones show that SCICL can lead to stable improvements in performance, demonstrating that SCICL can facilitate the network in extracting more discriminative feature representations.