Margin-aware Noise-robust Contrastive Learning for Partially View-aligned Problem

Abstract

In this paper, we study a challenging problem in contrastive learning when just a portion of data is aligned in multi-view dataset due to temporal, spatial, or spatio-temporal asynchronism across views. It is important to study partially view-aligned data since this type of data is common in real-world application and easily leads to data inconsistency among different views. Such a Partially View-aligned Problem (PVP) in contrastive learning has been relatively less touched so far, especially in downstream tasks, i.e., classification and clustering. In order to solve this problem, we introduce a flexible margin and propose margin-aware noise-robust contrastive learning to simultaneously identify the within-category counterparts from the other view of one data point based on the established cross-view correspondence and learn a shared representation. To be specific, the proposed learning framework is built on a novel margin-aware noise-robust contrastive loss. Since data pairs are used as input for the proposed margin-aware noise-robust contrastive learning, we build positive pairs according to the known correspondences and negative pairs in the manner of random sampling. Our margin-aware noise-robust contrastive learning framework is able to effectively reduce or remove the impacts caused by the possible existing noise for the constructed pairs in a margin-aware manner, i.e., false negative pairs leaded by random sampling in PVP. We relax the proposed margin-aware noise-robust contrastive loss and then give a detailed mathematical analysis for the effectiveness of our loss. As an instantiation, we construct an example under the proposed margin-aware noise-robust contrastive learning framework for validation in this work. To the best of our knowledge, this is the first attempt of extending contrastive learning to a margin-aware noise-robust version for dealing with PVP. We also enrich the learning paradigm when there is noise in the data. Extensive experiments on different datasets demonstrate the promising performance of the proposed method in the classification and clustering tasks.