Complementary Labels Research Articles

Leveraging weak complementary labels enhances semantic segmentation of hepatocellular carcinoma and intrahepatic cholangiocarcinoma

In this paper we present a deep learning segmentation approach to classify and quantify the two most prevalent primary liver cancers – hepatocellular carcinoma and intrahepatic cholangiocarcinoma – from hematoxylin and eosin (H&E) stained whole slide images. While semantic segmentation of medical images typically requires costly pixel-level annotations by domain experts, there often exists additional information which is routinely obtained in clinical diagnostics but rarely utilized for model training. We propose to leverage such weak information from patient diagnoses by deriving complementary labels that indicate to which class a sample cannot belong to. To integrate these labels, we formulate a complementary loss for segmentation. Motivated by the medical application, we demonstrate for general segmentation tasks that including additional patches with solely weak complementary labels during model training can significantly improve the predictive performance and robustness of a model. On the task of diagnostic differentiation between hepatocellular carcinoma and intrahepatic cholangiocarcinoma, we achieve a balanced accuracy of 0.91 (CI 95%: 0.86-0.95) at case level for 165 hold-out patients. Furthermore, we also show that leveraging complementary labels improves the robustness of segmentation and increases performance at case level.

Self-adaptive label discovery and multi-view fusion for complementary label learning

Unlike traditional supervised classification, complementary label learning (CLL) operates under a weak supervision framework, where each sample is annotated by excluding several incorrect labels, known as complementary labels (CLs). Despite reducing the labeling burden, CLL always suffers a decline in performance due to the weakened supervised information. To overcome such limitations, in this study, a multi-view fusion and self-adaptive label discovery based CLL method (MVSLDCLL) is proposed. The self-adaptive label discovery strategy leverages graph-based semi-supervised learning to capture the label distribution of each training sample as a convex combination of all its potential labels. The multi-view fusion module is designed to adapt to various views of feature representations. In specific, it minimizes the discrepancies of label projections between pairwise views, aligning with the consensus principle. Additionally, a straightforward mechanism inspired by a teamwork analogy is proposed to incorporate view-discrepancy for each sample. Experimental results demonstrate that MVSLDCLL learns more discriminative label distribution and achieves significantly higher accuracies compared to state-of-the-art CLL methods. Ablation study has also been performed to validate the effectiveness of both the self-adaptive label discovery strategy and the multi-view fusion module.

Recent advances in complementary label learning

Complementary Label Learning (CLL), a crucial aspect of weakly supervised learning, has seen significant theoretical and practical advancements. However, a comprehensive review of the field has been lacking. This survey provides the first exhaustive compilation and synthesis of state-of-the-art CLL approaches, filling a critical gap in the literature and serving as a foundational resource for the community. Key contributions of this survey include an extensive categorization of CLL methodologies, clarifying diverse algorithms based on their principles and applications. This classification scheme enhances understanding of the CLL landscape, highlighting its versatility across varied settings. Additionally, the survey examines the evolution of CLL, showcasing its adaptability and potential in addressing complex applications. It also explores experimental frameworks, including processes for generating complementary labels and datasets and numerical evaluation of existing state-of-the-art. Moreover, the survey delves into how CLL integrates with and enhances other weakly supervised and semi-supervised learning approaches, deepening understanding of its role in the broader machine learning ecosystem. Overall, this survey not only compiles CLL research but also guides future explorations, steering the field towards new horizons in weakly supervised learning.

Tackling biased complementary label learning with large margin

Complementary Label Learning (CLL) is a typical weakly supervised learning protocol, where each instance is associated with one complementary label to specify a class that the instance does not belong to. Current CLL approaches assume that complementary labels are uniformly sampled from all non-ground-truth labels, so as to implicitly and locally share complementary labels by solely reducing the logit of complementary label in one way or another. In this paper, we point out that, when the uniform assumption does not hold, existing CLL methods are weakened their ability to share complementary labels and fail in creating classifiers with large logit margin (LM), resulting in a significant performance drop. To address these issues, we instead present complementary logit margin (CLM) and empirically prove that increasing CLM contributes to the share of complementary labels under the biased CLL setting. Accordingly, we propose a surrogate complementary one-versus-rest loss (COVR) and demonstrate that optimization on COVR can effectively increase CLM with both theoretical and empirical evidences. Extensive experiments verify that the proposed COVR exhibits substantial improvement for both the biased CLL and even a more practical CLL setting: instance-dependent complementary label learning.

One model to use them all: training a segmentation model with complementary datasets

PurposeUnderstanding surgical scenes is crucial for computer-assisted surgery systems to provide intelligent assistance functionality. One way of achieving this is via scene segmentation using machine learning (ML). However, such ML models require large amounts of annotated training data, containing examples of all relevant object classes, which are rarely available. In this work, we propose a method to combine multiple partially annotated datasets, providing complementary annotations, into one model, enabling better scene segmentation and the use of multiple readily available datasets.MethodsOur method aims to combine available data with complementary labels by leveraging mutual exclusive properties to maximize information. Specifically, we propose to use positive annotations of other classes as negative samples and to exclude background pixels of these binary annotations, as we cannot tell if a positive prediction by the model is correct.ResultsWe evaluate our method by training a DeepLabV3 model on the publicly available Dresden Surgical Anatomy Dataset, which provides multiple subsets of binary segmented anatomical structures. Our approach successfully combines 6 classes into one model, significantly increasing the overall Dice Score by 4.4% compared to an ensemble of models trained on the classes individually. By including information on multiple classes, we were able to reduce the confusion between classes, e.g. a 24% drop for stomach and colon.ConclusionBy leveraging multiple datasets and applying mutual exclusion constraints, we developed a method that improves surgical scene segmentation performance without the need for fully annotated datasets. Our results demonstrate the feasibility of training a model on multiple complementary datasets. This paves the way for future work further alleviating the need for one specialized large, fully segmented dataset but instead the use of already existing datasets.

Adaptive Integration of Partial Label Learning and Negative Learning for Enhanced Noisy Label Learning

There has been significant attention devoted to the effectiveness of various domains, such as semi-supervised learning, contrastive learning, and meta-learning, in enhancing the performance of methods for noisy label learning (NLL) tasks. However, most existing methods still depend on prior assumptions regarding clean samples amidst different sources of noise (e.g., a pre-defined drop rate or a small subset of clean samples). In this paper, we propose a simple yet powerful idea called NPN, which revolutionizes Noisy label learning by integrating Partial label learning (PLL) and Negative learning (NL). Toward this goal, we initially decompose the given label space adaptively into the candidate and complementary labels, thereby establishing the conditions for PLL and NL. We propose two adaptive data-driven paradigms of label disambiguation for PLL: hard disambiguation and soft disambiguation. Furthermore, we generate reliable complementary labels using all non-candidate labels for NL to enhance model robustness through indirect supervision. To maintain label reliability during the later stage of model training, we introduce a consistency regularization term that encourages agreement between the outputs of multiple augmentations. Experiments conducted on both synthetically corrupted and real-world noisy datasets demonstrate the superiority of NPN compared to other state-of-the-art (SOTA) methods. The source code has been made available at https://github.com/NUST-Machine-Intelligence-Laboratory/NPN.

Complementary Labels Learning with Augmented Classes

Complementary Labels Learning (CLL) arises in many real-world tasks such as private questions classification and online learning, which aims to alleviate the annotation cost compared with standard supervised learning. Unfortunately, most previous CLL algorithms were in a stable environment rather than an open and dynamic scenarios, where data collected from unseen augmented classes in the training process might emerge in the testing phase. In this paper, we propose a novel problem setting called Complementary Labels Learning with Augmented Classes (CLLAC), which brings the challenge that classifiers trained by complementary labels should not only be able to classify the instances from observed classes accurately, but also recognize the instance from the Augmented Classes in the testing phase. Specifically, by using unlabeled data, we propose an unbiased estimator of classification risk for CLLAC, which is guaranteed to be provably consistent. Moreover, we provide generalization error bound for proposed method which shows that the optimal parametric convergence rate is achieved for estimation error. Finally, the experimental results on several benchmark datasets verify the effectiveness of the proposed method.

Biased Complementary-Label Learning Without True Labels.

In complementary-label learning (CLL), the complementary transition matrix, denoting the probabilities that true labels flip into complementary labels (CLs) which specify classes observations do not belong to, is crucial to building statistically consistent classifiers. Most existing works implicitly assume that the transition probabilities are identical, which is not true in practice and may lead to undesirable bias in solutions. Few recent works have extended the problem to a biased setting but limit their explorations to modeling the transition matrix by exploiting the complementary class posteriors of anchor points (i.e., instances that almost certainly belong to a specific class). However, due to the severe corruption and unevenness of biased CLs, both anchor points and complementary class posteriors are difficult to predict accurately in the absence of true labels. In this article, rather than directly predicting these two error-prone items, we instead propose a divided-T estimator as an alternative to effectively learn transition matrices from only biased CLs. Specifically, we exploit semantic clustering to mitigate the adverse effects arising from CLs. By introducing the learned semantic clusters as an intermediate class, we factorize the original transition matrix into the product of two easy-to-estimate matrices that are not reliant on the two error-prone items. Both theoretical analyses and empirical results justify the effectiveness of the divided- T estimator for estimating transition matrices under a mild assumption. Experimental results on benchmark datasets further demonstrate that the divided- T estimator outperforms state-of-the-art (SOTA) methods by a substantial margin.

Learning From Incorrectness: Active Learning With Negative Pre-Training and Curriculum Querying for Histological Tissue Classification.

Patch-level histological tissue classification is an effective pre-processing method for histological slide analysis. However, the classification of tissue with deep learning requires expensive annotation costs. To alleviate the limitations of annotation budgets, the application of active learning (AL) to histological tissue classification is a promising solution. Nevertheless, there is a large imbalance in performance between categories during application, and the tissue corresponding to the categories with relatively insufficient performance are equally important for cancer diagnosis. In this paper, we propose an active learning framework called ICAL, which contains Incorrectness Negative Pre-training (INP) and Category-wise Curriculum Querying (CCQ) to address the above problem from the perspective of category-to-category and from the perspective of categories themselves, respectively. In particular, INP incorporates the unique mechanism of active learning to treat the incorrect prediction results that obtained from CCQ as complementary labels for negative pre-training, in order to better distinguish similar categories during the training process. CCQ adjusts the query weights based on the learning status on each category by the model trained by INP, and utilizes uncertainty to evaluate and compensate for query bias caused by inadequate category performance. Experimental results on two histological tissue classification datasets demonstrate that ICAL achieves performance approaching that of fully supervised learning with less than 16% of the labeled data. In comparison to the state-of-the-art active learning algorithms, ICAL achieved better and more balanced performance in all categories and maintained robustness with extremely low annotation budgets. The source code will be released at https://github.com/LactorHwt/ICAL.

Towards Robust Learning with Noisy and Pseudo Labels for Text Classification

Unlike Positive Training (PT), Negative Training (NT) is an indirect learning technique that trains the model on a combination of clean and noisy data using complementary labels, which are randomly generated from the label space except for the actual label. Although clean samples have identical distributions to the test samples, they are treated with the same level of uncertainty as noisy samples because of the complementary labeling of NT. Consequently, their contribution to the overall performance is relatively lower. We propose a Learning with Noisy and Pseudo Label (LNPL) framework, which jointly trains the model using PT and NT on clean and noisy data, respectively. We aim to enable direct learning on clean samples while leveraging the robustness of NT against noise in a unified framework. To mitigate the abundance of noisy instances, we leverage a gradient reversal layer at the top of LNPL as a regularization term to mislead the recognition of the source of the instance (e.g., clean or noisy). Moreover, we introduce a self-training LNPL that performs a semi-supervised text classification task as a learning with noisy pseudo-label problem. Extensive experiments on various textual benchmark datasets demonstrate that LNPL is robust and consistently outperforms the alternatives. The code is available on GitHub.1

Rethinking precision of pseudo label: Test-time adaptation via complementary learning

Test-time adaptation (TTA) has been proven that is capable of transferring the source model to a different data distribution. The main challenge of TTA is how to learn the target-domain representation without the help of ground truth annotation. Previous studies have employed unsupervised objectives, such as the entropy of model predictions, as optimization targets to effectively learn features for test-time samples. However, the performance of the model is easily compromised by the quality of pseudo labels. In this work, we propose a novel way to leverage complementary learning to facilitate target feature learning, along with a complementary-labeling scenario and dynamical refining of the decision boundary. The use of complementary label learning effectively prevents the model from collapsing to high-confidence instances with incorrect pseudo labels. The complementary labels are less informative than ordinary labels and thus a suitable risk estimator is derived to better learn from them. We highlight that this learning of complementary labels agrees with Vanilla learning with ordinary data distribution. Comprehensive experiments show that the proposed learning algorithm achieves state-of-the-art performance on different datasets and experiment settings.

Boosting semi-supervised learning with Contrastive Complementary Labeling

Semi-supervised learning (SSL) approaches have achieved great success in leveraging a large amount of unlabeled data to learn deep models. Among them, one popular approach is pseudo-labeling which generates pseudo labels only for those unlabeled data with high-confidence predictions. As for the low-confidence ones, existing methods often simply discard them because these unreliable pseudo labels may mislead the model. Unlike existing methods, we highlight that these low-confidence data can be still beneficial to the training process. Specifically, although we cannot determine which class a low-confidence sample belongs to, we can assume that this sample should be very unlikely to belong to those classes with the lowest probabilities (often called complementary classes/labels). Inspired by this, we propose a novel Contrastive Complementary Labeling (CCL) method that constructs a large number of reliable negative pairs based on the complementary labels and adopts contrastive learning to make use of all the unlabeled data. Extensive experiments demonstrate that CCL significantly improves the performance on top of existing advanced methods and is particularly effective under the label-scarce settings. For example, CCL yields an improvement of 2.43% over FixMatch on CIFAR-10 only with 40 labeled data.

Cross-Domain Facial Expression Recognition through Reliable Global–Local Representation Learning and Dynamic Label Weighting

Cross-Domain Facial Expression Recognition (CD-FER) aims to develop a facial expression recognition model that can be trained in one domain and deliver consistent performance in another. CD-FER poses a significant challenges due to changes in marginal and class distributions between source and target domains. Existing methods primarily emphasize achieving domain-invariant features through global feature adaptation, often neglecting the potential benefits of transferable local features across different domains. To address this issue, we propose a novel framework for CD-FER that combines reliable global–local representation learning and dynamic label weighting. Our framework incorporates two key modules: the Pseudo-Complementary Label Generation (PCLG) module, which leverages pseudo-labels and complementary labels obtained using a credibility threshold to learn domain-invariant global and local features, and the Label Dynamic Weight Matching (LDWM) module, which assesses the learning difficulty of each category and adaptively assigns corresponding label weights, thereby enhancing the classification performance in the target domain. We evaluate our approach through extensive experiments and analyses on multiple public datasets, including RAF-DB, FER2013, CK+, JAFFE, SFW2.0, and ExpW. The experimental results demonstrate that our proposed model outperforms state-of-the-art methods, with an average accuracy improvement of 3.5% across the five datasets.

ComCo: Complementary supervised contrastive learning for complementary label learning

Complementary label learning (CLL) is an important problem that aims to reduce the cost of obtaining large-scale accurate datasets by only allowing each training sample to be equipped with labels the sample does not belong. Despite its promise, CLL remains a challenging task. Previous methods have proposed new loss functions or introduced deep learning-based models to CLL, but they mostly overlook the semantic information that may be implicit in the complementary labels. In this work, we propose a novel method, ComCo, which leverages a contrastive learning framework to assist CLL. Our method includes two key strategies: a positive selection strategy that identifies reliable positive samples and a negative selection strategy that skillfully integrates and leverages the information in the complementary labels to construct a negative set. These strategies bring ComCo closer to supervised contrastive learning. Empirically, ComCo significantly achieves better representation learning and outperforms the baseline models and the current state-of-the-art by up to 14.61% in CLL.

A fundus image classification framework for learning with noisy labels.

Fundus images are widely used in the screening and diagnosis of eye diseases. Current classification algorithms for computer-aided diagnosis in fundus images rely on large amounts of data with reliable labels. However, the appearance of noisy labels degrades the performance of data-dependent algorithms, such as supervised deep learning. A noisy label learning framework suitable for the multiclass classification of fundus diseases is presented in this paper, which combines data cleansing (DC), adaptive negative learning (ANL), and sharpness-aware minimization (SAM) modules. Firstly, the DC module filters the noisy labels in the training dataset based on the prediction confidence. Then, the ANL module modifies the loss function by choosing complementary labels, which are neither the given labels nor the labels with the highest confidence. Moreover, for better generalization, the SAM module is applied by simultaneously optimizing the loss and its sharpness. Extensive experiments on both private and public datasets show that our method greatly promotes the performance for classification of multiple fundus diseases with noisy labels.

Class-imbalanced complementary-label learning via weighted loss

Complementary-label learning (CLL) is widely used in weakly supervised classification, but it faces a significant challenge in real-world datasets when confronted with class-imbalanced training samples. In such scenarios, the number of samples in one class is considerably lower than in other classes, which consequently leads to a decline in the accuracy of predictions. Unfortunately, existing CLL approaches have not investigate this problem. To alleviate this challenge, we propose a novel problem setting that enables learning from class-imbalanced complementary labels for multi-class classification. To tackle this problem, we propose a novel CLL approach called Weighted Complementary-Label Learning (WCLL). The proposed method models a weighted empirical risk minimization loss by utilizing the class-imbalanced complementary labels, which is also applicable to multi-class imbalanced training samples. Furthermore, we derive an estimation error bound to provide theoretical assurance. To evaluate our approach, we conduct extensive experiments on several widely-used benchmark datasets and a real-world dataset, and compare our method with existing state-of-the-art methods. The proposed approach shows significant improvement in these datasets, even in the case of multiple class-imbalanced scenarios. Notably, the proposed method not only utilizes complementary labels to train a classifier but also solves the problem of class imbalance.

Simple and Effective Complementary Label Learning Based on Mean Square Error Loss

In this paper, we propose a simple and effective complementary label learning approach to address the label noise problem for deep learning model. Different surrogate losses have been proposed for complementary label learning, however, are often sophisticated designed, as the losses are required to satisfy the classifier consistency property. We propose an effective square loss for complementary label learning under unbiased and biased assumptions. We also show theoretically that our method assurances that the optimal classifier under complementary labels is also the optimal classifier under ordinary labels. Finally, we test our method on three different benchmark datasets with biased and unbiased assumptions to verify the effectiveness of our method.

Complementary label learning based on knowledge distillation.

Complementary label learning (CLL) is a type of weakly supervised learning method that utilizes the category of samples that do not belong to a certain class to learn their true category. However, current CLL methods mainly rely on rewriting classification losses without fully leveraging the supervisory information in complementary labels. Therefore, enhancing the supervised information in complementary labels is a promising approach to improve the performance of CLL. In this paper, we propose a novel framework called Complementary Label Enhancement based on Knowledge Distillation (KDCL) to address the lack of attention given to complementary labels. KDCL consists of two deep neural networks: a teacher model and a student model. The teacher model focuses on softening complementary labels to enrich the supervision information in them, while the student model learns from the complementary labels that have been softened by the teacher model. Both the teacher and student models are trained on the dataset that contains only complementary labels. To evaluate the effectiveness of KDCL, we conducted experiments on four datasets, namely MNIST, F-MNIST, K-MNIST and CIFAR-10, using two sets of teacher-student models (Lenet-5+MLP and DenseNet-121+ResNet-18) and three CLL algorithms (PC, FWD and SCL-NL). Our experimental results demonstrate that models optimized by KDCL outperform those trained only with complementary labels in terms of accuracy.

GAN-CL: Generative Adversarial Networks for Learning From Complementary Labels.

Learning from complementary labels (CLs) is a useful learning paradigm, where the CL specifies the classes that the instance does not belong to, instead of providing the ground truth as in the ordinary supervised learning scenario. In general, although it is less laborious and more efficient to collect CLs compared with ordinary labels, the less informative signal in the complementary supervision is less helpful to learn competent feature representation. Consequently, the final classifier's performance greatly deteriorates. In this article, we leverage generative adversarial networks (GANs) to derive an algorithm GAN-CL to effectively learn from CLs. In addition to the role in original GAN, the discriminator also serves as a normal classifier in GAN-CL, with the objective constructed partly with the complementary information. To further prove the effectiveness of our schema, we study the global optimality of both generator and discriminator for the GAN-CL under mild assumptions. We conduct extensive experiments on benchmark image datasets using deep models, to demonstrate the compelling improvements, compared with state-of-the-art CL learning approaches.

Self-supervised knowledge distillation for complementary label learning

In this paper, we tackle a new learning paradigm called learning from complementary labels, where the training data specifies classes that instances do not belong to, instead of the accuracy labels. In general, it is more efficient to collect the complementary labels compared with collecting the supervised ones, with no need for selecting the correct one from a number of candidates. While current state-of-the-art methods design various loss functions to train competitive models by the limited supervised information, they overlook learning from the data and model themselves, which always contain fruitful information that can improve the performance of complementary label learning. In this paper, we propose a novel learning framework, which seamlessly integrates self-supervised and self-distillation to complementary learning. Based on the general complementary learning framework, we employ an entropy regularization term to guarantee the network outputs exhibit a sharper state. Then, to intensively learn information from the data, we leverage the self-supervised learning based on rotation and transformation operations as a plug-in auxiliary task to learn better transferable representations. Finally, knowledge distillation is introduced to further extract the “dark knowledge” from a network to guide the training of a student network. In the extensive experiments, our method surprisingly demonstrates compelling performance in accuracy over several state-of-the-art approaches.