Label Noise Research Articles

SPORT: A Subgraph Perspective on Graph Classification with Label Noise

Graph neural networks (GNNs) have achieved great success recently on graph classification tasks using supervised end-to-end training. Unfortunately, extensive noisy graph labels could exist in the real world because of the complicated processes of manual graph data annotations, which may significantly degrade the performance of GNNs. Therefore, we investigate the problem of graph classification with label noise, which is demanding because of the complex graph representation learning issue and serious memorization of noisy samples. In this work, we present a novel approach called S ubgra p h Set Netw or k with Sample Selection and Consis t ency Learning (SPORT) for this problem. To release the overfitting of GNNs, SPORT proposes to characterize each graph as a set of subgraphs generated by certain predefined stratagems, which can be viewed as samples from its underlying semantic distribution in graph space. Then we develop an equivariant network to encode the subgraph set with the consideration of the symmetry group. To further release the influences of noisy examples, we leverage the predictions of subgraphs to measure the likelihood of a sample being clean or noisy, followed by effective label updating. In addition, we propose a joint loss to advance the model generalizability by introducing consistency regularization. Comprehensive experiments on a wide range of graph classification datasets demonstrate the effectiveness of our SPORT. Specifically, SPORT outperforms the most competing baseline by up to 6.4%.

Phenotyping Alfalfa (Medicago sativa L.) Root Structure Architecture via Integrating Confident Machine Learning with ResNet-18.

Background: Root system architecture (RSA) is of growing interest in implementing plant improvements with belowground root traits. Modern computing technology applied to images offers new pathways forward to plant trait improvements and selection through RSA analysis (using images to discern/classify root types and traits). However, a major stumbling block to image-based RSA phenotyping is image label noise, which reduces the accuracies of models that take images as direct inputs. To address the label noise problem, this study utilized an artificial intelligence model capable of classifying the RSA of alfalfa (Medicago sativa L.) directly from images and coupled it with downstream label improvement methods. Images were compared with different model outputs with manual root classifications, and confident machine learning (CL) and reactive machine learning (RL) methods were tested to minimize the effects of subjective labeling to improve labeling and prediction accuracies. Results: The CL algorithm modestly improved the Random Forest model's overall prediction accuracy of the Minnesota dataset (1%) while larger gains in accuracy were observed with the ResNet-18 model results. The ResNet-18 cross-population prediction accuracy was improved (~8% to 13%) with CL compared to the original/preprocessed datasets. Training and testing data combinations with the highest accuracies (86%) resulted from the CL- and/or RL-corrected datasets for predicting taproot RSAs. Similarly, the highest accuracies achieved for the intermediate RSA class resulted from corrected data combinations. The highest overall accuracy (~75%) using the ResNet-18 model involved CL on a pooled dataset containing images from both sample locations. Conclusions: ResNet-18 DNN prediction accuracies of alfalfa RSA image labels are increased when CL and RL are employed. By increasing the dataset to reduce overfitting while concurrently finding and correcting image label errors, it is demonstrated here that accuracy increases by as much as ~11% to 13% can be achieved with semi-automated, computer-assisted preprocessing and data cleaning (CL/RL).

Dynamic training for handling textual label noise

Dynamic training for handling textual label noise

A novel transformer-based few-shot learning method for intelligent fault diagnosis with noisy labels under varying working conditions

Recent years have witnessed the success of Few-shot Learning (FSL) methods in equipment reliability enhancement and fault diagnosis, by virtue of learning from limited data and adapting to new operating conditions. However, due to sensor bias, manual collection, and mislabeling, label noise is inevitably introduced into the dataset, which further reduces the quality of supervised information contained in the few-shot dataset, posing significant challenges for accurate fault diagnosis. In this paper, the problem of Few-shot Fault Diagnosis with Noisy Labels (FFDNL) is studied for the first time, and a novel method named Enhanced Transformer with Asymmetric Loss Function (ETALF) is proposed. ETALF leverages the self-attention mechanism of the transformer to dynamically measure the similarity between fault samples in the support set to enhance the model's robustness against label noise, then naturally aggregates the similar samples into corresponding correct prototypes. Furthermore, an asymmetric loss function is designed, which adaptively assigns the model with larger penalties for incorrect category predictions and smaller penalties for correct category predictions, thereby enhancing fault diagnostic performance through inherent asymmetry. Comprehensive experiments are conducted on two benchmark datasets, and the compared results with representative approaches validate the effectiveness of our proposed ETALF in performing intelligent fault diagnosis using limited and noise-labeled data under varying working conditions, which achieves accuracies of 97.77% and 95.78% with 0.2 noisy-level labels during meta-training and meta-testing on the CWRU and KAIST datasets, respectively.

Noise-robust re-identification with triple-consistency perception

Traditional re-identification (ReID) methods heavily rely on clean and accurately annotated training data, rendering them susceptible to label noise in real-world scenarios. Although some noise-robust learning methods have been proposed and achieved promising recognition performance, however, most of these methods are designed for the image classification task and they are not suitable in ReID (engaging in the association and matching of objects rather than solely focusing on their identification). To address this problem, in this paper, we propose a Triple-consistency Perception based Noise-robust Re-identification Model (TcP-ReID), in which we make the model mine and focus more on the clean samples and reliable relationships among samples from different perspectives. Specifically, the self-consistency strategy guides the model to emphasize and prioritize clean samples, thereby preventing overfitting to noise labels during the initial stages of model training. Rather than focusing solely on individual samples, the context-consistency loss exploits similarities between samples in the feature space, promoting predictions for each sample to align with those of its nearest neighbors. Moreover, to further enforce the robustness of our model, a Jensen-Shannon divergence based cross-view consistency loss is introduced by encouraging the consistency of samples across different views. Extensive experiments demonstrate the superiority of the proposed TcP-ReID over the competing methods under instance-dependent noise and instance-independent noise. For instance, on the Market1501 dataset, our method achieves 85.8% in rank-1 accuracy and 56.3% in mAP score (5.6% and 8.7% improvements) under instance-independent noise with noise ratio 50%, and similarly 5.7% and 1.4% under instance-dependent label noise.

Mitigating data imbalance and noise: A divergence-based approach with enhanced sample selection

Label noise and distribution imbalance are common challenges in real-world datasets. To address noisy labels, sample selection methods are typically used. These methods identify small-loss data as correctly labeled and utilize only these clean samples for updating parameters during training. However, our thorough analysis shows that in scenarios with a long-tailed distribution, methods based on small-loss tend to favor samples from dominant classes, leading to a significant drop in performance. To tackle this issue, we introduce a divergence-based separation mechanism coupled with prototype distance for clean sample identification across each class, effectively mitigating the adverse effects of distribution imbalance on sample selection. Furthermore, drawing on recent theoretical insights that highlight the benefits of pseudo-labels in managing class imbalances, we enhance our approach by integrating semi-supervised and contrastive learning techniques. Comprehensive tests conducted on two synthetic noisy long-tailed datasets and one real-world dataset demonstrate the efficacy of our method in dealing with label noise and long-tailed distributions.

Label noise learning with the combination of CausalNL and CGAN models

Label noise learning with the combination of CausalNL and CGAN models

Dynamic selection for reconstructing instance-dependent noisy labels

As an inevitable issue in annotating large-scale datasets, instance-dependent label noise (IDN) can cause serious overfitting in neural networks. To combat IDN, label reconstruction methods have been developed with noise transition matrices or DNNs to simulate the transition from clean labels to noisy labels. Nevertheless, the absence of correct supervisions will lead to learning wrong noise transitions. This motivates us to select samples with clean labels to fetch the correct supervisions. However, the difficulty in obtaining prior knowledge of the noise rate prohibits the use of existing sample selection methods. To this end, we propose a dynamic sample selection method, namely Identity Mapping (IdMap), to overcome this limitation. Inspired by the feature-dependent characteristic of IDN, we first introduce the extracted instance features and pseudo-ground-truth labels to reconstruct noisy labels. A partial identity mapping between two labels is then established and samples with consistent identity mapping output are selected as clean data to update the classifier. Extensive experiments on both artificial and real-world noisy datasets demonstrate the superiority of IdMap compared with other state-of-the-art methods.

CoMix: Confronting with Noisy Label Learning with Co-training Strategies on Textual Mislabeling

The existence of noisy labels is inevitable in real-world large-scale corpora. As deep neural networks are notably vulnerable to overfitting on noisy samples, this highlights the importance of the ability of language models to resist noise for efficient training. However, little attention has been paid to alleviating the influence of label noise in natural language processing. To address this problem, we present CoMix, a robust Noise-Against training strategy taking advantage of Co-training that deals with textual annotation errors in text classification tasks. In our proposed framework, the original training set is first split into labeled and unlabeled subsets according to a sample partition criteria and then applies label refurbishment on the unlabeled subsets. We implement textual interpolation in hidden space between samples on the updated subsets. Meanwhile, we employ peer diverged networks simultaneously leveraging co-training strategies to avoid the accumulation of confirm bias. Experimental results on three popular text classification benchmarks demonstrate the effectiveness of CoMix in bolstering the network’s resistance to label mislabeling under various noise types and ratios, which also outperforms the state-of-the-art methods.

Robust consistency learning for facial expression recognition under label noise

Robust consistency learning for facial expression recognition under label noise

Robust two-stage instance-level cost-sensitive learning method for class imbalance problem

The class imbalance problem is a pervasive challenge in various applications, prompting researchers to design specialized loss functions for instance-level cost-sensitive learning models. However, the performance deteriorates when datasets are contaminated by label noise, especially in scenarios with uneven noise distribution. Additionally, specific hard-to-classify samples can further impede performance. Addressing these issues and establishing a robust instance-level cost-sensitive learning framework remains a formidable challenge. In this paper, we propose a robust Two-Stage instance-level cost-sensitive learning method with the Bounded Quadratic Type Squared Error (BQTSE) loss function, termed TSBQT. The asymmetric and bounded nature of the BQTSE loss function not only allows for distinct cost assignments to majority and minority instances, but also enhances the robustness to uneven noise. Moreover, our two-stage learning strategy prioritizes hard-to-classify instances by considering their classification difficulty. We solve TSBQT using the alternating direction method of multipliers and gradient descent algorithm. The generalization capability of TSBQT is analyzed with the Rademacher complexity theory. Extensive experiments validate the effectiveness of TSBQT.

Neighborhood repartition-based oversampling algorithm for multiclass imbalanced data with label noise

The classification of imbalanced data remains one of the most significant topics in contemporary data analysis. Existing classification algorithms tend to favor majority classes, leading to false predictions and difficulties in addressing class overlap or label noise. These challenges are particularly evident in multiclass settings, where the mutual imbalance relationships among classes become more complex. Despite this, the vast majority of research in this field has concentrated on binary problems, while the more difficult multiclass problems are relatively underexplored. In this paper, we propose a novel data-sampling technique, a Multiclass Neighborhood Repartition-based Oversampling (MC-NRO) algorithm. The innovation of this method lies in it considers local data characteristics of each class to constrain the oversampled neighborhood. MC-NRO calculates the mutual potential of different classes to precisely optimize the subregions for generating new instances. By selecting different repartition neighborhoods to meet the needs of specific domain, it can detect outliers and label noise, expand the decision boundary of minority class, and avoid class overlap through data cleaning. The experimental results demonstrate that MC-NRO outperforms other advanced oversampling strategies, ranking first on average across the three evaluation metrics, and exhibits robustness, especially in datasets with high noise levels. More importantly, MC-NRO is highly versatile and can be flexibly applied to various classifiers, and is particularly suitable for processing naturally complex (i.e., not affected by noise) datasets.

Intuitionistic fuzzy generalized eigenvalue proximal support vector machine

Generalized eigenvalue proximal support vector machine (GEPSVM) has attracted widespread attention due to its simple architecture, rapid execution, and commendable performance. GEPSVM gives equal significance to all samples, thereby diminishing its robustness and efficacy when confronted with real-world datasets containing noise and outliers. In order to reduce the impact of noises and outliers, we propose a novel intuitionistic fuzzy generalized eigenvalue proximal support vector machine (IF-GEPSVM). The proposed IF-GEPSVM assigns the intuitionistic fuzzy score to each training sample based on its location and surroundings in the high-dimensional feature space by using a kernel function. The solution of the IF-GEPSVM optimization problem is obtained by solving a generalized eigenvalue problem. Further, we propose an intuitionistic fuzzy improved generalized eigenvalue proximal support vector machine (IF-IGEPSVM) by solving standard eigenvalue decomposition resulting in simpler optimization problems with less computation cost which leads to an efficient intuitionistic fuzzy-based model. We conduct a comprehensive evaluation of the proposed IF-GEPSVM and IF-IGEPSVM models on UCI and KEEL benchmark datasets. Moreover, to evaluate the robustness of the proposed IF-GEPSVM and IF-IGEPSVM models, label noise is introduced into some UCI and KEEL datasets. The experimental findings showcase the superior generalization performance of the proposed IF-GEPSVM and IF-IGEPSVM models when compared to the existing baseline models, both with and without label noise. Our experimental results, supported by rigorous statistical analyses, confirm the superior generalization abilities of the proposed IF-GEPSVM and IF-IGEPSVM models over the baseline models. Furthermore, we implement the proposed IF-GEPSVM and IF-IGEPSVM models on the USPS recognition dataset, yielding promising results that underscore the models’ effectiveness in practical and real-world applications. The source code of the proposed IF-GEPSVM and IF-IGEPSVM models are available at https://github.com/mtanveer1/IF-GEPSVM.

MFILS: Tri-Selection via Convex and Nonconvex Regularizations.

In many real-world applications, data are represented by multiple instances and simultaneously associated with multiple labels. These data are always redundant and generally contaminated by different noise levels. As a result, several machine learning models fail to achieve good classification and find an optimal mapping. Feature selection, instance selection, and label selection are three effective dimensionality reduction techniques. Nevertheless, the literature was limited to feature and/or instance selection but has, to some extent, neglected label selection, which also plays an essential role in the preprocessing step, as label noises can adversely affect the performance of the underlying learning algorithms. In this article, we propose a novel framework termed multilabel Feature Instance Label Selection (mFILS) that simultaneously performs feature, instance, and label selections in both convex and nonconvex scenarios. To the best of our knowledge, this article offers, for the first time ever, a study using the triple and simultaneous selection of features, instances, and labels based on convex and nonconvex penalties in a multilabel scenario. Experimental results are built on some known benchmark datasets to validate the effectiveness of the proposed mFILS.

Lazy Multi-Label Classification algorithms based on Non-Parametric Predictive Inference

Multi-Label Classification (MLC) extends standard classification in the sense that an instance might belong to multiple labels simultaneously. Many lazy approaches to MLC have been proposed so far. The majority of them, to classify an instance, use statistical estimators from the neighboring instances based on classical probability theory. In this work, we propose lazy algorithms for MLC that employ the Non-Parametric Predictive Inference Model (NPI-M) for the statistical estimators based on the neighboring instances. It is shown that our proposed lazy MLC algorithms are more suitable to tackle the class-imbalance problem that usually arises in MLC, especially when data contain label noise. This issue is corroborated via an exhaustive experimental analysis.

Broad Learning System under Label Noise: A Novel Reweighting Framework with Logarithm Kernel and Mixture Autoencoder.

The Broad Learning System (BLS) has demonstrated strong performance across a variety of problems. However, BLS based on the Minimum Mean Square Error (MMSE) criterion is highly sensitive to label noise. To enhance the robustness of BLS in environments with label noise, a function called Logarithm Kernel (LK) is designed to reweight the samples for outputting weights during the training of BLS in order to construct a Logarithm Kernel-based BLS (L-BLS) in this paper. Additionally, for image databases with numerous features, a Mixture Autoencoder (MAE) is designed to construct more representative feature nodes of BLS in complex label noise environments. For the MAE, two corresponding versions of BLS, MAEBLS, and L-MAEBLS were also developed. The extensive experiments validate the robustness and effectiveness of the proposed L-BLS, and MAE can provide more representative feature nodes for the corresponding version of BLS.

Addressing label noise for electronic health records: insights from computer vision for tabular data

The analysis of extensive electronic health records (EHR) datasets often calls for automated solutions, with machine learning (ML) techniques, including deep learning (DL), taking a lead role. One common task involves categorizing EHR data into predefined groups. However, the vulnerability of EHRs to noise and errors stemming from data collection processes, as well as potential human labeling errors, poses a significant risk. This risk is particularly prominent during the training of DL models, where the possibility of overfitting to noisy labels can have serious repercussions in healthcare. Despite the well-documented existence of label noise in EHR data, few studies have tackled this challenge within the EHR domain. Our work addresses this gap by adapting computer vision (CV) algorithms to mitigate the impact of label noise in DL models trained on EHR data. Notably, it remains uncertain whether CV methods, when applied to the EHR domain, will prove effective, given the substantial divergence between the two domains. We present empirical evidence demonstrating that these methods, whether used individually or in combination, can substantially enhance model performance when applied to EHR data, especially in the presence of noisy/incorrect labels. We validate our methods and underscore their practical utility in real-world EHR data, specifically in the context of COVID-19 diagnosis. Our study highlights the effectiveness of CV methods in the EHR domain, making a valuable contribution to the advancement of healthcare analytics and research.

Contrastive Learning Joint Regularization for Pathological Image Classification with Noisy Labels

The annotation of pathological images often introduces label noise, which can lead to overfitting and notably degrade performance. Recent studies have attempted to address this by filtering samples based on the memorization effects of DNNs. However, these methods often require prior knowledge of the noise rate or a small, clean validation subset, which is extremely difficult to obtain in real medical diagnosis processes. To reduce the effect of noisy labels, we propose a novel training strategy that enhances noise robustness without prior conditions. Specifically, our approach includes self-supervised regularization to encourage the model to focus more on the intrinsic connections between images rather than relying solely on labels. Additionally, we employ a historical prediction penalty module to ensure consistency between successive predictions, thereby slowing down the model’s shift from memorizing clean labels to memorizing noisy labels. Furthermore, we design an adaptive separation module to perform implicit sample selection and flip the labels of noisy samples identified by this module and mitigate the impact of noisy labels. Comprehensive evaluations of synthetic and real pathological datasets with varied noise levels confirm that our method outperforms state-of-the-art methods. Notably, our noise handling process does not require any prior conditions. Our method achieves highly competitive performance in low-noise scenarios which aligns with current pathological image noise situations, showcasing its potential for practical clinical applications.

Deep learning with noisy labels in medical prediction problems: a scoping review.

Medical research faces substantial challenges from noisy labels attributed to factors like inter-expert variability and machine-extracted labels. Despite this, the adoption of label noise management remains limited, and label noise is largely ignored. To this end, there is a critical need to conduct a scoping review focusing on the problem space. This scoping review aims to comprehensively review label noise management in deep learning-based medical prediction problems, which includes label noise detection, label noise handling, and evaluation. Research involving label uncertainty is also included. Our scoping review follows the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched 4 databases, including PubMed, IEEE Xplore, Google Scholar, and Semantic Scholar. Our search terms include "noisy label AND medical/healthcare/clinical," "uncertainty AND medical/healthcare/clinical," and "noise AND medical/healthcare/clinical." A total of 60 papers met inclusion criteria between 2016 and 2023. A series of practical questions in medical research are investigated. These include the sources of label noise, the impact of label noise, the detection of label noise, label noise handling techniques, and their evaluation. Categorization of both label noise detection methods and handling techniques are provided. From a methodological perspective, we observe that the medical community has been up to date with the broader deep-learning community, given that most techniques have been evaluated on medical data. We recommend considering label noise as a standard element in medical research, even if it is not dedicated to handling noisy labels. Initial experiments can start with easy-to-implement methods, such as noise-robust loss functions, weighting, and curriculum learning.

Proxy-based robust deep metric learning in the presence of label noise

Real-world datasets contain label noise data that can deteriorate the performance of a deep learning model. Cleaning annotations manually requires substantial efforts from experts and is not practical in large datasets. Therefore, many methods are proposed to enhance the robustness of deep models in the presence of label noise. However, these methods are primarily developed for classification tasks and cannot be directly applied to Deep Metric Learning (DML) applications. To bridge this gap, we present a proxy-based deep metric learning method to detect wrong labels through the estimation of the posterior distribution for observed and true labels. Specifically, we model the distribution of the observed annotations as a mixture of two components: one component represents the clean label distribution and the other denotes the noisy labels. Besides, we present an EM-like algorithm that precisely identifies label noise data jointly with the similarity learning method. We also exploit information of the identified noisy labeled data by utilizing state-of-the-art semi-supervised learning (SSL) techniques. The results of the extensive experiments on datasets with real or synthetic label noise indicate that our method consistently outperforms the state-of-the-art methods by a large margin. Moreover, the proposed method accurately detects noisy labeled data and generates correct pseudo labels for them after only a few epochs.