Noisy Labels Research Articles

Searching to Exploit Memorization Effect in Deep Learning With Noisy Labels.

Sample selection approaches are popular in robust learning from noisy labels. However, how to control the selection process properly so that deep networks can benefit from the memorization effect is a hard problem. In this paper, motivated by the success of automated machine learning (AutoML), we propose to control the selection process by bi-level optimization. Specifically, we parameterize the selection process by exploiting the general patterns of the memorization effect in the upper-level, and then update these parameters using predicting accuracy obtained from model training in the lower-level. We further introduce semi-supervised learning algorithms to utiilize noisy-labeled data as unlabeled data. To solve the bi-level optimization problem efficiently, we consider more information from the validation curvature by the Newton method and cubic regularization method. We provide convergence analysis for both optimization methods. Results show that while both methods can converge to an (approximately) stationary point, the cubic regularization method can find better local optimal than the Newton method with less time. Experiments on both benchmark and real-world data sets demonstrate that the proposed searching method can lead to significant improvements upon existing methods. Compared with existing AutoML approaches, our method is much more efficient on finding a good selection schedule.

Overcoming Noisy Labels and Non-IID Data in Edge Federated Learning

Overcoming Noisy Labels and Non-IID Data in Edge Federated Learning

SCNet: semi-supervised and contrastive learning against noisy labels with two selection strategy approach

The deep learning-driven artificial intelligence diagnostic model heavily relies on high-quality and detailed annotated data during algorithm training. However, this process is susceptible to interference from noisy label information. To enhance the model's robustness and prevent the adverse effects of noisy labels, we propose a contrastive learning-based approach for noisy label image classification. Specifically, we designed three components: a mixed feature embedding module that takes mixed augmented images as input, a momentum update mechanism to explore abstract distributed feature representations, and two selection strategies. We incorporated a flexible pseudo-label boosting strategy, refining supervised information for noisy data with pseudo-labels based on initial classification predictions. By measuring the similarity between classification distributions, we effectively select more reliable pairs of confidence, thereby reducing the impact of noisy labels on contrastive learning. Furthermore, we employed a noise-robust loss function to ensure that samples with correct labels dominate the learning process. To validate the model's effectiveness, we conducted experiments on our collected data SAID (Sensitive Audio Information Detection), the CIFAR-10 and CIFAR-100 datasets, simulating scenarios with noisy label conditions.

Target Structure Learning Framework for Unsupervised Multi-Class Domain Adaptation

Unsupervised multi-class domain adaptation (multi-class UDA) has recently been proposed to fill the gap between empirically practical methods for multi-class classification and well-founded theory with the setting of binary classification. Nevertheless, the multi-class UDA methods use model predictions to characterize the disagreement of multi-class scoring hypotheses, which is used to optimize the divergence between domain distributions. Such self-training manner may bring inaccurate model predictions, which would damage the target structure due to the absence of labels of the target domain, leading to sub-optimal performance. On the other hand, this disagreement between multi-class scoring hypotheses does not involve the relationships among all of the multiple classes. It causes that multi-class UDA cannot properly connect the advanced practical UDA methods that consider class-conditional distribution alignment. Thus, we propose to exploit the target structure information and then incorporate it into multi-class UDA to achieve class-conditional distribution alignment. We theoretically and experimentally explain the importance of accurate target structure information to reduce the expected error on the target domain. Notably, our method achieves state-of-the-art results on three commonly-used benchmarks with different scales. In addition, using the target structure information, we propose a variant to cope with noisy open-world source domains such as noisy labels and out-of-distribution samples, enhancing the robustness of our method. The source code is available at https://github.com/jingzhengli/Multi_Class_UDA .

A novel binary hashing for agricultural scenery classification

In this research, we present PerceptHashing, a technique designed to categorize million-scale agricultural scenic images by incorporating human gaze shifting paths (GSPs) into a hashing framework. For each agricultural image, we identify visually and semantically significant object patches, such as fields, crops, and water bodies. These patches are linked to form a graphlet, establishing a network of spatially adjacent patches, and a GSP is then extracted using an active learning algorithm. The GSP reflects the distribution of human gaze across different regions of each agricultural scene, typically involving fewer than 12 regions, as validated by cross-validation. We then design a binary hashing framework that effectively leverages the semantics encoded in these GSPs. This framework integrates three key elements: (i) refinement of noisy labels, (ii) incorporation of deep image-level agricultural semantics, and (iii) updates to the adaptive data graph. The resulting hash codes from each GSP are converted into a kernelized visual descriptor for classification. To evaluate the influence of GSPs on agricultural image classification both qualitatively and quantitatively, we conducted an extensive user study comparing GSPs from typical observers with those from Alzheimer’s patients. The results demonstrate that: (1) the classification accuracy of 1.22 million agricultural aerial images using our approach is significantly higher than those processed by other methods, and (2) GSPs from 33 Alzheimer’s patients differ markedly from those of 37 normal observers, leading to notable differences in classification accuracy (accnormal=0.694\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$acc_{normal}=0.694$$\\end{document} versus accpatient=0.322\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$acc_{patient}=0.322$$\\end{document}).

SPORT: A Subgraph Perspective on Graph Classification with Label Noise

Graph neural networks (GNNs) have achieved great success recently on graph classification tasks using supervised end-to-end training. Unfortunately, extensive noisy graph labels could exist in the real world because of the complicated processes of manual graph data annotations, which may significantly degrade the performance of GNNs. Therefore, we investigate the problem of graph classification with label noise, which is demanding because of the complex graph representation learning issue and serious memorization of noisy samples. In this work, we present a novel approach called S ubgra p h Set Netw or k with Sample Selection and Consis t ency Learning (SPORT) for this problem. To release the overfitting of GNNs, SPORT proposes to characterize each graph as a set of subgraphs generated by certain predefined stratagems, which can be viewed as samples from its underlying semantic distribution in graph space. Then we develop an equivariant network to encode the subgraph set with the consideration of the symmetry group. To further release the influences of noisy examples, we leverage the predictions of subgraphs to measure the likelihood of a sample being clean or noisy, followed by effective label updating. In addition, we propose a joint loss to advance the model generalizability by introducing consistency regularization. Comprehensive experiments on a wide range of graph classification datasets demonstrate the effectiveness of our SPORT. Specifically, SPORT outperforms the most competing baseline by up to 6.4%.

Patient-specific visual neglect severity estimation for stroke patients with neglect using EEG.

We aim to assess the severity of spatial neglect through detailing patients' field of view (FOV) using EEG. Spatial neglect, a prevalent neurological syndrome in stroke patients, typically results from unilateral brain injuries, leading to inattention to the contralesional space. Commonly used Neglect detection methods like the Behavioral Inattention Test - Conventional (BIT-C) lack the capability to assess the full extent and severity of neglect. Although the Catherine Bergego Scale (CBS) provides valuable clinical information, it does not detail the specific field of view affected in neglect patients. Building on our previously developed EEG-based Brain-Computer Interface (BCI) system, AREEN (AR-guided EEG-based Neglect Detection, Assessment, and Rehabilitation System), we aim to map neglect severity across a patient's field of view. We have demonstrated that AREEN can assess neglect severity in a patient-agnostic manner. However, its effectiveness in patient-specific scenarios, which is crucial for creating a generalizable plug-and-play system, remains unexplored. This paper introduces a novel EEG-based combined Spatio-Temporal Network (ESTNet) that processes both time and frequency domain data to capture essential frequency band information associated with spatial neglect. We also propose a field of view correction system using Bayesian fusion, leveraging AREEN's recorded response times for enhanced accuracy by addressing noisy labels within the dataset. Extensive testing of ESTNet on our proprietary dataset has demonstrated its superiority over benchmark methods, achieving 79.62% accuracy, 76.71% sensitivity, and 86.36% specificity. Additionally, we provide saliency maps to enhance model explainability and establish clinical correlations. These findings underscore ESTNet's potential combined with Bayesian fusion-based FOV correction as an effective tool for generalized neglect assessment in clinical setting.

ANNE: Adaptive Nearest Neighbours and Eigenvector-based sample selection for robust learning with noisy labels

An important stage of most state-of-the-art (SOTA) noisy-label learning methods consists of a sample selection procedure that classifies samples from the noisy-label training set into noisy-label or clean-label subsets. The process of sample selection typically consists of one of the two approaches: loss-based sampling, where high-loss samples are considered to have noisy labels, or feature-based sampling, where samples from the same class tend to cluster together in the feature space and noisy-label samples are identified as anomalies within those clusters. Empirically, loss-based sampling is robust to a wide range of noise rates, while feature-based sampling tends to work effectively in particular scenarios, e.g., the filtering of noisy instances via their eigenvectors (FINE) sampling exhibits greater robustness in scenarios with low noise rates, and the K nearest neighbour (KNN) sampling mitigates better high noise-rate problems. This paper introduces the Adaptive Nearest Neighbours and Eigenvector-based (ANNE) sample selection methodology, a novel approach that integrates loss-based sampling with the feature-based sampling methods FINE and Adaptive KNN to optimize performance across a wide range of noise rate scenarios. ANNE achieves this integration by first partitioning the training set into high-loss and low-loss sub-groups using loss-based sampling. Subsequently, within the low-loss subset, sample selection is performed using FINE, while the high-loss subset employs Adaptive KNN for effective sample selection. We integrate ANNE into the noisy-label learning state of the art (SOTA) method SSR+, and test it on CIFAR-10/-100 (with symmetric, asymmetric and instance-dependent noise), Webvision and ANIMAL-10, where our method shows better accuracy than the SOTA in most experiments, with a competitive training time. The code is available at https://github.com/filipe-research/anne.

Separating Noisy Samples From Tail Classes for Long-Tailed Image Classification With Label Noise.

Most existing methods that cope with noisy labels usually assume that the classwise data distributions are well balanced. They are difficult to deal with the practical scenarios where training samples have imbalanced distributions, since they are not able to differentiate noisy samples from tail classes' clean samples. This article makes an early effort to tackle the image classification task in which the provided labels are noisy and have a long-tailed distribution. To deal with this problem, we propose a new learning paradigm which can screen out noisy samples by matching between inferences on weak and strong data augmentations. A leave-noise-out regularization (LNOR) is further introduced to eliminate the effect of the recognized noisy samples. Besides, we propose a prediction penalty based on the online classwise confidence levels to avoid the bias toward easy classes which are dominated by head classes. Extensive experiments on five datasets including CIFAR-10, CIFAR-100, MNIST, FashionMNIST, and Clothing1M demonstrate that the proposed method outperforms the existing algorithms for learning with long-tailed distribution and label noise.

Fine-grained recognition algorithm of crop pests based on cross-layer bilinear aggregation and multi-task learning

Fine-grained recognition of crop pests is a crucial concern in the field of agriculture, as the accuracy of recognition and generalization ability directly affect the yield and quality of crops. Aiming at the characteristics of crop pests with a wide variety of species, small inter-class and large intra-class differences in external morphology, as well as the problems of uneven sample distribution and noisy labels in fine-grained image datasets under complex environments, we propose a fine-grained recognition model of crop pests (MT-MACLBPHSNet) based on cross-layer bilinear aggregation and multi-task learning, which consists of three key modules: the backbone network module, the cross-layer bilinear aggregation module, and the multi-task learning module. A new union loss function is designed in the primary task of the multi-task learning module, which is used to alleviate the two problems existing in the model training fine-grained image datasets. The experimental results show that the model effectively balances the model complexity and recognition accuracy in a comparative analysis with several existing excellent network models on the IP102-CP13 dataset, with the recognition accuracy reaching 75.37%, which is 7.06% higher than the Baseline model, and the F1-score reaching 67.06%. Additionally, the generalization of the model is also verified on the IP102-VP16 dataset, and the model outperforms most of the models in terms of recognition accuracy and generalization ability, which can provide an effective reference for fine-grained recognition of crop pests.

Noise-Tolerant Hybrid Prototypical Learning with Noisy Web Data

We focus on the challenging problem of learning an unbiased classifier from a large number of potentially relevant but noisily labeled web images given only a few clean labeled images. This problem is particularly practical because it reduces the expensive annotation costs by utilizing freely accessible web images with noisy labels. Typically, prototypes are representative images or features used to classify or identify other images. However, in the few clean and many noisy scenarios, the class prototype can be severely biased due to the presence of irrelevant noisy images. The resulting prototypes are less compact and discriminative, as previous methods do not take into account the diverse range of images in the noisy web image collections. On the other hand, the relation modeling between noisy and clean images is not learned for the class prototype generation in an end-to-end manner, which results in a suboptimal class prototype. In this article, we introduce a similarity maximization loss named SimNoiPro. Our SimNoiPro first generates noise-tolerant hybrid prototypes composed of clean and noise-tolerant prototypes and then pulls them closer to each other. Our approach considers the diversity of noisy images by explicit division and overcomes the optimization discrepancy issue. This enables better relation modeling between clean and noisy images and helps extract judicious information from the noisy image set. The evaluation results on two extended few-shot classification benchmarks confirm that our SimNoiPro outperforms prior methods in measuring image relations and cleaning noisy data.

Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval.

Deep hashing technology, known for its low-cost storage and rapid retrieval, has become a focal point in cross-modal retrieval research as multimodal data continue to grow. However, existing supervised methods often overlook noisy labels and multiscale features in different modal datasets, leading to higher information entropy in the generated hash codes and features, which reduces retrieval performance. The variation in text annotation information across datasets further increases the information entropy during text feature extraction, resulting in suboptimal outcomes. Consequently, reducing the information entropy in text feature extraction, supplementing text feature information, and enhancing the retrieval efficiency of large-scale media data are critical challenges in cross-modal retrieval research. To tackle these, this paper introduces the Text-Enhanced Graph Attention Hashing for Cross-Modal Retrieval (TEGAH) framework. TEGAH incorporates a deep text feature extraction network and a multiscale label region fusion network to minimize information entropy and optimize feature extraction. Additionally, a Graph-Attention-based modal feature fusion network is designed to efficiently integrate multimodal information, enhance the affinity of the network for different modes, and retain more semantic information. Extensive experiments on three multilabel datasets demonstrate that the TEGAH framework significantly outperforms state-of-the-art cross-modal hashing methods.

Partial Multi-Label Learning via Exploiting Instance and Label Correlations

The goal of partial multi-label learning is to induce a multi-label classifier from partial multi-label data where each instance is annotated with a number of candidate labels but only a subset of them are valid. Many of the existing studies either fail to fully utilize instance and label correlations to eliminate noisy labels or build an oversimplified multi-label classifier, both of which are unfavorable for the improvement of generalization performance. In this article, we put forward a novel model named Pml-ilc to learn a multi-label classifier from partial multi-label data. Specifically, Pml-ilc first encodes instances and labels into a compact semantic space and takes full advantage of instance and label correlations to eliminate noisy labels. Then, it induces a linear mapping from the feature space to the label space while exploiting label-specific features and instance correlations to facilitate the multi-label classifier learning process. Finally, the above two steps are combined into a joint optimization problem and an efficient alternating optimization procedure is developed to find a satisfactory solution. Extensive experiments show that Pml-ilc achieves superior performance on both real-world and synthetic partial multi-label datasets in terms of different evaluation metrics.

Perturbation defense ultra high-speed weak target recognition

Ultra high-speed target recognition in complex electromagnetic environments is a critical and fundamental machine perception issue. It is difficult to ensure privacy protection and intelligent confrontation with centralized training in many practical situations. In this paper, an efficient ultra high-speed target recognition approach, named InVision, for robot systems is proposed using deep trustworthy federated learning (DTFL). InVision is accurate, safe, fast, and robust. Particularly, geometric component transformer (GCT) is presented to significantly promote neural element's complex representation description ability. And an ambiguity-aware cooperative learning (AACL) scheme is developed to relieve the noisy label problem. Moreover, decentralized federated training (DFT) is designed to mitigate the intractable privacy protection problem, to efficiently search for similarities and reduce the representation redundancy. Furthermore, to promote the running speed of the system in real-world environments, a lightweight deep architecture, called Mobile-XB, is developed. Extensive quantitative and qualitative experiments are carried out, and the results demonstrate that InVision greatly outperforms the outstanding comparison methods, establishing efficient connections and extraction, and providing security guarantees.

Multi-instance learning in the presence of positive and unlabeled bags

Multi-instance learning is a classic algorithm within the weakly supervised learning framework. Most previous approaches to multi-instance learning have typically assumed that every bag comes with its own label. However, in real-world applications, obtaining fully-labeled bags can be a challenging and resource-intensive task, primarily due to the substantial time and labor required for labeling each instance. Fortunately, collecting bags with only positive and unlabeled instances is a more feasible option. In this paper, we aim to learn an efficient binary classifier using positive and unlabeled bags. To address the absence of negative bags, we adopt a novel strategy that combines label noise learning and multi-instance kernels. Our approach involves a comprehensive analysis of empirical risk minimization in the presence of label noise, allowing us to decompose the loss function for negative bags with false negative labels into two distinct parts. Importantly, only the second part of this loss function is affected by the presence of noisy labels. To mitigate the influence of noisy labels on the negative bags, we propose the Multi-Instance Kernel-based Centroid Estimation (MIKCE) technique. MIKCE is a versatile method that can be employed with various types of data, including linear and nonlinear cases. Additionally, we provide an upper bound on the generalization error, ensuring the convergence of the MIKCE algorithm. Finally, we conduct numerical experiments on 31 public datasets to empirically validate the effectiveness of the MIKCE approach.

Dual structure-aware image filterings for semi-supervised medical image segmentation

Semi-supervised image segmentation has attracted great attention recently. The key is how to leverage unlabeled images in the training process. Most methods maintain consistent predictions of the unlabeled images under variations (e.g., adding noise/perturbations, or creating alternative versions) in the image and/or model level. In most image-level variation, medical images often have prior structure information, which has not been well explored. In this paper, we propose novel dual structure-aware image filterings (DSAIF) as the image-level variations for semi-supervised medical image segmentation. Motivated by connected filtering that simplifies image via filtering in structure-aware tree-based image representation, we resort to the dual contrast invariant Max-tree and Min-tree representation. Specifically, we propose a novel connected filtering that removes topologically equivalent nodes (i.e. connected components) having no siblings in the Max/Min-tree. This results in two filtered images preserving topologically critical structure. Applying the proposed DSAIF to mutually supervised networks decreases the consensus of their erroneous predictions on unlabeled images. This helps to alleviate the confirmation bias issue of overfitting to noisy pseudo labels of unlabeled images, and thus effectively improves the segmentation performance. Extensive experimental results on three benchmark datasets demonstrate that the proposed method significantly/consistently outperforms some state-of-the-art methods. The source codes will be publicly available.

Experts still needed: boosting long-term android malware detection with active learning

The continuous evolution of cyber threats imposes a critical challenge to malware detection systems, so operational detection solutions in real-world settings must keep up-to-date malware knowledge databases. Machine learning-based solutions are not exempt from this requirement as handling concept drift constitutes the primary building block for keeping high detection performance in the long term. However, maintaining non-stationary malware detection models is highly demanding due to the high cost of labeling. This study applies several active learning-based approaches for maintaining a non-stationary model for Android malware detection in a 7-year-long time frame and conducts a comprehensive analysis to understand the impact of feature space selection, different data balancing techniques, and timestamping methods, utilized for locating the instances along the historical timeline, on the model’s detection performance over time. The detection accuracy and labeling costs are compared with various baselines. Additionally, the study investigates the resilience of such models against noisy labeling, a common problem in production environments due to unintentional expert errors and adversarial attacks. This research fills a significant gap in the literature by conducting a comprehensive analysis of active learning approaches to address concept drift in non-stationary settings established for mobile malware detection.

Beyond clean data: Exploring the effects of label noise on object detection performance

In recent years, the growth of large-scale datasets has significantly propelled the progress of deep learning applications. Yet, annotating these datasets remains a labor-intensive endeavor, pushing the reliance on cost-effective but less specialized data collection methods and internet data sources. This often results in noisy and inaccurate labels, compromising data quality. Traditional machine learning models assume clean data, but real-world datasets often exhibit significant label noise. This paper examines the impact of such noise on object detection performance, a pivotal aspect of computer vision. We analyze the influence of noisy labels using three renowned object detection frameworks: YOLOv5, Faster R-CNN, and the recent YOLOv8, alongside established datasets: MS COCO, VOC, and ExDARK. Additionally, experiments with the UVM dataset explore domain-specific tasks in dense object scenarios. Two new metrics — Model Health and Detection Capability — were introduced to evaluate the results. Findings indicate that models maintain over 80% of their health (a 20% decline in mAP from the baseline) with up to 40% label corruption. However, Detection Capability deteriorates more sharply under the same conditions. The research also employs the D-RISE method for model explainability, highlighting crucial image regions affecting detection outcomes. Despite the noise, critical detection areas in models remain similar to those in clean data up to the 40% corruption level, as verified by similarity metrics. This study underscores the resilience of object detection models to label noise and provides insights into maintaining performance amidst data quality challenges.

Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videos.

Formulating learning systems for the detection of real-world anomalous events using only video-level labels is a challenging task mainly due to the presence of noisy labels as well as the rare occurrence of anomalous events in the training data. We propose a weakly supervised anomaly detection system that has multiple contributions including a random batch selection mechanism to reduce interbatch correlation and a normalcy suppression block (NSB) which learns to minimize anomaly scores over normal regions of a video by utilizing the overall information available in a training batch. In addition, a clustering loss block (CLB) is proposed to mitigate the label noise and to improve the representation learning for the anomalous and normal regions. This block encourages the backbone network to produce two distinct feature clusters representing normal and anomalous events. An extensive analysis of the proposed approach is provided using three popular anomaly detection datasets including UCF-Crime, ShanghaiTech, and UCSD Ped2. The experiments demonstrate the superior anomaly detection capability of our approach.

Label-noise learning via uncertainty-aware neighborhood sample selection

Existing deep learning methods often require a large amount of high-quality labeled data. Yet, the presence of noisy labels in the real-world training data seriously affects the generalization ability of the model. Sample selection techniques, the current dominant approach to mitigating the effects of noisy labels on models, use the consistency of sample predictions and observed labels to make clean selections. However, these methods rely heavily on the accuracy of the sample predictions and inevitably suffer when the model predictions are unstable. To address these issues, we propose an uncertainty-aware neighborhood sample selection method. Especially, it calibrates for sample prediction by neighbor prediction and reassigns model attention to the selected samples based on sample uncertainty. By alleviating the influence of prediction bias on sample selection and avoiding the occurrence of prediction bias, our proposed method achieves excellent performance in extensive experiments. In particular, we achieved an average of 5% improvement in asymmetric noise scenarios.