Label Noise Research Articles

Reverse stable diffusion: What prompt was used to generate this image?

Text-to-image diffusion models have recently attracted the interest of many researchers, and inverting the diffusion process can play an important role in better understanding the generative process and how to engineer prompts in order to obtain the desired images. To this end, we study the task of predicting the prompt embedding given an image generated by a generative diffusion model. We consider a series of white-box and black-box models (with and without access to the weights of the diffusion network) to deal with the proposed task. We propose a novel learning framework comprising a joint prompt regression and multi-label vocabulary classification objective that generates improved prompts. To further improve our method, we employ a curriculum learning procedure that promotes the learning of image-prompt pairs with lower labeling noise (i.e. that are better aligned). We conduct experiments on the DiffusionDB data set, predicting text prompts from images generated by Stable Diffusion. In addition, we make an interesting discovery: training a diffusion model on the prompt generation task can make the model generate images that are much better aligned with the input prompts, when the model is directly reused for text-to-image generation. Our code is publicly available for download at https://github.com/CroitoruAlin/Reverse-Stable-Diffusion.

Enhanced Plant Leaf Classification over a Large Number of Classes Using Machine Learning

In botany and agriculture, classifying leaves is a crucial process that yields vital information for studies on biodiversity, ecological studies, and the identification of plant species. The Cope Leaf Dataset offers a comprehensive collection of leaf images from various plant species, enabling the development and evaluation of advanced classification algorithms. This study presents a robust methodology for classifying leaf images within the Cope Leaf Dataset by enhancing the feature extraction and selection process. Cope Leaf Dataset has 99 classes and 64 features with 1584 records. Features are extracted based on the margin, texture, and shape of the leaves. It is challenging to classify a large number of labels because of class imbalance, feature complexity, overfitting, and label noise. Our approach combines advanced feature selection techniques with robust preprocessing methods, including normalization, imputation, and noise reduction. By systematically integrating these techniques, we aim to reduce dimensionality, eliminate irrelevant or redundant features, and improve data quality. Increasing accuracy in classification, especially when dealing with large datasets and many classes, involves a combination of data preprocessing, model selection, regularization techniques, and fine-tuning. The results indicate that the Multilayer Perception algorithm gives 89.48%, the Naïve Bayes Classifier gives 89.63%, Convolutional Neural Networks has 88.72%, and the Hoeffding Tree algorithm gives 89.92% accuracy for the classification of 99 label plant leaf classification problems.

SPORT: A Subgraph Perspective on Graph Classification with Label Noise

Graph neural networks (GNNs) have achieved great success recently on graph classification tasks using supervised end-to-end training. Unfortunately, extensive noisy graph labels could exist in the real world because of the complicated processes of manual graph data annotations, which may significantly degrade the performance of GNNs. Therefore, we investigate the problem of graph classification with label noise, which is demanding because of the complex graph representation learning issue and serious memorization of noisy samples. In this work, we present a novel approach called S ubgra p h Set Netw or k with Sample Selection and Consis t ency Learning (SPORT) for this problem. To release the overfitting of GNNs, SPORT proposes to characterize each graph as a set of subgraphs generated by certain predefined stratagems, which can be viewed as samples from its underlying semantic distribution in graph space. Then we develop an equivariant network to encode the subgraph set with the consideration of the symmetry group. To further release the influences of noisy examples, we leverage the predictions of subgraphs to measure the likelihood of a sample being clean or noisy, followed by effective label updating. In addition, we propose a joint loss to advance the model generalizability by introducing consistency regularization. Comprehensive experiments on a wide range of graph classification datasets demonstrate the effectiveness of our SPORT. Specifically, SPORT outperforms the most competing baseline by up to 6.4%.

Separating Noisy Samples From Tail Classes for Long-Tailed Image Classification With Label Noise.

Most existing methods that cope with noisy labels usually assume that the classwise data distributions are well balanced. They are difficult to deal with the practical scenarios where training samples have imbalanced distributions, since they are not able to differentiate noisy samples from tail classes' clean samples. This article makes an early effort to tackle the image classification task in which the provided labels are noisy and have a long-tailed distribution. To deal with this problem, we propose a new learning paradigm which can screen out noisy samples by matching between inferences on weak and strong data augmentations. A leave-noise-out regularization (LNOR) is further introduced to eliminate the effect of the recognized noisy samples. Besides, we propose a prediction penalty based on the online classwise confidence levels to avoid the bias toward easy classes which are dominated by head classes. Extensive experiments on five datasets including CIFAR-10, CIFAR-100, MNIST, FashionMNIST, and Clothing1M demonstrate that the proposed method outperforms the existing algorithms for learning with long-tailed distribution and label noise.

Information mismatch in PHH3-assisted mitosis annotation leads to interpretation shifts in H&E slide analysis

The count of mitotic figures (MFs) observed in hematoxylin and eosin (H&E)-stained slides is an important prognostic marker, as it is a measure for tumor cell proliferation. However, the identification of MFs has a known low inter-rater agreement. In a computer-aided setting, deep learning algorithms can help to mitigate this, but they require large amounts of annotated data for training and validation. Furthermore, label noise introduced during the annotation process may impede the algorithms’ performance. Unlike H&E, where identification of MFs is based mainly on morphological features, the mitosis-specific antibody phospho-histone H3 (PHH3) specifically highlights MFs. Counting MFs on slides stained against PHH3 leads to higher agreement among raters and has therefore recently been used as a ground truth for the annotation of MFs in H&E. However, as PHH3 facilitates the recognition of cells indistinguishable from H&E staining alone, the use of this ground truth could potentially introduce an interpretation shift and even label noise into the H&E-related dataset, impacting model performance. This study analyzes the impact of PHH3-assisted MF annotation on inter-rater reliability and object level agreement through an extensive multi-rater experiment. Subsequently, MF detectors, including a novel dual-stain detector, were evaluated on the resulting datasets to investigate the influence of PHH3-assisted labeling on the models’ performance. We found that the annotators’ object-level agreement significantly increased when using PHH3-assisted labeling (F1: 0.53 to 0.74). However, this enhancement in label consistency did not translate to improved performance for H&E-based detectors, neither during the training phase nor the evaluation phase. Conversely, the dual-stain detector was able to benefit from the higher consistency. This reveals an information mismatch between the H&E and PHH3-stained images as the cause of this effect, which renders PHH3-assisted annotations not well-aligned for use with H&E-based detectors. Based on our findings, we propose an improved PHH3-assisted labeling procedure.

Efficient ensemble uncertainty estimation in Gaussian processes regression

Abstract Reliable uncertainty measures are required when using data-based machine learning interatomic potentials (MLIPs) for atomistic simulations. In this work, we propose for sparse Gaussian process regression (GPR) type MLIPs a stochastic uncertainty measure akin to the query-by-committee approach often used in conjunction with neural network based MLIPs. The uncertainty measure is coined ‘label noise’ ensemble uncertainty as it emerges from adding noise to the energy labels in the training data. We find that this method of calculating an ensemble uncertainty is as well calibrated as the one obtained from the closed-form expression for the posterior variance when the sparse GPR is treated as a projected process. Comparing the two methods, our proposed ensemble uncertainty is, however, faster to evaluate than the closed-form expression. Finally, we demonstrate that the proposed uncertainty measure acts better to support a Bayesian search for optimal structure of Au20 clusters.

Sample feature enhancement model based on heterogeneous graph representation learning for few-shot relation classification

Few-Shot Relation Classification (FSRC) aims to predict novel relationships by learning from limited samples. Graph Neural Network (GNN) approaches for FSRC constructs data as graphs, effectively capturing sample features through graph representation learning. However, they often face several challenges: 1) They tend to neglect the interactions between samples from different support sets and overlook the implicit noise in labels, leading to sub-optimal sample feature generation. 2) They struggle to deeply mine the diverse semantic information present in FSRC data. 3) Over-smoothing and overfitting limit the model's depth and adversely affect overall performance. To address these issues, we propose a Sample Representation Enhancement model based on Heterogeneous Graph Neural Network (SRE-HGNN) for FSRC. This method leverages inter-sample and inter-class associations (i.e., label mutual attention) to effectively fuse features and generate more expressive sample representations. Edge-heterogeneous GNNs are employed to enhance sample features by capturing heterogeneous information of varying depths through different edge attentions. Additionally, we introduce an attention-based neighbor node culling method, enabling the model to stack higher levels and extract deeper inter-sample associations, thereby improving performance. Finally, experiments are conducted for the FSRC task, and SRE-HGNN achieves an average accuracy improvement of 1.84% and 1.02% across two public datasets.

Domain adaptation for semantic segmentation of road scenes via two-stage alignment of traffic elements

Unsupervised domain adaptation has been used to reduce the domain shift, which would improve the performance of semantic segmentation on unlabeled real-world data. However, existing methodologies fall short in effectively addressing the domain shift issue prevalent in traffic scenarios, leading to less than satisfactory segmentation results. In this paper, we propose a novel domain adaptation method for semantic segmentation via unsupervised alignment of traffic elements. Firstly, we introduce a two-stage self-training framework that leverages a blended set of training samples to enhance the training process. In the first stage, we leverage generated mixup training samples as inputs within our two-stage self-training framework and have developed corresponding loss functions for both the source and target domains to direct the training process. Then, the alignment modules for dynamic and static traffic elements are designed to achieve accurate matching between the source and the target domain images. The cosine similarity maximization is applied to the alignment of dynamic traffic elements, while the prototype learning is utilized for the static traffic elements. Additionally, we present a new technique for reducing noise in pseudo labels by constructing thresholds that adjust to each class. Meanwhile, we formulate the associated target domain loss function for vacant pseudo label pixels. The experimental results demonstrate that the proposed method is superior to the existing methods on five different domain adaptation tasks, which is more applicable to semantic segmentation of road scenes.

Multi-instance learning in the presence of positive and unlabeled bags

Multi-instance learning is a classic algorithm within the weakly supervised learning framework. Most previous approaches to multi-instance learning have typically assumed that every bag comes with its own label. However, in real-world applications, obtaining fully-labeled bags can be a challenging and resource-intensive task, primarily due to the substantial time and labor required for labeling each instance. Fortunately, collecting bags with only positive and unlabeled instances is a more feasible option. In this paper, we aim to learn an efficient binary classifier using positive and unlabeled bags. To address the absence of negative bags, we adopt a novel strategy that combines label noise learning and multi-instance kernels. Our approach involves a comprehensive analysis of empirical risk minimization in the presence of label noise, allowing us to decompose the loss function for negative bags with false negative labels into two distinct parts. Importantly, only the second part of this loss function is affected by the presence of noisy labels. To mitigate the influence of noisy labels on the negative bags, we propose the Multi-Instance Kernel-based Centroid Estimation (MIKCE) technique. MIKCE is a versatile method that can be employed with various types of data, including linear and nonlinear cases. Additionally, we provide an upper bound on the generalization error, ensuring the convergence of the MIKCE algorithm. Finally, we conduct numerical experiments on 31 public datasets to empirically validate the effectiveness of the MIKCE approach.

EEG-based emotional valence and emotion regulation classification: a data-centric and explainable approach

Emotion classification using electroencephalographic (EEG) data is a challenging task in the field of Artificial Intelligence. While many researchers have focused on finding the best model or feature extraction technique to achieve optimal results, few have attempted to select the best methodological steps for working with the dataset. In this study, we applied two different theoretical approaches based on the noise of the dataset: curriculum learning and confident learning. Curriculum learning involves presenting training examples to the model in a specific order, starting with easier examples and gradually increasing in difficulty. This approach has been shown to improve model performance. Confident learning is a method for identifying and correcting label errors in datasets. By identifying and correcting these errors, confident learning can improve the performance of machine learning models trained on noisy datasets. We then applied the Integrated Gradient technique in order to assess the explainability of each model. Our aim was to explore the impact of different models and methods on emotion classification performance using EEG data. We collected and used an EEG dataset in which participants rated the emotional valence of positive and negative pictures while performing an emotion regulation (ER) task, comparing a control condition (Look) with two ER strategies: cognitive reappraisal and expressive suppression. We performed a multilabel classification to identify emotional neutrality or polarization of emotional valence (both positive and negative) rated by participants and the emotion regulation strategy adopted during the task. We compared the performance of models trained on three datasets selected based on label noise and evaluated their suitability for this task. Our results suggest different patterns based on the architecture used for feature importance, highlighting both advantages and criticisms.

Mises-Fisher similarity-based boosted additive angular margin loss for breast cancer classification

To enhance the accuracy of breast cancer diagnosis, current practices rely on biopsies and microscopic examinations. However, this approach is known for being time-consuming, tedious, and costly. While convolutional neural networks (CNNs) have shown promise for their efficiency and high accuracy, training them effectively becomes challenging in real-world learning scenarios such as class imbalance, small-scale datasets, and label noises. Angular margin-based softmax losses, which concentrate on the angle between features and classifiers embedded in cosine similarity at the classification layer, aim to regulate feature representation learning. Nevertheless, the cosine similarity’s lack of a heavy tail impedes its ability to compactly regulate intra-class feature distribution, limiting generalization performance. Moreover, these losses are constrained to target classes when margin penalties are applied, which may not always optimize effectiveness. Addressing these hurdles, we introduce an innovative approach termed MF-BAM (Mises-Fisher Similarity-based Boosted Additive Angular Margin Loss), which extends beyond traditional cosine similarity and is anchored in the von Mises-Fisher distribution. MF-BAM not only penalizes the angle between deep features and their corresponding target class weights but also considers angles between deep features and weights associated with non-target classes. Through extensive experimentation on the BreaKHis dataset, MF-BAM achieves outstanding accuracies of 99.92%, 99.96%, 100.00%, and 98.05% for magnification levels of ×40, ×100, ×200, and ×400, respectively. Furthermore, additional experiments conducted on the BACH dataset for breast cancer classification, as well as on the LFW and YTF datasets for face recognition, affirm the generalization capability of our proposed loss function.

Sound quality characteristics of reciprocating hermetic refrigeration compressors

In recent years, sound quality (SQ) has become an increasingly important factor in the home appliance industry, enhancing the customer experience and comfort. Manufacturers that prioritize SQ in their designs can gain a competitive advantage in the market and increase customer satisfaction. The compressor is one of the main sources of noise in a refrigerator. While manufacturers typically provide refrigerator sound quality and noise labels, research that specifically evaluates the compressor sound quality metrics at a component level is not common. Therefore, this study aims to isolate the compressor from the refrigerator to investigate SQ metrics for hermetic refrigeration compressors. To achieve this, compressors with different capacities are operated and recorded at different working speeds for different thermal conditions. Objective parameters are calculated and tested using time domain signals and analyzed with spectral analysis methods. In addition to the objective parameters, subjective results are obtained through jury tests consisting of 75 people of different genders and age ranges. By considering objective parameters of sound quality as input and sound quality evaluation values as output, a regression model is established. It is found that Loudness, Sharpness, and Degree of Modulation play an important role in compressor sound quality evaluations consider.

Beyond clean data: Exploring the effects of label noise on object detection performance

In recent years, the growth of large-scale datasets has significantly propelled the progress of deep learning applications. Yet, annotating these datasets remains a labor-intensive endeavor, pushing the reliance on cost-effective but less specialized data collection methods and internet data sources. This often results in noisy and inaccurate labels, compromising data quality. Traditional machine learning models assume clean data, but real-world datasets often exhibit significant label noise. This paper examines the impact of such noise on object detection performance, a pivotal aspect of computer vision. We analyze the influence of noisy labels using three renowned object detection frameworks: YOLOv5, Faster R-CNN, and the recent YOLOv8, alongside established datasets: MS COCO, VOC, and ExDARK. Additionally, experiments with the UVM dataset explore domain-specific tasks in dense object scenarios. Two new metrics — Model Health and Detection Capability — were introduced to evaluate the results. Findings indicate that models maintain over 80% of their health (a 20% decline in mAP from the baseline) with up to 40% label corruption. However, Detection Capability deteriorates more sharply under the same conditions. The research also employs the D-RISE method for model explainability, highlighting crucial image regions affecting detection outcomes. Despite the noise, critical detection areas in models remain similar to those in clean data up to the 40% corruption level, as verified by similarity metrics. This study underscores the resilience of object detection models to label noise and provides insights into maintaining performance amidst data quality challenges.

Pseudo-label refinement via hierarchical contrastive learning for source-free unsupervised domain adaptation

Source-free unsupervised domain adaptation aims to adapt a source model to an unlabeled target domain without accessing the source data due to privacy considerations. Existing works mainly solve the problem by self-training methods and representation learning. However, these works typically learn the representation on a single semantic level and barely exploit the rich hierarchical semantic information to obtain clear decision boundaries, which makes it hard for these methods to achieve satisfactory generalization performance. In this paper, we propose a novel hierarchical contrastive domain adaptation algorithm that exploits self-supervised contrastive learning on both fine-grained instances and coarse-grained cluster semantics. On the one hand, we propose an adaptive prototype pseudo-labeling strategy to obtain much more reliable labels. On the other hand, we propose hierarchical contrastive representation learning on both fine-grained instance-wise level and coarse-grained cluster level to reduce the negative effect of label noise and stabilize the whole training procedure. Extensive experiments are conducted on primary unsupervised domain adaptation benchmark datasets, and the results demonstrate the effectiveness of the proposed method.

Radiomic Prediction of Parkinson’s Disease Deep Brain Stimulation Surgery Outcomes using Quantitative Susceptibility Mapping and Label Noise Compensation

Radiomic Prediction of Parkinson’s Disease Deep Brain Stimulation Surgery Outcomes using Quantitative Susceptibility Mapping and Label Noise Compensation

A New Joint Training Method for Facial Expression Recognition with Inconsistently Annotated and Imbalanced Data

Facial expression recognition (FER) plays a crucial role in various applications, including human–computer interaction and affective computing. However, the joint training of an FER network with multiple datasets is a promising strategy to enhance its performance. Nevertheless, widespread annotation inconsistencies and class imbalances among FER datasets pose significant challenges to this approach. This paper proposes a new multi-dataset joint training method, Sample Selection and Paired Augmentation Joint Training (SSPA-JT), to address these challenges. SSPA-JT models annotation inconsistency as a label noise problem and selects clean samples from auxiliary datasets to expand the overall dataset size while maintaining consistent annotation standards. Additionally, a dynamic matching algorithm is developed to pair clean samples of the tail class with noisy samples, which enriches the tail classes with diverse background information. Experimental results demonstrate that SSPA-JT achieved superior or comparable performance compared with the existing methods by addressing both annotation inconsistencies and class imbalance during multi-dataset joint training. It achieved state-of-the-art performance on RAF-DB and CAER-S datasets with accuracies of 92.44% and 98.22%, respectively, reflecting improvements of 0.2% and 3.65% over existing methods.

Clustering Aided Weakly Supervised Training to Detect Anomalous Events in Surveillance Videos.

Formulating learning systems for the detection of real-world anomalous events using only video-level labels is a challenging task mainly due to the presence of noisy labels as well as the rare occurrence of anomalous events in the training data. We propose a weakly supervised anomaly detection system that has multiple contributions including a random batch selection mechanism to reduce interbatch correlation and a normalcy suppression block (NSB) which learns to minimize anomaly scores over normal regions of a video by utilizing the overall information available in a training batch. In addition, a clustering loss block (CLB) is proposed to mitigate the label noise and to improve the representation learning for the anomalous and normal regions. This block encourages the backbone network to produce two distinct feature clusters representing normal and anomalous events. An extensive analysis of the proposed approach is provided using three popular anomaly detection datasets including UCF-Crime, ShanghaiTech, and UCSD Ped2. The experiments demonstrate the superior anomaly detection capability of our approach.

A performance-based mental workload identification method for special vehicle crews

Detecting the mental workload state of armored vehicle crews is of great significance for monitoring the driving state of the crew and improving comprehensive combat effectiveness. In this manuscript, we propose a performance-based mental workload identification method and carry out experimental validation to improve the accuracy of crew mental workload identification and realize the effective classification of mental workload. Based on the virtual simulation system of the special vehicle crew task, this manuscript selects 20 subjects for the mental workload experiment of special vehicle crews. The experiment collected NASA-TLX scale, EEG, eye-tracking data, and performance data. The results show that the mental workload of the crews fluctuates in the segmented tasks of complex operations in typical scenes of special vehicles. The method of determining mental workload using NASA-TLX generates label noise in classification, which is not suitable for special vehicle tasks. Performance-based mental workload identification method is able to recognize fluctuations in the crew's mental workload during segmented tasks. Performance-based and NASA-TXL-based methods were classified using linear discriminant analysis. The results show that the accuracy of the method based on performance is improved by 15.72 %. This manuscript found the NASA-TXL scale is not suitable for the complex tasks of special vehicles, and proposed a performance-based identification method that can help to categorize the mental workload states of special vehicle crews.

Critical insights into data curation and label noise for accurate prediction of aerobic biodegradability of organic chemicals.

The focus of this work is to enhance state-of-the-art Machine Learning (ML) models that can predict the aerobic biodegradability of organic chemicals through a data-centric approach. To do that, an already existing dataset that was previously used to train ML models was analyzed for mismatching chemical identifiers and data leakage between test and training set and the detected errors were corrected. Chemicals with high variance between study results were removed and an XGBoost was trained on the dataset. Despite extensive data curation, only marginal improvement was achieved in the classification model's performance. This was attributed to three potential reasons: (1) a significant number of data labels were noisy, (2) the features could not sufficiently represent the chemicals, and/or (3) the model struggled to learn and generalize effectively. All three potential reasons were examined and point (1) seemed to be the most decisive one that prevented the model from generating more accurate results. Removing data points with possibly noisy labels by performing label noise filtering using two other predictive models increased the classification model's balanced accuracy from 80.9% to 94.2%. The new classifier is therefore better than any previously developed classification model for ready biodegradation. The examination of the key characteristics (molecular weight of the substances, proportion of halogens present and distribution of degradation labels) and the applicability domain indicate that no/not a large share of difficult-to-learn substances has been removed in the label noise filtering, meaning that the final model is still very robust.

Intuitionistic fuzzy least squares MLTSVM for noisy label data using label-specific features and local label correlation

The multilabel twin support vector machine (MLTSVM) has been widely applied to multilabel classification fields because of its excellent classification performance, but it has the following three disadvantages. (a) In practical classification tasks, due to human negligence or some objective factors, there are many data with noise labels. Noisy label data have a negative effect on generating classification hyperplanes. However, MLTSVM cannot effectively reduce the negative effects on noisy label data. (b) Each label has its own specific features. However, for each label, MLTSVM can only use the same feature representation for classification. Therefore, it cannot obtain the optimal classification results. (c) In multilabel classification problems, there is a general correlation among the labels. However, when MLTSVM constructs a classification hyperplane for each label, it ignores the label correlation information. To address the above disadvantages, in this paper, a novel algorithm is proposed the called intuitionistic fuzzy least squares MLTSVM for noisy label data using label-specific features and local label correlation (SFLLC-IFLSMLTSVM). First, SFLLC-IFLSMLTSVM constructs an intuitionistic fuzzy set for each label; second, SFLLC-IFLSMLTSVM exploits the membership and nonmembership degrees in the intuitionistic fuzzy set to filter out the samples with noise labels and then performs clustering analysis to extract the label-specific features and the structure information of samples; finally, SFLLC-IFLSMLTSVM mines the local label correlation information in the process of constructing classification hyperplanes and exploits the membership and nonmembership degrees to suppress the negative effects of noise labels. Extensive comparative experiments on multilabel datasets show that SFLLC-IFLSMLTSVM can efficiently handle the multilabel classification problems with noise labels.