Multiple Noisy Labels Research Articles

Three-way decision-based label integration for crowdsourcing

In crowdsourcing learning, label integration is often used to infer instances’ integrated labels from their multiple noisy labels. However, almost all existing label integration algorithms apply the same strategy to infer different instances’ integrated labels, which limits their performance. This paper argues that different instances should enjoy different label integration strategies alone. Thanks to the three-way decision theory, a three-way decision-based label integration (TDLI) algorithm is proposed. In TDLI, we at first evaluate the label qualities of each instance and its K-nearest neighbors, and then utilize them to divide the whole crowdsourced datasets into three disjoint subsets, called positive set, boundary set and negative set, respectively. For each instance in the positive set, we directly apply the simplest majority voting (MV) to infer its integrated label. For each instance in the boundary set, we absorb its K-nearest neighbors’ multiple noisy labels to infer its integrated label by the weighted MV. For each instance in the negative set, we merge the positive and boundary sets to train a classifier to infer its integrated label by fusing its own multiple noisy label distribution and the predicted label distribution. Extensive experiments demonstrate that TDLI distinctly outperforms all the other existing label integration algorithms used to compare.

Instance redistribution-based label integration for crowdsourcing

Crowdsourcing provides an inexpensive solution to employ crowd workers labeling instances, and hence each instance is labeled with a multiple noisy label set instead of its true label. Label integration aims to infer a true label from its multiple noisy label set. However, existing fine-tuned methods increase the model complexity while enhancing the performance of label integration. Therefore, it is a challenge to search for a simple but efficient label integration method. Meeting this challenge, this paper proposes instance redistribution-based label integration (IRLI), a novel method for label integration. Firstly, from the label view, we estimate the probability of an instance belonging to each class based on its multiple noisy label set and obtain a multiple noisy label distribution vector. Secondly, from the attribute view, we reestimate the probability of an instance belonging to each class based on its attribute set and obtain an instance redistribution vector. Finally, we combine the multiple noisy label distribution vector and the instance redistribution vector of each instance to infer its integrated label. Comprehensive experimental results on both simulated and real-world crowdsourced datasets validate the superiority of IRLI.

Label noise correction for crowdsourcing using dynamic resampling

Crowdsourcing provides a cost-effective labeling solution for the acquisition of labeled training samples for machine learning by employing workers on the Internet. A common approach to improving the label quality is to employ a truth inference method to infer integrated labels for samples from their multiple noisy labels obtained from different crowd workers. Although the quality of integrated labels is significantly improved compared with that of the original noisy ones, it still cannot completely eliminate the noises inevitably existing in the integrated labels. To further improve the label quality, this paper proposes a novel label noise correction method for crowdsourcing based on dynamic resampling (DRNC). DRNC first divides the dataset with inferred labels into a clean set and a noisy set through a filter. According to a certain proportion, the clean set and the noisy set are resampled to train multiple heterogeneous classifiers, which form an ensemble classifier. Then, the dataset is divided by the ensemble classifier into a new sub-noisy set and a sub-clean set. The whole process repeats multiple rounds, generating multiple sub-clean sets. Finally, these sub-clean sets are used to train classifiers, which jointly correct the wrong labels in the dataset by voting. Experimental results on 25 simulated and 4 real-world datasets consistently show that the proposed DRNC averagely improves the quality of labels as well as the quality of learned models in the range of 1 to 10 percentage points, compared with four state-of-the-art crowdsourcing noise correction methods.

Worker similarity-based noise correction for crowdsourcing

Crowdsourcing offers a cost-effective way to obtain multiple noisy labels for each instance by employing multiple crowd workers. Then label integration is used to infer its integrated label. Despite the effectiveness of label integration algorithms, there always remains a certain degree of noise in the integrated labels. Thus noise correction algorithms have been proposed to reduce the impact of noise. However, almost all existing noise correction algorithms only focus on individual workers but ignore the correlations among workers. In this paper, we argue that similar workers have similar annotating skills and tend to be consistent in annotating same or similar instances. Based on this premise, we propose a novel noise correction algorithm called worker similarity-based noise correction (WSNC). At first, WSNC exploits the annotating information of similar workers on similar instances to estimate the quality of each label annotated by each worker on each instance. Then, WSNC re-infers the integrated label of each instance based on the qualities of its multiple noisy labels. Finally, WSNC considers the instance whose re-inferred integrated label differs from its original integrated label as a noise instance and further corrects it. The extensive experiments on a large number of simulated and three real-world crowdsourced datasets verify the effectiveness of WSNC.

Dual-View Noise Correction for Crowdsourcing

In crowdsourcing scenarios, each instance obtains a multiple noisy label set from different crowd workers on the Internet and then gets its integrated label via label integration. Although label integration algorithms are often effective, a certain level of noise still remains in the integrated labels. To reduce the impact of noise on label quality, many noise correction algorithms have been proposed in recent years. However, most of them are hard to fully utilize the joint information of the original attribute view and multiple noisy label view. Motivated by multi-view learning, in this paper, we propose a dual-view noise correction (DVNC) algorithm. Benefiting from the complementary and consensus principle, DVNC can fully utilize the joint information of two views and thus enhance the effect of noise correction. In DVNC, we at first construct the original attribute view and multiple noisy label view for each instance and respectively train a classifier on each view. Then, we use the trained two classifiers to filter each noise instance and thus obtain a clean set and noise set. Finally, we train a classifier on the clean set to correct each instance in the noise set via reclassifying it as the class with the maximum posterior probability. The experimental results on simulated and real-world datasets indicate that DVNC significantly outperforms all the other noise correction algorithms used for comparison.

Three-way decision-based noise correction for crowdsourcing

In crowdsourcing learning, a certain degree of label noise still exists in the integrated labels, which are obtained by label integration algorithms. In order to decrease the impact of label noise, many scholars focus on noise correction algorithms. Existing noise correction algorithms usually divide a data set with integrated labels into a clean set and a noise set, directly. However, there are certain noise instances in the clean set and clean instances in the noise set. To reduce the interference from noise instances and improve the utilization of clean instances, inspired by the three-way decision theory, we propose a three-way decision-based noise correction (TDNC) algorithm. Firstly, we calculate the compactness of each instance's multiple noisy label set and estimate the label certainty of each instance's integrated label. Secondly, we divide a data set with integrated labels into three disjoint sets, called positive set, boundary set and negative set, respectively. Thirdly, we further divide and add the instances in the boundary set into the positive set and the negative set according to the nearest neighbor classification. Finally, we train two heterogeneous classifiers on the new positive set to correct the noise instances in the new negative set using a consensus voting strategy. The experimental results on 34 simulated and two real-world data sets consistently show that TDNC significantly outperforms the existing state-of-the-art noise correction algorithms in terms of the noise ratio.

Neighborhood Weighted Voting-Based Noise Correction for Crowdsourcing

In crowdsourcing scenarios, we can obtain each instance’s multiple noisy labels set from different crowd workers and then use a ground truth inference algorithm to infer its integrated label. Despite the effectiveness of ground truth inference algorithms, a certain level of noise still remains in the integrated labels. To reduce the impact of noise, many noise correction algorithms have been proposed in recent years. To the best of our knowledge, however, nearly all existing noise correction algorithms only exploit each instance’s own multiple noisy label sets but ignore the multiple noisy label sets of its neighbors. Here neighbors refer to the nearest instances found in the feature space based on the distance metric learning. In this article, we propose neighborhood weighted voting-based noise correction (NWVNC). In NWVNC, we at first take advantage of the multiple noisy label sets of each instance’s neighbors (including itself) to estimate the probability that it belongs to its integrated label. Then, we use the estimated probability to identify and filter noise instances and thus obtain a clean set and a noise set. Finally, we train three heterogeneous classifiers on the clean set and correct the noise instances by the consensus voting of three trained classifiers. The experimental results on 34 simulated and two real-world crowdsourced datasets show that NWVNC significantly outperforms all the other state-of-the-art noise correction algorithms used for comparison.

Not so weak PICO: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation.

The aim of this study was to test the feasibility of PICO (participants, interventions, comparators, outcomes) entity extraction using weak supervision and natural language processing. We re-purpose more than 127 medical and nonmedical ontologies and expert-generated rules to obtain multiple noisy labels for PICO entities in the evidence-based medicine (EBM)-PICO corpus. These noisy labels are aggregated using simple majority voting and generative modeling to get consensus labels. The resulting probabilistic labels are used as weak signals to train a weakly supervised (WS) discriminative model and observe performance changes. We explore mistakes in the EBM-PICO that could have led to inaccurate evaluation of previous automation methods. In total, 4081 randomized clinical trials were weakly labeled to train the WS models and compared against full supervision. The models were separately trained for PICO entities and evaluated on the EBM-PICO test set. A WS approach combining ontologies and expert-generated rules outperformed full supervision for the participant entity by 1.71% macro-F1. Error analysis on the EBM-PICO subset revealed 18-23% erroneous token classifications. Automatic PICO entity extraction accelerates the writing of clinical systematic reviews that commonly use PICO information to filter health evidence. However, PICO extends to more entities-PICOS (S-study type and design), PICOC (C-context), and PICOT (T-timeframe) for which labelled datasets are unavailable. In such cases, the ability to use weak supervision overcomes the expensive annotation bottleneck. We show the feasibility of WS PICO entity extraction using freely available ontologies and heuristics without manually annotated data. Weak supervision has encouraging performance compared to full supervision but requires careful design to outperform it.

Label confidence-based noise correction for crowdsourcing

In crowdsourcing scenarios, each instance obtains multiple noisy labels from different crowd workers and then gets its integrated label via a label aggregation method. In spite of the effectiveness of label aggregation methods, a certain level of noise still remains in the integrated labels. To solve this problem, several noise correction methods have been proposed in recent years. However, to our best knowledge, these methods seldom take the label confidence of each instance into account. Therefore, we propose a label confidence-based noise correction (LCNC) method. At first, LCNC calculates the label confidence of each instance using its multiple noisy labels to filter all instances and obtains an original clean set and noise set. Second, LCNC builds multiple random trees on the original clean set to recalculate the label confidence of each instance using its predicted labels generated by these random trees and then refilters all instances to obtain a new clean set and noise set. Finally, LCNC builds two heterogeneous classifiers on the new clean set to correct the noise instances in the new noise set according to a consensus voting strategy. The experimental results on 34 simulated and two real-world crowdsourced datasets show that LCNC significantly outperforms all the other state-of-the-art noise correction methods used for comparison.

Learning From Crowds With Multiple Noisy Label Distribution Propagation.

Crowdsourcing services provide a fast, efficient, and cost-effective way to obtain large labeled data for supervised learning. Unfortunately, the quality of crowdsourced labels cannot satisfy the standards of practical applications. Ground-truth inference, simply called label integration, designs proper aggregation methods to infer the unknown true label of each instance (sample) from the multiple noisy label set provided by ordinary crowd labelers (workers). However, nearly all existing label integration methods focus solely on the multiple noisy label set per individual instance while totally ignoring the intercorrelation among multiple noisy label sets of different instances. To solve this problem, a multiple noisy label distribution propagation (MNLDP) method is proposed in this article. MNLDP at first estimates the multiple noisy label distribution of each instance from its multiple noisy label set and then propagates its multiple noisy label distribution to its nearest neighbors. Consequently, each instance absorbs a fraction of the multiple noisy label distributions from its nearest neighbors and yet simultaneously maintains a fraction of its own original multiple noisy label distribution. Empirical studies on a collection of an artificial dataset, six simulated UCI datasets, and three real-world crowdsourced datasets show that MNLDP outperforms all other existing state-of-the-art label integration methods in terms of the integration accuracy and classification accuracy.

Instance difficulty-based noise correction for crowdsourcing

Crowdsourcing offers an efficient way to obtain a multiple noisy label set of each instance from different crowd workers and then label integration algorithms are used to infer its integrated label. In spite of the effectiveness of label integration algorithms, there always exists a certain degree of noise in the integrated labels, and thus noise correction algorithms have been proposed to reduce the effect of noise. However, existing noise correction algorithms seldom consider the effect of instance difficulty on noise correction. In this paper, we argue that the greater the difficulty of an instance, the fewer crowd workers can label it correctly, and the more likely the instance is a noise instance. Based on this premise, we propose a simple but very effective noise correction algorithm called instance difficulty-based noise correction (IDNC). In IDNC, we at first propose two methods to measure the difficulty of each instance. Then, we use the proposed two methods to filter the noise instances to obtain a clean set and a noise set. Finally, we build two different classifiers on the clean set to correct the noise instances in the noise set via the consensus voting. The extensive experiments on both simulated and real-world crowdsourced datasets validated the effectiveness and efficiency of our proposed IDNC.

Noise-Robust Learning from Multiple Unsupervised Sources of Inferred Labels

Deep Neural Networks (DNNs) generally require large-scale datasets for training. Since manually obtaining clean labels for large datasets is extremely expensive, unsupervised models based on domain-specific heuristics can be used to efficiently infer the labels for such datasets. However, the labels from such inferred sources are typically noisy, which could easily mislead and lessen the generalizability of DNNs. Most approaches proposed in the literature to address this problem assume the label noise depends only on the true class of an instance (i.e., class-conditional noise). However, this assumption is not realistic for the inferred labels as they are typically inferred based on the features of the instances. The few recent attempts to model such instance-dependent (i.e., feature-dependent) noise require auxiliary information about the label noise (e.g., noise rates or clean samples). This work proposes a theoretically motivated framework to correct label noise in the presence of multiple labels inferred from unsupervised models. The framework consists of two modules: (1) MULTI-IDNC, a novel approach to correct label noise that is instance-dependent yet not class-conditional; (2) MULTI-CCNC, which extends an existing class-conditional noise-robust approach to yield improved class-conditional noise correction using multiple noisy label sources. We conduct experiments using nine real-world datasets for three different classification tasks (images, text and graph nodes). Our results show that our approach achieves notable improvements (e.g., 6.4% in accuracy) against state-of-the-art baselines while dealing with both instance-dependent and class-conditional noise in inferred label sources.

Label augmented and weighted majority voting for crowdsourcing

Crowdsourcing provides an efficient way to obtain multiple noisy labels from different crowd workers for each unlabeled instance. Label integration methods are designed to infer the unknown true label of each instance from its multiple noisy label set. We argue that when the label quality is higher than random classification, the more the number of labels, the better the performance of label integration methods. However, in real-world crowdsourcing scenarios, each instance cannot obtain enough labels for saving costs. To solve this problem, this paper proposes a novel label integration method called label augmented and weighted majority voting (LAWMV). At first, LAWMV uses the K-nearest neighbors (KNN) algorithm to find each instance’s K-nearest neighbors (including itself) and merges their multiple noisy label sets to obtain its augmented multiple noisy label set. Then, the labels from different neighbors are weighted by the distances and the label similarities between each instance and its neighbors. Finally, the integrated label of each instance is inferred by weighted majority voting (MV). The experimental results on 34 simulated and two real-world crowdsourced datasets show that LAWMV significantly outperforms all the other state-of-the-art label integration methods.

Label distribution‐based noise correction for multiclass crowdsourcing

In crowdsourcing scenarios, we can often obtain each instance's multiple noisy labels from different crowd workers and then use a label integration method to infer its integrated label. In spite of the effectiveness of label integration methods, a certain level of label noise still exists in integrated labels. To reduce the impact of label noise, noise correction has attracted much attention from researchers, and therefore a certain number of noise correction methods were proposed in recent years. Among them, Between-class Margin-based Noise Correction (BMNC) has demonstrated remarkable denoising performance. However, BMNC can only handle binary classification tasks. For multiclass classification tasks, we propose an effective but very simple noise correction method in this paper. We refer to our proposed method as Label Distribution-based Noise Correction (LDNC). At first, LDNC transforms each instance's multiple noisy labels into a label distribution. Then, LDNC uses the margin between the first- and second-largest label probabilities in the label distribution to identify and filter each possible noise instance and thus obtains a clean set and a noise set. Finally, LDNC builds a classifier on the clean set to relabel all instances in the noise set. The experimental results on 16 simulated and one real-world multiclass crowdsourced data sets show that LDNC significantly outperforms all the other existing state-of-the-art noise correction methods.

Improving data and model quality in crowdsourcing using co-training-based noise correction

Crowdsourcing makes it much faster and cheaper to obtain labels for a large amount of data used in supervised learning. In the crowdsourcing scenario, an integrated label is inferred from a multiple noisy label set for each instance using ground truth inference algorithms, which is called label integration. However, a certain level of label noise remains in the integrated dataset, which degrades the performance of the models trained on it. To the best of our knowledge, existing label noise correction algorithms only use the original attribute space and do not use the information contained in the multiple noisy label sets for building models. To solve these problems, we propose a novel integrated label noise correction algorithm called co-training-based noise correction (CTNC). In CTNC, the weight is first calculated from the information provided by the multiple noisy label set for each instance. Subsequently, a label noise filter is used to identify noisy instances; a clean set and a noisy set are thus obtained. Another attribute view of each instance in both the clean and noisy sets is then generated by the classifiers trained on the original attribute view of the clean set. Finally, a co-training framework is used to train two classifiers to relabel the integrated instances. The performance on 34 simulated datasets and 2 real-world datasets demonstrates that our proposed CTNC outperforms all state-of-the-art label noise correction algorithms used for comparison.

Differential evolution-based weighted soft majority voting for crowdsourcing

Crowdsourcing has attracted considerable attention in recent years. A large amount of labeled data can be obtained efficiently and cheaply from the crowdsourcing platform. Obviously, the labeling quality of crowd workers directly influences the quality of the labeled data. Although a small amount of label integration strategies have recently noticed the differences in the quality of crowd workers labeling different instances, which just utilize the statistical characteristics of multiple noisy labels to estimate the quality of crowd workers and thus are rough and sub-optimal. In addition, they can only deal with binary classification problems, which restricts the practical applications of crowdsourcing. To simultaneously solve these two issues, we propose three differential evolution-based weighted soft majority voting strategies for multi-class classification. In our proposed strategies, we exploit a differential evolution (DE) algorithm to estimate the quality of crowd workers labeling different instances by minimizing the Error, Gini and Entropy of weighted multiple noisy labels. Extensive experimental results on simulated and real-world datasets show that our proposed strategies significantly outperform all the other existing state-of-the-art label integration strategies.

Quantifying Parkinson’s disease motor severity under uncertainty using MDS-UPDRS videos

Parkinson’s disease (PD) is a brain disorder that primarily affects motor function, leading to slow movement, tremor, and stiffness, as well as postural instability and difficulty with walking/balance. The severity of PD motor impairments is clinically assessed by part III of the Movement Disorder Society Unified Parkinson’s Disease Rating Scale (MDS-UPDRS), a universally-accepted rating scale. However, experts often disagree on the exact scoring of individuals. In the presence of label noise, training a machine learning model using only scores from a single rater may introduce bias, while training models with multiple noisy ratings is a challenging task due to the inter-rater variabilities. In this paper, we introduce an ordinal focal neural network to estimate the MDS-UPDRS scores from input videos, to leverage the ordinal nature of MDS-UPDRS scores and combat class imbalance. To handle multiple noisy labels per exam, the training of the network is regularized via rater confusion estimation (RCE), which encodes the rating habits and skills of raters via a confusion matrix. We apply our pipeline to estimate MDS-UPDRS test scores from their video recordings including gait (with multiple Raters, R=3) and finger tapping scores (single rater). On a sizable clinical dataset for the gait test (N=55), we obtained a classification accuracy of 72% with majority vote as ground-truth, and an accuracy of ∼84% of our model predicting at least one of the raters’ scores. Our work demonstrates how computer-assisted technologies can be used to track patients and their motor impairments, even when there is uncertainty in the clinical ratings. The latest version of the code will be available at https://github.com/mlu355/PD-Motor-Severity-Estimation.

Improving crowd labeling using Stackelberg models

Crowdsourcing systems provide an easy means of acquiring labeled training data for supervised learning. However, the labels provided by non-expert crowd workers (labelers) often appear low quality. In order to solve this problem, in practice each sample always obtains a multiple noisy label set from multiple different labelers, then ground truth inference algorithms are employed to obtain integrated labels of samples. So ground truth inference methods directly determine the label quality of samples. In this paper, we propose a novel label integration method based on game theory. We assume that there is an adversary in crowdsourcing system who intentionally provides incorrect integrated labels. We model the interaction between the data miner and the adversary as a Stackelberg game in which one player (the data miner) controls the predictive model whereas another (the adversary) tries to choose the integrated labels which would be most harmful for the current classifier. On this basis, we transform the label integration problem into a repeated Stackelberg model. We call our method Stackelberg label inference (SLI). SLI does not need to estimate the quality of labelers, and avoids the chicken-egg problem that can lead to poor result. Moreover, because SLI has little involvement of multiple noisy label sets on the noise data set, it is not very sensitive to the number of labelers. SLI shows better performance when the number of labelers is relatively small. In term of both label quality and model quality, the experimental results show that SLI is superior to the other state-of-the-art ground truth inference methods used to compare.

JMLNet: Joint Multi-Label Learning Network for Weakly Supervised Semantic Segmentation in Aerial Images

Weakly supervised semantic segmentation in aerial images has attracted growing research attention due to the significant saving in annotation cost. Most of the current approaches are based on one specific pseudo label. These methods easily overfit the wrongly labeled pixels from noisy label and limit the performance and generalization of the segmentation model. To tackle these problems, we propose a novel joint multi-label learning network (JMLNet) to help the model learn common knowledge from multiple noisy labels and prevent the model from overfitting one specific label. Our combination strategy of multiple proposals is that we regard them all as ground truth and propose three new multi-label losses to use the multi-label guide segmentation model in the training process. JMLNet also contains two methods to generate high-quality proposals, which further improve the performance of the segmentation task. First we propose a detection-based GradCAM (GradCAMD) to generate segmentation proposals from object detectors. Then we use GradCAMD to adjust the GrabCut algorithm and generate segmentation proposals (GrabCutC). We report the state-of-the-art results on the semantic segmentation task of iSAID and mapping challenge dataset when training with bounding boxes annotations.

Improving data and model quality in crowdsourcing using cross-entropy-based noise correction

Crowdsourcing services provide a fast, efficient, and cost-effective approach to obtaining labeled data, particularly for human-like tasks. In a crowdsourcing scenario, after ground truth inference methods have been employed to obtain integrated instance labels, label noise remains present in the integrated labels. Label noise handling techniques can then be implemented to mitigate the effects of this noise. In this study, we propose a Cross-Entropy-based Noise Correction (CENC) method for crowdsourcing. CENC uses the entropies of the label distributions generated from multiple noisy label sets to filter noisy instances. It then exploits the cross-entropies between each possible true class probability distribution and each predicted class probability distribution to rectify the noisy instances. Using both simulated benchmark data and real-world crowdsourced data, we show that CENC outperforms all other existing state-of-the-art noise correction methods.