Noisy Training Set Research Articles

Knockoffs-SPR: Clean Sample Selection in Learning With Noisy Labels.

A noisy training set usually leads to the degradation of the generalization and robustness of neural networks. In this article, we propose a novel theoretically guaranteed clean sample selection framework for learning with noisy labels. Specifically, we first present a Scalable Penalized Regression (SPR) method, to model the linear relation between network features and one-hot labels. In SPR, the clean data are identified by the zero mean-shift parameters solved in the regression model. We theoretically show that SPR can recover clean data under some conditions. Under general scenarios, the conditions may be no longer satisfied; and some noisy data are falsely selected as clean data. To solve this problem, we propose a data-adaptive method for Scalable Penalized Regression with Knockoff filters (Knockoffs-SPR), which is provable to control the False-Selection-Rate (FSR) in the selected clean data. To improve the efficiency, we further present a split algorithm that divides the whole training set into small pieces that can be solved in parallel to make the framework scalable to large datasets. While Knockoffs-SPR can be regarded as a sample selection module for a standard supervised training pipeline, we further combine it with a semi-supervised algorithm to exploit the support of noisy data as unlabeled data. Experimental results on several benchmark datasets and real-world noisy datasets show the effectiveness of our framework and validate the theoretical results of Knockoffs-SPR.

Uncertainty Quantification of a Machine Learning Model for Identification of Isolated Nonlinearities with Conformal Prediction

Abstract Structural nonlinearities are often spatially localized, such joints and interfaces, localized damage, or isolated connections, in an otherwise linearly behaving system. Quinn and Brink [12] modeled this localized nonlinearity as a deviatoric force component. In other previous work [13], the authors proposed a physics-informed machine learning framework to determine the deviatoric force from measurements obtained only at the boundary of the nonlinear region, assuming a noise-free environment. However, in real experimental applications, the data are expected to contain noise from a variety of sources. In the present work, we explore the sensitivity of the trained network by comparing the network responses when trained on deterministic (“noise-free”) model data and model data with additive noise (“noisy”). As the neural network does not yield a closed-form transformation from the input distribution to the response distribution, we leverage the use of conformal sets to build an illustration of sensitivity. Through the conformal set assumption of exchangeability, we may build a distribution-free prediction interval for both network responses of the clean and noisy training sets. This work will explore the application of conformal sets for uncertainty quantification of a deterministic structure-preserving neural network and its deployment in a structural health monitoring framework to detect deviations from a baseline state based on noisy measurements.

A balanced random learning strategy for CNN based Landsat image segmentation under imbalanced and noisy labels

Landsat image segmentation is important for obtaining large-scale land cover maps. The accuracy of CNN-based Landsat image segmentation highly depends on the quantity and quality of the training samples. However, enough accurate labels for Landsat images are difficult to access. Fortunately, traditional classifier induced segmentation results can be considered as an alternative, although they are noisy and unbalanced to a certain extent. To resist noisy labels and alleviate the impact of imbalanced samples, this paper proposes a confidence interval based balanced random learning strategy. Firstly, a confidence interval-based mask is employed to control the random learning rate of the network from the entire noisy training set. Then, the multi-layer feature maps of CNN are fully utilized to compensate for the information loss in random learning, in which down-sampled labels are used to decrease the uncertainty brought by up-sampling CNN feature maps. In addition, considering the corruption of noisy labels on different classes, a balanced random learning with different confidence levels is performed on each class to further improve the learning ability of CNN. Experimental results on two widely used backbones, namely VGGNet and ResNet, demonstrate that the proposed balanced random learning strategy can effectively improve the performance of CNN under imbalanced and noisy labels, which can be improved by 3.41%.

Multi-proxy feature learning for robust fine-grained visual recognition

Visual representation for fine-grained visual recognition can be learned by mandatorily enforcing all samples of the same category into a uniform representation. This strict training objective performs well under closed-set setting but is not applicable to data in the wild containing noisy annotations and long-tailed distributions, e.g., it may lead to a feature space biased to head categories. This paper tackles this challenge by pursuing a more balanced and discriminative feature space by first retaining intra-class variances to isolate noises, then eliminating intra-class variances to improve the visual recognition performance. We propose the Compact Memory Updater to maintain a memory bank, which memorizes proxy features to represent multiple typical appearances of each category in the training set. The Proxy-based Feature Enhancement hence leverages proxy features to ensure samples of the same category have similar features. Iteratively running those two modules boosts the robustness and discriminative power of the learnt representation, hence facilitates various fine-grained visual recognition tasks including person re-identification (re-id), image classification and retrieval. Extensive experiments on noisy and long-tailed training sets show this Multi-Proxy Feature Learning (MPFL) framework achieves promising performance. For instance on a training set with 90% one-shot categories, MPFL outperforms the recent long-tailed person re-id method LEAP-AF by 16.9% in rank-1 accuracy.

A Credit Card Fraud Model Prediction Method Based on Penalty Factor Optimization AWTadaboost

With the popularity of online payment, how to perform credit card fraud detection more accurately has also become a hot issue. And with the emergence of the adaptive boosting algorithm (Adaboost), credit card fraud detection has started to use this method in large numbers, but the traditional Adaboost is prone to overfitting in the presence of noisy samples. Therefore, in order to alleviate this phenomenon, this paper proposes a new idea: using the number of consecutive sample misclassifications to determine the noisy samples, while constructing a penalty factor to reconstruct the sample weight assignment. Firstly, the theoretical analysis shows that the traditional Adaboost method is overfitting in a noisy training set, which leads to the degradation of classification accuracy. To this end, the penalty factor constructed by the number of consecutive misclassifications of samples is used to reconstruct the sample weight assignment to prevent the classifier from over-focusing on noisy samples, and its reasonableness is demonstrated. Then, by comparing the penalty strength of the three different penalty factors proposed in this paper, a more reasonable penalty factor is selected. Meanwhile, in order to make the constructed model more in line with the actual requirements on training time consumption, the Adaboost algorithm with adaptive weight trimming (AWTAdaboost) is used in this paper, so the penalty factor-based AWTAdaboost (PF_AWTAdaboost) is finally obtained. Finally, PF_AWTAdaboost is experimentally validated against other traditional machine learning algorithms on credit card fraud datasets and other datasets. The results show that the PF_AWTAdaboost method has better performance, including detection accuracy, model recall and robustness, than other methods on the credit card fraud dataset. And the PF_AWTAdaboost method also shows excellent generalization performance on other datasets. From the experimental results, it is shown that the PF_AWTAdaboost algorithm has better classification performance.

Attentive-Adaptive Network for Hyperspectral Images Classification With Noisy Labels

With the development of deep neural networks, hyperpsectral image (HSI) classification systems have achieved a significant improvement. These systems require numerous and accurate labeled hyperspectral data to be adequately trained. However, noisy labels are inherent in real-world hyperspectral systems, resulting in unreliable decisions. To handle noisy labels in hyperpsectral classification, an end-to-end attentive-adaptive network (AAN) is proposed for robust HSI classification training. The goal is to build a classifier with strong generalization capabilities that can be applied to both clean and noisy training sets without explicit noise label pre-treatment. Specifically, a spectral stem network with non-adjacent shortcut is exploited initially to re-distribute the sensitive layers for noisy labels to achieve robust spectral representation. Then, a group-shuffle attention module is proposed to capture the discriminative and robust spatial-spectral features in the presence of noisy labels. Finally, an adaptive noise-robust loss function is developed to fight against noisy labels by learning a parameter to balance the normalized cross entropy (NCE) and reverse cross entropy (RCE). Experimental results on three HSI benchmark datasets with simulated noisy labels demonstrate the effectiveness of AAN on HSI classification.

Parameter-shared variational auto-encoding adversarial network for desert seismic data denoising in Northwest China

One of the key and difficult points in seismic data processing is seismic data denoising. Under the influence of the acquisition environment, the collected seismic data usually have low signal-to-noise ratio (SNR) and low resolution. In desert region of western China, different from other regions, the random noise there has complex characteristics of non-Gaussian, non-stationary and non-linearity. Its main frequency is quite low and its spectral overlap with those of effective signals. Moreover, large amount of data and intelligent requirements make traditional denoising methods encounter difficulties. In addition, the existing deep learning denoising methods also reveal two problems: first, they can only effectively suppress simple seismic noise such as Gaussian noise. But for the case of desert noise, they usually mistakenly judge noise as effective signals and destroy signal characteristic structure. Second, what is more important, in supervised-based methods paired noisy and pure data are usually used as training set. However, there is no noise-free data in actual desert seismic data, which severely limits the processing performance of supervised-based networks. Thus, a new parameter-shared variational auto-encoding adversarial network (PS-VAAN) is proposed for desert seismic data denoising in this paper. This new method includes two encoders, two generators and two discriminators. It can realize domain conversion from noisy data domain to pure data domain. We construct relatively complete unpaired pure and noisy data training sets for training, and design reconstruction loss function and adversarial loss function to optimize network parameters. Moreover, cycle-consistency loss is introduced to make sure two domains mapping into the same space. Compared with the state-of-art denoising methods on synthetic and actual seismic records, the proposed method not only has superior denoising ability, but also has strong feature extraction and fitting ability to recover effective signals with almost no energy loss.

Training redefinition with entropy-based structure set density for supervised hyperspectral imagery classification

ABSTRACT Reliable labelled samples have always played a vital role in the supervised paradigm of hyperspectral imagery (HSI) classification due to the fact that the inclusion of incorrect label information in the training set can seriously degrade the performance of classification methods. Recently, although some inter-class difference-based detection algorithms have been developed to remove mislabelled samples (i.e. noisy labels) in training set, the benefit of contextual information for each sample has not been fully explored yet. In this paper, a training redefinition with entropy-based structure set density (ESSD) method is designed, which consists of following main steps. First, the proposed ESSD method employs an over-segmentation technique to cluster the HSI into many shape-adaptive regions that correspond to sample sets. Then, each sample set is represented with an affine hull (AH) model, which exploits both the similarity and variance of samples within each sample set to adaptively characterize the set. Specifically, considering spectral and spatial weak assumptions among samples in each sample set, the idea of entropy trick-based -nearest neighbour is introduced into each sample set to redefine its structure by removing different class from the sample set. Next, the distance among AH corresponding to each training sample is calculated based on the AH model. Meanwhile, the set-to-set distance is fed to the density peak algorithm to obtain the density of training samples. Finally, a decision-making value is applied to the density of each training sample to cleanse mislabelled samples within noisy training set. Experimental results on real HSI date sets demonstrate the superiority of the proposed method over several well-known training redefinition methods in terms of detection accuracy.

Training deep retrieval models with noisy datasets: Bag exponential loss

Although the CNNs are a very powerful tool for image retrieval, the need of training datasets properly adapted to the application at hand hinders the usefulness of such networks, specially since the datasets need to be free of noise to avoid spoiling the learning process. An ad hoc preprocessing of the dataset to mitigate the noise is a possible solution, but it is usually non-trivial and requires significant human intervention. In this paper, we pave the road for training CNNs for image retrieval with noisy datasets. In particular, we propose a novel Bag Exponential Loss function that, inspired by the Multiple Instance Learning framework, works with bags of matching images instead of single pairs, and allows a dynamical weighting of the relevance of each sample as the training progresses. The formulation of the proposed model is general enough and may serve to other purposes than dealing with noise if parameters are chosen appropriately. Extensive experimental results show the superior performance of the proposed loss with respect to the current state-of-the-art as well as its ability to cope with noisy training sets. Pytorch code available in https://github.com/tmcortes/BELoss

Hyperspectral Classification With Noisy Label Detection via Superpixel-to-Pixel Weighting Distance

Classification is an important technique for remotely sensed hyperspectral image (HSI) exploitation. Often, the presence of wrong (noisy) labels presents a drawback for accurate supervised classification. In this article, we introduce a new framework for noisy label detection that combines a superpixel-to-pixel weighting distance (SPWD) and density peak clustering. The proposed method is able to accurately detect and remove noisy labels in the training set before HSI classification. It considers two weak assumptions when exploiting the spectral-spatial information contained in the HSI: 1) all the pixels in a superpixel belong to the same class and 2) close pixels in spectral space have the same label. The proposed method consists of the following steps. First, a superpixel segmentation step is used to obtain self-adaptive spatial information for each training sample. Then, a metric is utilized to measure the spectral distance information between each superpixel and pixel. Meanwhile, in order to overcome the first weak assumption, we use K nearest neighbors to obtain the closest neighborhoods of pixels around each superpixel, and a Gaussian weight is employed to mitigate the second weak assumption by adapting the original distance information. Next, the noisy labels in the original training set are removed by a density threshold-based decision function. Finally, the support vector machine (SVM) classifier is employed to evaluate the effectiveness of the proposed SPWD detection method in terms of classification accuracy. Experiments performed on several real HSI data sets demonstrate that the method can effectively improve the performance of classifiers trained with noisy training sets in terms of classification accuracy.

Robust Learning of Mislabeled Training Samples for Remote Sensing Image Scene Classification

Label information plays an important role in supervised high-resolution remote sensing (HRRS) image scene classification. However, the labels of a dataset are probably unreliable and may contain “noisy” labels. Focusing on uncertain labels problem, a covariance matrix representation-based noisy label model (CMR-NLD) is designed for HRRS image scene classification. The main steps are as follows. First, a pretrained convolutional neural network model is employed to extract the scene images deep features and a principal component analysis based dimensionality reduction method is applied to the first fully connected layer to reduce the computational complexity. Then, the noisy training set is constructed by randomly selecting samples into a specific class from other classes samples. We use this set to simulate the actual situation of tag noise to simulate the actual situation of label noise. Second, the covariance between noisy training samples is calculated to obtain the corresponding covariance matrix, and the average value of the obtained covariance matrix is calculated by rows. As a feature of the matrix form, it can both enlarge the subtle differences between different classes and reduce the visual differences of scene images from the same semantic classes. Then, a decision threshold is set to realize the detection and removal of noisy labels. Finally, the improved training sample set will be evaluated by a support vector machine classifier to demonstrate the proposed detector's effectiveness. Experimental results indicate that the proposed method indeed shows great improvement in noisy label detection of HRRS image scene classification.

Set2Model networks: Learning discriminatively to learn generative models

We present a new “learning-to-learn”-type approach that enables rapid learning of concepts from small-to-medium sized training sets and is primarily designed for web-initialized image retrieval. At the core of our approach is a deep architecture (a Set2Model network) that maps sets of examples to simple generative probabilistic models such as Gaussians or mixtures of Gaussians in the space of high-dimensional descriptors. The parameters of the embedding into the descriptor space are trained in the end-to-end fashion in the meta-learning stage using a set of training learning problems. The main technical novelty of our approach is the derivation of the backprop process through the mixture model fitting, which makes the likelihood of the resulting models differentiable with respect to the positions of the input descriptors. While the meta-learning process for a Set2Model network is discriminative, a trained Set2Model network performs generative learning of generative models in the descriptor space, which facilitates learning in the cases when no negative examples are available, and whenever the concept being learned is polysemous or represented by noisy training sets. Among other experiments, we demonstrate that these properties allow Set2Model networks to pick visual concepts from the raw outputs of Internet image search engines better than a set of strong baselines.

TopologyNet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions.

Although deep learning approaches have had tremendous success in image, video and audio processing, computer vision, and speech recognition, their applications to three-dimensional (3D) biomolecular structural data sets have been hindered by the geometric and biological complexity. To address this problem we introduce the element-specific persistent homology (ESPH) method. ESPH represents 3D complex geometry by one-dimensional (1D) topological invariants and retains important biological information via a multichannel image-like representation. This representation reveals hidden structure-function relationships in biomolecules. We further integrate ESPH and deep convolutional neural networks to construct a multichannel topological neural network (TopologyNet) for the predictions of protein-ligand binding affinities and protein stability changes upon mutation. To overcome the deep learning limitations from small and noisy training sets, we propose a multi-task multichannel topological convolutional neural network (MM-TCNN). We demonstrate that TopologyNet outperforms the latest methods in the prediction of protein-ligand binding affinities, mutation induced globular protein folding free energy changes, and mutation induced membrane protein folding free energy changes. Availability: weilab.math.msu.edu/TDL/

Multispectral Image Analysis using Decision Trees

Many machine learning algorithms have been used to classify pixels in Landsat imagery. The maximum likelihood classifier is the widely-accepted classifier. Non-parametric methods of classification include neural networks and decision trees. In this research work, we implemented decision trees using the C4.5 algorithm to classify pixels of a scene from Juneau, Alaska area obtained with Landsat 8, Operation Land Imager (OLI). One of the concerns with decision trees is that they are often over fitted with training set data, which yields less accuracy in classifying unknown data. To study the effect of overfitting, we have considered noisy training set data and built decision trees using randomly-selected training samples with variable sample sizes. One of the ways to overcome the overfitting problem is pruning a decision tree. We have generated pruned trees with data sets of various sizes and compared the accuracy obtained with pruned trees to the accuracy obtained with full decision trees. Furthermore, we extracted knowledge regarding classification rules from the pruned tree. To validate the rules, we built a fuzzy inference system (FIS) and reclassified the dataset. In designing the FIS, we used threshold values obtained from extracted rules to define input membership functions and used the extracted rules as the rule-base. The classification results obtained from decision trees and the FIS are evaluated using the overall accuracy obtained from the confusion matrix.

A new hybrid semi-supervised algorithm for text classification with class-based semantics

Vector Space Models (VSM) are commonly used in language processing to represent certain aspects of natural language semantics. Semantics of VSM comes from the distributional hypothesis, which states that words that occur in similar contexts usually have similar meanings. In our previous work, we proposed novel semantic smoothing kernels based on classspecific transformations. These kernels use class-term matrices, which can be considered as a new type of VSM. By using the class as the context, these methods can extract class specific semantics by making use of word distributions both in documents and in different classes. In this study, we adapt two of these semantic classification approaches to build a novel and high performance semi-supervised text classification algorithm. These approaches include Helmholtz principle based calculation of term meanings in the context of classes for initial classification and a supervised term weighting based semantic kernel with Support Vector Machines (SVM) for the final classification model. The approach used in the first phase is especially good at learning with very small datasets, while the approach in the second phase is specifically good at eliminating noise in a relatively large and noisy training sets when building a classification model. Overall, as a semantic semi-supervised learning algorithm, our approach can effectively utilize abundant source of unlabeled instances to improve the classification accuracy significantly especially when the amount of labeled instances are limited.

Knowledge base population using semantic label propagation

Training relation extractors for the purpose of automated knowledge base population requires the availability of sufficient training data. The amount of manual labeling can be significantly reduced by applying distant supervision, which generates training data by aligning large text corpora with existing knowledge bases. This typically results in a highly noisy training set, where many training sentences do not express the intended relation. In this paper, we propose to combine distant supervision with minimal human supervision by annotating features (in particular shortest dependency paths) rather than complete relation instances. Such feature labeling eliminates noise from the initial training set, resulting in a significant increase of precision at the expense of recall. We further improve on this approach by introducing the Semantic Label Propagation (SLP) method, which uses the similarity between low-dimensional representations of candidate training instances to again extend the (filtered) training set in order to increase recall while maintaining high precision. Our strategy is evaluated on an established test collection designed for knowledge base population (KBP) from the TAC KBP English slot filling task. The experimental results show that SLP leads to substantial performance gains when compared to existing approaches while requiring an almost negligible human annotation effort.

Unconfused ultraconservative multiclass algorithms

We tackle the problem of learning linear classifiers from noisy datasets in a multiclass setting. The two-class version of this problem was studied a few years ago where the proposed approaches to combat the noise revolve around a Per-ceptron learning scheme fed with peculiar examples computed through a weighted average of points from the noisy training set. We propose to build upon these approaches and we introduce a new algorithm called UMA (for Unconfused Multiclass additive Algorithm) which may be seen as a generalization to the multiclass setting of the previous approaches. In order to characterize the noise we use the confusion matrix as a multiclass extension of the classification noise studied in the aforemen-tioned literature. Theoretically well-founded, UMA furthermore displays very good empirical noise robustness, as evidenced by numerical simulations conducted on both synthetic and real data.

Enhanced regulatory sequence prediction using gapped k-mer features.

Oligomers of length k, or k-mers, are convenient and widely used features for modeling the properties and functions of DNA and protein sequences. However, k-mers suffer from the inherent limitation that if the parameter k is increased to resolve longer features, the probability of observing any specific k-mer becomes very small, and k-mer counts approach a binary variable, with most k-mers absent and a few present once. Thus, any statistical learning approach using k-mers as features becomes susceptible to noisy training set k-mer frequencies once k becomes large. To address this problem, we introduce alternative feature sets using gapped k-mers, a new classifier, gkm-SVM, and a general method for robust estimation of k-mer frequencies. To make the method applicable to large-scale genome wide applications, we develop an efficient tree data structure for computing the kernel matrix. We show that compared to our original kmer-SVM and alternative approaches, our gkm-SVM predicts functional genomic regulatory elements and tissue specific enhancers with significantly improved accuracy, increasing the precision by up to a factor of two. We then show that gkm-SVM consistently outperforms kmer-SVM on human ENCODE ChIP-seq datasets, and further demonstrate the general utility of our method using a Naïve-Bayes classifier. Although developed for regulatory sequence analysis, these methods can be applied to any sequence classification problem.

A novel virtual sample generation method based on Gaussian distribution

Traditional machine learning algorithms are not with satisfying generalization ability on noisy, imbalanced, and small sample training set. In this work, a novel virtual sample generation (VSG) method based on Gaussian distribution is proposed. Firstly, the method determines the mean and the standard error of Gaussian distribution. Then, virtual samples can be generated by such Gaussian distribution. Finally, a new training set is constructed by adding the virtual samples to the original training set. This work has shown that training on the new training set is equivalent to a form of regularization regarding small sample problems, or cost-sensitive learning regarding imbalanced sample problems. Experiments show that given a suitable number of virtual sample replicates, the generalization ability of the classifiers on the new training sets can be better than that on the original training sets.

Weight decay backpropagation for noisy data

We present a formal evaluation of the effect of weight decay training for backpropagation on noisy data sets. Weight decay training is suggested as an implementation for achieving a robust neural network which is insensitive to noise. We investigate the noisy situations of noisy training set–clean test set, clean training set–noisy test set, and noisy training set–noisy test set. Statistically speaking, there is strong evidence indicating that the noisy situation of noisy training set–clean test set provides more accurate prediction than the other two noisy situations. This finding suggests the relative importance of maintaining to-be-predicted cases noise-free for neural network classification. Furthermore, experimental results show that weight decay training is at least as good as standard backpropagation in noisy situations and, in some data sets, weight decay training outperforms standard backpropagation by a significant difference. However, for clean data sets there is no significant difference between weight decay training and standard backpropagation. Another interesting finding in this study is the effect of the number of training epochs on weight decay training and standard backpropagation in noisy situations. Weight decay training can achieve convergence after a short training. For the same short training, weight decay training usually outperforms standard backpropagation. When additional training has a significant effect on performance, it is to improve standard backpropagation but deteriorate weight decay training.