Noisy Training Data Research Articles

Training neural networks with structured noise improves classification and generalization

Abstract The beneficial role of noise-injection in learning is a consolidated concept in the field of artificial neural networks, suggesting that even biological systems might take advantage of similar mechanisms to optimize their performance. The training-with-noise (TWN) algorithm proposed by Gardner and collaborators is an emblematic example of a noise-injection procedure in recurrent networks, which can be used to model biological neural systems. We show how adding structure to noisy training data can substantially improve the algorithm performance, allowing the network to approach perfect retrieval of the memories and wide basins of attraction, even in the scenario of maximal injected noise. We also prove that the so-called Hebbian Unlearning rule coincides with the TWN algorithm when noise is maximal and data are stable fixed points of the network dynamics.

Inverse analysis of soil hydraulic parameters of layered soil profiles using physics‐informed neural networks with unsaturated water flow models

AbstractInformation about the spatial distribution of soil hydraulic parameters is necessary for the accurate prediction of soil water flow and the coupled movement of chemicals and heat at the field scale using a process‐based model. Physics‐informed neural networks (PINNs), which can provide physical constraints in deep learning to obtain a mesh‐free solution, can be used to inversely estimate soil hydraulic parameters from less and noisy training data. Previous studies using PINNs have successfully estimated soil hydraulic parameters for homogeneous soil but estimating such parameters of layered soil profiles where the interface depth and the parameters are unknown still has some difficulties. The objective of this study was to develop PINNs to inversely estimate the distribution of soil hydraulic parameters, such as saturated hydraulic conductivity and α and n of the Mualem–van Genuchten model directly within layered soil profiles by predicting changes in the pressure head from training data based on simulation results at given depths during infiltration. The impact of factors affecting PINNs performance, such as the weights assigned to each component of the loss function, time range used in error computations, and number of samples used to assess the physical constraint, was investigated. By assigning a larger weight to the physical constraint and excluding the earlier stage of infiltration in the loss function, the changes in the pressure head and the three soil hydraulic parameter distributions within the layered soil were successfully estimated. The developed PINNs can be further applied to more complex soils and can be improved.

Identifying Plausible Labels from Noisy Training Data for a Land Use and Land Cover Classification Application in Amazônia Legal

Most studies in the field of land use and land cover (LULC) classification in remote sensing rely on supervised classification, which requires a substantial amount of accurate label data. However, reliable data are often not immediately available, and are obtained through time-consuming manual labor. One potential solution to this problem is the use of already available classification maps, which may not be the true ground truth and may contain noise from multiple possible sources. This is also true for the classification maps of the MapBiomas project, which provides land use and land cover (LULC) maps on a yearly basis, classifying the Amazon basin into more than 24 classes based on the Landsat data. In this study, we utilize the Sentinel-2 data with a higher spatial resolution in conjunction with the MapBiomas maps to evaluate a proposed noise removal method and to improve classification results. We introduce a novel noise detection method that relies on identifying anchor points in feature space through clustering with self-organizing maps (SOM). The pixel label is relabeled using nearest neighbor rules, or can be removed if it is unknown. A challenge in this approach is the quantification of noise in such a real-world dataset. To overcome this problem, highly reliable validation sets were manually created for quantitative performance assessment. The results demonstrate a significant increase in overall accuracy compared to MapBiomas labels, from 79.85% to 89.65%. Additionally, we trained the L2HNet using both MapBiomas labels and the filtered labels from our approach. The overall accuracy for this model reached 93.75% with the filtered labels, compared to the baseline of 74.31%. This highlights the significance of noise detection and filtering in remote sensing, and emphasizes the need for further research in this area.

Fake news detection in low-resource languages: A novel hybrid summarization approach

The proliferation of fake news across languages and domains on social media platforms poses a significant societal threat. Current automatic detection methods for low-resource languages (e.g., Swahili, Indonesian and other low-resource languages) face limitations due to two factors: sequential length restrictions in pre-trained language models (PLMs) like multilingual bidirectional encoder representation from transformers (mBERT), and the presence of noisy training data. This work proposes a novel and efficient multilingual fake news detection (MFND) approach that addresses these challenges. Our solution leverages a hybrid extractive and abstractive summarization strategy to extract only the most relevant content from news articles. This significantly reduces data length while preserving crucial information for fake news classification. The pre-processed data is then fed into mBERT for classification. Extensive evaluations on a publicly available multilingual dataset demonstrate the superiority of our approach compared to state-of-the-art (SOTA) methods. Our analysis, both quantitative and qualitative, highlights the strengths of this method, achieving new performance benchmarks and emphasizing the impact of content condensation on model accuracy and efficiency. This framework paves the way for faster, more accurate MFND, fostering more robust information ecosystems.

Mini-Workshop: Interpolation and Over-parameterization in Statistics and Machine Learning​

In recent years it has become clear that, contrary to traditional statistical beliefs, methods that interpolate (fit exactly) the noisy training data, can still be statistically optimal. In particular, this phenomenon of “be- nign overfitting” or “harmless interpolation” seems to be close to the practical regimes of modern deep learning systems, and, arguably, underlies many of their behaviors. This workshop brought together experts on the emerging theory of interpolation in statistical methods, its theoretical foundations and applications to machine learning and deep learning.

Meta learning addresses noisy and under-labeled data in machine learning-guided antibody engineering

Machine learning-guided protein engineering is rapidly progressing; however, collecting high-quality, large datasets remains a bottleneck. Directed evolution and protein engineering studies often require extensive experimental processes to eliminate noise and label protein sequence-function data. Meta learning has proven effective in other fields in learning from noisy data via bi-level optimization given the availability of a small dataset with trusted labels. Here, we leverage meta learning approaches to overcome noisy and under-labeled data and expedite workflows in antibody engineering. We generate yeast display antibody mutagenesis libraries and screen them for target antigen binding followed by deep sequencing. We then create representative learning tasks, including learning from noisy training data, positive and unlabeled learning, and learning out of distribution properties. We demonstrate that meta learning has the potential to reduce experimental screening time and improve the robustness of machine learning models by training with noisy and under-labeled training data.

Robust training for multi-view stereo networks with noisy labels

Recent multi-view stereo (MVS) works benefit a lot from deep neural networks with their great representation ability. However, top-performing networks require accurate depth ground truth rendered from scanned 3D models for training, which is quite challenging to capture in real-world scenarios. Thus, dealing with label noise contained in the dense depth labels is necessary for training when applied in complex real-world conditions, which has not been addressed before. In this paper, we investigate the effect of label noise in the ground truth depth maps by append noisy labels to the given accurate labels, and find that MVS networks are susceptible to noisy training data. To tackle this problem, we proposed a novel label noise-robust training framework for MVS networks. First, we propose a co-teaching based noise-robust training method to filter label noise contained in the dense depth ground truth, which trains two parallel network branches simultaneously and exchanges the noise selection capability to each other. Then, we present a co-training scheme that pseudo labels, the predicted depth maps that passes the geometric consistency check, are utilized as supervision to the peer network. Finally, the photometric consistency loss is introduced as additional supervision for all pixels. Experimental results demonstrate the robustness of deep MVS networks trained by our algorithm under low-level noise rate (40%) and high-level noise rate (65%), and the comparable 3D reconstruction performance to the network trained with label noise free datasets.

Denoising convolutional autoencoder for improving the classification performance based on noisy galaxy images

Images of galaxy objects are of great importance to the work of astronomers. Nowadays, the task of galaxy classification has been aided by Neural-Network-based classification models, who are powerful yet vulnerable to the attack of noisy images. In this research, RGB noises and bright spots simulating stars were generated, and a Convolutional-Neural-Network (CNN) based lightweight denoising image autoencoder was proposed. Firstly, a benchmark CNN classifier using DenseNet structure was trained on the Galaxy 10 DECals dataset, which consists of over 17, 000 RGB color galaxy images. Then, noisy images were generated by adding bright spots of different size and color simulating stars and applying gaussian RGB noises over the original images. The CNN autoencoder that consists of Convolutional layer in its encoder and Convolution Transpose layers in decoder was trained on the raw and noisy training data to learn effective galaxy image denoising. Finally, the effect of the autoencoder was evaluated by contrasting the performance of the CNN classifier over the noisy and denoised images. In contrast with being evaluated on the raw testing set, the CNN classifiers accuracy dropped by 0.41 when tested on the generated noisy testing images, indicating the effectiveness of the attack of image noises. While after denoising with the proposed autoencoder, the classifiers accuracy increased significantly by 0.37. Output denoised images also suggest that the autoencoder can effectively remove the applied bright spot and gaussian RGB noise, recreating the original shape of the galaxy.

Adaptive Template Reconstruction for Effective Pattern Classification.

A novel instance-based algorithm for pattern classification is presented and evaluated in this paper. This new method is motivated by the challenge of pattern classifications where only limited and/or noisy training data are available. For every classification, the proposed system transforms the query data and the training templates based on their distributions in the feature space. One of the major novelties of the proposed method is the concept of template reconstruction enabling improved performance with limited training data. The technique is compared with similar algorithms and evaluated using both the image and time-series modalities to demonstrate its effectiveness and versatility. Two public image databases, FASHION-MNIST and CIFAR-10, were used to test its effectiveness for the classification of images using small amounts of training samples. An average classification improvement of 2~3% was observed while using a small subset of the training database, compared to the performances achieved by state-of-the-art techniques using the full datasets. To further explore its capability in solving more challenging classification problems such as non-stationary time-series electroencephalography (EEG) signals, a clinical grade 64-electrode EEG database, as well as a low-quality (high-noise level) EEG database, obtained using a low-cost system equipped with a single dry sensor, have also been used to test the algorithm. Adaptive reconstruction of the feature instances has been seen to have substantially improved class separation and matching performance for both still images and time-series signals. In particular, the method is found to be effective for the classification of noisy non-stationary data with limited training data volumes, indicating its potential suitability for a wide range of applications.

A self-validation Noise2Noise training framework for image denoising

ABSTRACT Image denoising is a crucial algorithm in image processing that aims to enhance image quality. Deep learning-based image denoising methods can be categorized into supervised and unsupervised approaches. Supervised learning requires pairs of noisy and noise-free training data, which is impractical in real-world scenarios. Unsupervised learning uses pairs of noisy images for training, but it may yield lower accuracy. Additionally, deep learning-based methods often require a large amount of training data. To overcome these challenges, this research proposes a self-validation Noise2Noise (SV-N2N) framework that generates validation sets using only noisy images without requiring noise-free pairs. The proposed SV-N2N method effectively reduces noise, comparable to supervised and unsupervised methods, without requiring a noise-free ground truth, which is efficient for solving real-world scenarios.

COMIRE: A Consistence-Based Mislabeled Instances Removal Method.

Training neural network classifiers (NNCs) usually requires all instances to be correctly labeled, which is difficult and/or expensive to satisfy in some practical applications. When label noise is present, mislabeled data will severely mislead the training of NNCs, resulting in poor generalization performance. In this work, we address the label noise issue by removing mislabeled instances from the training data. A COnsistence-based Mislabeled Instances REmoval (COMIRE) method is proposed. The main idea is based on the observation that during the training of the NNC, the training loss and the model's prediction uncertainty of correctly labeled instances show similar trends, while those of mislabeled instances have quite different trends. Thus, the consistency between the two trends can be used to distinguish correctly labeled instances from mislabeled ones. On this basis, an iteration scheme is introduced to further increase the separability between the two types of data. Experimental results show that COMIRE can effectively identify the mislabeled instances. Moreover, the classification performance is significantly improved after removing the identified instances from the noisy training data.

Extreme Events Prediction from Nonlocal Partial Information in a Spatiotemporally Chaotic Microcavity Laser.

The forecasting of high-dimensional, spatiotemporal nonlinear systems has made tremendous progress with the advent of model-free machine learning techniques. However, in real systems it is not always possible to have all the information needed; only partial information is available for learning and forecasting. This can be due to insufficient temporal or spatial samplings, to inaccessible variables, or to noisy training data. Here, we show that it is nevertheless possible to forecast extreme event occurrences in incomplete experimental recordings from a spatiotemporally chaotic microcavity laser using reservoir computing. Selecting regions of maximum transfer entropy, we show that it is possible to get higher forecasting accuracy using nonlocal data vs local data, thus allowing greater warning times of at least twice the time horizon predicted from the nonlinear local Lyapunov exponent.

Impact of Noisy Labels on Dental Deep Learning-Calculus Detection on Bitewing Radiographs.

Supervised deep learning requires labelled data. On medical images, data is often labelled inconsistently (e.g., too large) with varying accuracies. We aimed to assess the impact of such label noise on dental calculus detection on bitewing radiographs. On 2584 bitewings calculus was accurately labeled using bounding boxes (BBs) and artificially increased and decreased stepwise, resulting in 30 consistently and 9 inconsistently noisy datasets. An object detection network (YOLOv5) was trained on each dataset and evaluated on noisy and accurate test data. Training on accurately labeled data yielded an mAP50: 0.77 (SD: 0.01). When trained on consistently too small BBs model performance significantly decreased on accurate and noisy test data. Model performance trained on consistently too large BBs decreased immediately on accurate test data (e.g., 200% BBs: mAP50: 0.24; SD: 0.05; p < 0.05), but only after drastically increasing BBs on noisy test data (e.g., 70,000%: mAP50: 0.75; SD: 0.01; p < 0.05). Models trained on inconsistent BB sizes showed a significant decrease of performance when deviating 20% or more from the original when tested on noisy data (mAP50: 0.74; SD: 0.02; p < 0.05), or 30% or more when tested on accurate data (mAP50: 0.76; SD: 0.01; p < 0.05). In conclusion, accurate predictions need accurate labeled data in the training process. Testing on noisy data may disguise the effects of noisy training data. Researchers should be aware of the relevance of accurately annotated data, especially when testing model performances.

Defect Detection in Printed Circuit Boards Using Semi-Supervised Learning

Defect inspection is essential in the semiconductor industry to fabricate printed circuit boards (PCBs) with minimum defect rates. However, conventional inspection systems are labor-intensive and time-consuming. In this study, a semi-supervised learning (SSL)-based model called PCB_SS was developed. It was trained using labeled and unlabeled images under two different augmentations. Training and test PCB images were acquired using automatic final vision inspection systems. The PCB_SS model outperformed a completely supervised model trained using only labeled images (PCB_FS). The performance of the PCB_SS model was more robust than that of the PCB_FS model when the number of labeled data is limited or comprises incorrectly labeled data. In an error-resilience test, the proposed PCB_SS model maintained stable accuracy (error increment of less than 0.5%, compared with 4% for PCB_FS) for noisy training data (with as much as 9.0% of the data labeled incorrectly). The proposed model also showed superior performance when comparing machine-learning and deep-learning classifiers. The unlabeled data utilized in the PCB_SS model helped with the generalization of the deep-learning model and improved its performance for PCB defect detection. Thus, the proposed method alleviates the burden of the manual labeling process and provides a rapid and accurate automatic classifier for PCB inspections.

Learning from multiple annotators for medical image segmentation

Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the ”actual” segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator’s “unreliable behavior” (we call it “maximally unreliable”) while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method’s efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators’ mistakes.

Semantic segmentation of water bodies in very high-resolution satellite and aerial images

This study evaluates the performance of convolutional neural networks for semantic segmentation of water bodies in very high-resolution satellite and aerial images from multiple sensors with particular focus on flood emergency response applications. Different model architectures (U-Net and DeepLab-V3+) are combined with encoder backbones (MobileNet-V3, ResNet-50 and EfficientNet-B4) and tested for their ability to delineate inundated areas under varying environmental conditions and data availability scenarios. An unprecedented reference dataset of 1120 globally sampled images with quality checked binary water masks is introduced and used to train, validate and test the models for water body segmentation. Furthermore, independent test datasets are developed to test the generalization ability of the trained models across regions, sensors (IKONOS, GeoEye-1, WorldView-2, WorldView-3 and four different airborne camera systems) and tasks (normal water and flood water segmentation). Results indicate that across all tested scenarios a U-Net model with Mobilenet-V3 backbone pre-trained on ImageNet performs best. While using R-G-B image bands performs well, adding the near infrared band (if available) slightly improves prediction results. Similarly, adding slope information from an independent digital elevation model increases accuracies. Train-time augmentation and contrast enhancement could improve transferability across sensors and in particular between satellite and aerial images. Moreover, adding noisy training data from freely available online resources could further improve performance with minimal annotation effort.

Analysis of Interpolating Regression Models and the Double Descent Phenomenon

A regression model with more parameters than data points in the training data is overparametrized and has the capability to interpolate the training data. Based on the classical bias-variance tradeoff expressions, It is commonly assumed that models which interpolate noisy training data are poor to generalize. In some cases, this is not true. The best models obtained are overparametrized and the testing error exhibits the double descent behavior as the model order increases. In this contribution, we provide some analysis to explain the double descent phenomenon, first reported in the machine learning literature. We focus on interpolating models derived from the minimum norm solution to the classical least-squares problem and also briefly discuss model fitting using ridge regression. We derive a result based on the behavior of the smallest singular value of the regression matrix that explains the peak location and the double descent shape of the testing error as a function of model order.

Impact of Label Noise on the Learning Based Models for a Binary Classification of Physiological Signal

Label noise is omnipresent in the annotations process and has an impact on supervised learning algorithms. This work focuses on the impact of label noise on the performance of learning models by examining the effect of random and class-dependent label noise on a binary classification task: quality assessment for photoplethysmography (PPG). PPG signal is used to detect physiological changes and its quality can have a significant impact on the subsequent tasks, which makes PPG quality assessment a particularly good target for examining the impact of label noise in the field of biomedicine. Random and class-dependent label noise was introduced separately into the training set to emulate the errors associated with fatigue and bias in labeling data samples. We also tested different representations of the PPG, including features defined by domain experts, 1D raw signal and 2D image. Three different classifiers are tested on the noisy training data, including support vector machine (SVM), XGBoost, 1D Resnet and 2D Resnet, which handle three representations, respectively. The results showed that the two deep learning models were more robust than the two traditional machine learning models for both the random and class-dependent label noise. From the representation perspective, the 2D image shows better robustness compared to the 1D raw signal. The logits from three classifiers are also analyzed, the predicted probabilities intend to be more dispersed when more label noise is introduced. From this work, we investigated various factors related to label noise, including representations, label noise type, and data imbalance, which can be a good guidebook for designing more robust methods for label noise in future work.

A multi-fidelity active learning method for global design optimization problems with noisy evaluations

A multi-fidelity (MF) active learning method is presented for design optimization problems characterized by noisy evaluations of the performance metrics. Namely, a generalized MF surrogate model is used for design-space exploration, exploiting an arbitrary number of hierarchical fidelity levels, i.e., performance evaluations coming from different models, solvers, or discretizations, characterized by different accuracy. The method is intended to accurately predict the design performance while reducing the computational effort required by simulation-driven design (SDD) to achieve the global optimum. The overall MF prediction is evaluated as a low-fidelity trained surrogate corrected with the surrogates of the errors between consecutive fidelity levels. Surrogates are based on stochastic radial basis functions (SRBF) with least squares regression and in-the-loop optimization of hyperparameters to deal with noisy training data. The method adaptively queries new training data, selecting both the design points and the required fidelity level via an active learning approach. This is based on the lower confidence bounding method, which combines the performance prediction and the associated uncertainty to select the most promising design regions. The fidelity levels are selected considering the benefit-cost ratio associated with their use in the training. The method’s performance is assessed and discussed using four analytical tests and three SDD problems based on computational fluid dynamics simulations, namely the shape optimization of a NACA hydrofoil, the DTMB 5415 destroyer, and a roll-on/roll-off passenger ferry. Fidelity levels are provided by both adaptive grid refinement and multi-grid resolution approaches. Under the assumption of a limited budget for function evaluations, the proposed MF method shows better performance in comparison with the model trained by high-fidelity evaluations only.

Identifying Varying Thermal Diffusivity of Inhomogeneous Materials Based on a Hybrid Physics-Informed Neural Network

Inhomogeneous materials are characterized by uneven material properties with changing dimensions. Effective and precise identification of nonuniform material properties is essential for studying material inhomogeneity. In this study, we propose a hybrid physics-informed neural network (PINN) to identify the inhomogeneous thermal diffusivity only using temperature data collected in a transient heat conduction problem. A PINN and a fully connected neural network are combined in the proposed model. The PINN model is trained to learn the law of physics governed by the transient heat conduction equation. The identification network is trained to uncover the thermal diffusivity variation with changing dimensions. The proposed model successfully identifies two types of inhomogeneity: temperature-dependent and space-dependent thermal diffusivity, and both two-dimensional and three-dimensional heat conduction problems are investigated. The proposed model is examined to be robust to small datasets and noisy training data. In addition to heat conduction problems, the proposed model can be adopted to identify the material properties of various types of inhomogeneous materials.