Noisy Datasets Research Articles

A new binary classifier robust on noisy domains based on kNN algorithm

Classification is an effective technique commonly used in data analysis by systematically arranging groups or categories according to established criteria. The classifier's success relies on the classifier itself and the quality of the data. However, in real-world applications, it is inevitable for datasets to contain mislabeled instances, which may cause misclassification challenges that classifiers have to handle. This study aims for a quantitative assessment of the classification of noisy data through a new kNN-based classification algorithm and to increase the performance of classical kNN by efficiently classifying the data. We perform various numerical experiments on real-world data sets to prove our new algorithm's performance. We obtain high standards of accuracy levels on various noisy datasets. We propose that this new technique can provide high standard accuracy levels in binary classification problems. We compared the new kNN and classical kNN algorithms in various noise levels (10%, 20%, 30%, and 40%) on distinct datasets by measuring in terms of test accuracy. Also, we compared our new algorithm with popular classification algorithms and in the vast majority, we obtained better test accuracy results.

Robust Trend Analysis in Environmental Remote Sensing: A Case Study of Cork Oak Forest Decline

We introduce a novel methodological framework for robust trend analysis (RTA) using remote sensing data to enhance the accuracy and reliability of detecting significant environmental trends. Our approach sequentially integrates the Theil–Sen (TS) slope estimator, the Contextual Mann–Kendall (CMK) test, and the false discovery rate (FDR) control. This comprehensive method addresses common challenges in trend analysis, such as handling small, noisy datasets with outliers and issues related to spatial autocorrelation, cross-correlation, and multiple testing. We applied this RTA workflow to study tree cover trends in Los Alcornocales Natural Park (Southern Spain), Europe’s largest cork oak forest, analysing interannual changes in tree cover from 2000 to 2022 using Terra MODIS MOD44B data. Our results reveal that the TS estimator provides a robust measure of trend direction and magnitude, but its effectiveness is dramatically enhanced when combined with the CMK test. This combination highlights significant trends and effectively corrects for spatial autocorrelation and cross-correlation, ensuring that genuine environmental signals are distinguished from statistical noise. Unlike previous workflows, our approach incorporates the FDR control, which successfully filtered out 29.6% of false discoveries in the case study, resulting in a more stringent assessment of true environmental trends captured by multi-temporal remotely sensed data. In the case study, we found that approximately one-third of the area exhibits significant and statistically robust declines in tree cover, with these declines being geographically clustered. Importantly, these trends correspond with relevant changes in tree cover, emphasising the ability of RTA to detect relevant environmental changes. Overall, our findings underscore the crucial importance of combining these methods, as their synergy is essential for accurately identifying and confirming robust environmental trends. The proposed RTA framework has significant implications for environmental monitoring, modelling, and management.

EEG-based emotional valence and emotion regulation classification: a data-centric and explainable approach

Emotion classification using electroencephalographic (EEG) data is a challenging task in the field of Artificial Intelligence. While many researchers have focused on finding the best model or feature extraction technique to achieve optimal results, few have attempted to select the best methodological steps for working with the dataset. In this study, we applied two different theoretical approaches based on the noise of the dataset: curriculum learning and confident learning. Curriculum learning involves presenting training examples to the model in a specific order, starting with easier examples and gradually increasing in difficulty. This approach has been shown to improve model performance. Confident learning is a method for identifying and correcting label errors in datasets. By identifying and correcting these errors, confident learning can improve the performance of machine learning models trained on noisy datasets. We then applied the Integrated Gradient technique in order to assess the explainability of each model. Our aim was to explore the impact of different models and methods on emotion classification performance using EEG data. We collected and used an EEG dataset in which participants rated the emotional valence of positive and negative pictures while performing an emotion regulation (ER) task, comparing a control condition (Look) with two ER strategies: cognitive reappraisal and expressive suppression. We performed a multilabel classification to identify emotional neutrality or polarization of emotional valence (both positive and negative) rated by participants and the emotion regulation strategy adopted during the task. We compared the performance of models trained on three datasets selected based on label noise and evaluated their suitability for this task. Our results suggest different patterns based on the architecture used for feature importance, highlighting both advantages and criticisms.

A deep autoencoder for electric double layer capacitance prediction in electrochemical sensors

This study explores the application of a deep autoencoder neural network to accurately predict the electric double layer capacitance from real-world parameters in binary, asymmetric electrolytes under low concentration conditions. By utilizing a modest simulation-based dataset of just 250 samples, the deep autoencoder neural network model developed in this study effectively predicted the capacitance by learning the critical features and relationships of the electric double layer model and encoding this learned representation into a low-dimensional latent space. From the latent variables, the decoder block of the neural network learned to effectively recreate the high-dimensional input. To enhance the model's robustness, prevent overfitting, and better simulate real-world conditions, noise was incorporated into the training and test data. The model demonstrated strong performance across various conditions, such as ionic size, ionic charge, and surface potential, yielding satisfactory results on both clean and noisy test datasets. A key feature of this approach was the mapping of real-world electric double layer parameters to the latent variables of the model, allowing for direct input of physical parameters to predict the electric double layer capacitance. This research highlights the potential of machine learning techniques to expedite the design and analysis of complex multi-physics systems such as electrochemical sensors by reducing the dependence on extensive domain expertise throughout the design process.

Uncertainty in Fourier Transforms: A Fuzzy Logic Perspective

This study explores the integration of fuzzy logic with Fourier transforms to address the challenges of uncertainty, noise, and imprecision in real-world data. Fuzzy Fourier transforms extend traditional Fourier methods by incorporating fuzzy numbers, allowing for more robust frequency analysis, signal processing, and data reconstruction, particularly in noisy or incomplete datasets. The study examines the mathematical formulation of fuzzy Fourier transforms, computational trade-offs, and their performance advantages in comparison to classical Fourier methods. Real-world applications are discussed, including signal processing, image reconstruction, and time-series forecasting. Furthermore, the research highlights the increased computational complexity associated with fuzzy methods, the challenges of interpreting fuzzy results, and the limitations of handling probabilistic uncertainty. Future directions include the development of multi-dimensional fuzzy Fourier transforms, hybrid models integrating machine learning, and quantum computing applications. These advancements have broad implications for fields such as telecommunications, medical imaging, and financial forecasting, where handling uncertainty is critical.

MTC-NET: A Multi-Channel Independent Anomaly Detection Method for Network Traffic.

In recent years, deep learning-based approaches, particularly those leveraging the Transformer architecture, have garnered widespread attention for network traffic anomaly detection. However, when dealing with noisy data sets, directly inputting network traffic sequences into Transformer networks often significantly degrades detection performance due to interference and noise across dimensions. In this paper, we propose a novel multi-channel network traffic anomaly detection model, MTC-Net, which reduces computational complexity and enhances the model's ability to capture long-distance dependencies. This is achieved by decomposing network traffic sequences into multiple unidimensional time sequences and introducing a patch-based strategy that enables each sub-sequence to retain local semantic information. A backbone network combining Transformer and CNN is employed to capture complex patterns, with information from all channels being fused at the final classification header in order to achieve modelling and detection of complex network traffic patterns. The experimental results demonstrate that MTC-Net outperforms existing state-of-the-art methods in several evaluation metrics, including accuracy, precision, recall, and F1 score, on four publicly available data sets: KDD Cup 99, NSL-KDD, UNSW-NB15, and CIC-IDS2017.

Segmenting hotspots from medical thermal images using Density-based modified FC-Pc FS with spatial information

ABSTRACT Thermal cameras complemented with robust and automated computational intelligence-aided diagnostic systems can be deployed at public gathering points and on smartphones to raise alarms about thriving diseases of their body due to inflammation in a timely manner, thereby strengthening human health systems. In this work, we have proposed two robust Density-based modified Picture Fuzzy Clustering methods with spatial information to segment hotspots from human’s abnormal thermal images. We formulated and solved the objective function by combining P c FS-based clustering method with modified Renyi’s Entropy. We demonstrated the robustness of our methods, DSIFC-Pc FS and DSIMFC-Pc FS, on three publicly available datasets and an artificially created noisy dataset of thermal images. We have obtained a ZC score of 0.933, 0.947 and 0.933 for DB-DMR-IR, DB-FOOT-IR and DB-THY-IR datasets, respectively. Also, our method converged to an optimal partition matrix in minimum iterations. The experimental performance shows that the proposed approaches, DSIFC- P c FS and DSIMFC- P c FS, outperformed other ten related approaches in segmenting the medical thermal images with and without noise, indicating their robustness and efficacy, in terms of various unsupervised evaluation metrics. Also, we statistically analysed their performance in contrast to other methods using the Friedman Test, which advocate their superiority.

Pose‐to‐Motion: Cross‐Domain Motion Retargeting with Pose Prior

AbstractCreating plausible motions for a diverse range of characters is a long‐standing goal in computer graphics. Current learning‐based motion synthesis methods rely on large‐scale motion datasets, which are often difficult if not impossible to acquire. On the other hand, pose data is more accessible, since static posed characters are easier to create and can even be extracted from images using recent advancements in computer vision. In this paper, we tap into this alternative data source and introduce a neural motion synthesis approach through retargeting, which generates plausible motion of various characters that only have pose data by transferring motion from one single existing motion capture dataset of another drastically different characters. Our experiments show that our method effectively combines the motion features of the source character with the pose features of the target character, and performs robustly with small or noisy pose data sets, ranging from a few artist‐created poses to noisy poses estimated directly from images. Additionally, a conducted user study indicated that a majority of participants found our retargeted motion to be more enjoyable to watch, more lifelike in appearance, and exhibiting fewer artifacts. Our code and dataset can be accessed here.

MD-BiMamba: An aero-engine inter-shaft bearing fault diagnosis method based on Mamba with modal decomposition and bidirectional features fusion strategy

Inter-shaft bearing fault diagnosis can greatly save maintenance costs and guarantee the normal operation of aero-engines. However, existing methods are limited in their ability to handle sensor signals with high dimensional and complex noise. To tackle the above problems, we propose a Mamba network structure with Complete Ensemble Empirical Mode Decomposition with Adaptive Noise (CEEMDAN) and bidirectional feature fusion for aero-engine inter-shaft bearing fault diagnosis, called MD-BiMamba. Firstly, we utilize the CEEMDAN technique to decompose the sensor signals of inter-shaft bearings and extract the intrinsic modal function by improving the noise addition method and decomposition strategy. Then, bidirectional Mamba is designed to extract the bidirectional long-range correlation of the sequence to achieve global feature extraction and fast detection of fault types with linear complexity. Moreover, an adaptive attention fusion method is proposed to achieve bidirectional feature fusion. Finally, we use the label smoothing regularization strategy to optimize the cross-entropy loss and enhance the generalization performance. Experimental results on real inter-shaft bearing datasets and noisy datasets show that the proposed method achieves 99.88 % and 95.81 % accuracy in the standard dataset and −10 dB noise environment, respectively, which is superior to other remarkable methods, in addition to achieving lower model complexity.

Learning Hamiltonian dynamics with reproducing kernel Hilbert spaces and random features

A method for learning Hamiltonian dynamics from a limited and noisy dataset is proposed. The method learns a Hamiltonian vector field on a reproducing kernel Hilbert space (RKHS) of inherently Hamiltonian vector fields, and in particular, odd Hamiltonian vector fields. This is done with a symplectic kernel, and it is shown how the kernel can be modified to an odd symplectic kernel to impose the odd symmetry. A random feature approximation is developed for the proposed odd kernel to reduce the problem size. The performance of the method is validated in simulations for three Hamiltonian systems. It is demonstrated that the use of an odd symplectic kernel improves prediction accuracy and data efficiency, and that the learned vector fields are Hamiltonian and exhibit the imposed odd symmetry characteristics.

NEURO DEEP FUZZY GENETIC ALGORITHM APPROACH FOR CLASSIFICATION AND DETECTION OF BRAIN TUMOR FROM LARGE DATASETS

Brain tumors represent one of the most critical and life-threatening diseases. Early detection and accurate classification are essential for effective treatment planning and survival rate improvement. With the exponential growth of medical data, particularly in imaging and diagnostic datasets, traditional algorithms face limitations in handling large-scale, high-dimensional data efficiently. Conventional machine learning and diagnostic methods often struggle with classification accuracy, computational complexity, and overfitting in large, noisy datasets. Addressing these issues is crucial for the development of robust diagnostic tools capable of handling real-world clinical data. This paper presents a novel hybrid approach combining Neural Networks, Deep Learning, Fuzzy Logic, and Genetic Algorithms, termed Neuro Deep Fuzzy Genetic Algorithm (NDFGA), for the classification and detection of brain tumors from large datasets. Neural Networks and Deep Learning architectures are leveraged for feature extraction and hierarchical learning. Fuzzy Logic improves interpretability and manages uncertainty in medical data, while Genetic Algorithms optimize feature selection and model parameters. This hybrid method is designed to maximize classification accuracy while minimizing false positives and computational overhead. The proposed approach was tested on a large brain tumor dataset comprising over 10,000 MRI scans. The NDFGA approach demonstrated superior performance compared to standalone methods. It achieved a classification accuracy of 97.8%, a sensitivity of 96.5%, and a specificity of 98.1%. The model also showed improved robustness in handling large datasets, reducing false positives by 12% and computational time by 15% compared to traditional methods. This hybrid model presents a scalable and efficient solution for brain tumor detection, especially in clinical environments.

Systematization of sunflower genotypes based on seed phenotypic characteristics using neural networks

In recent years, various datasets related to the phenotyping of sunflower genotypes have become increasingly accessible. However, one of the key challenges remains the efficient and accurate prediction of phenotypes based on genotypes in the context of climate change. Analyzing phenotypes at different levels of organization and detecting connections between phenotypes and genotypes require the integration and processing of large, diverse, and often noisy datasets. Machine learning offers a broad arsenal of methods and approaches for identifying predictive patterns in such data. Therefore, the research aimed to develop a methodology for the systematization of sunflower genotypes based on seed phenotypic characteristics using the data vector quantization method and neural networks. The study revealed the phenotypic characteristics of sunflower seeds from various genotypes selected by the Institute of Oilseed Crops of NAAS, grown in the southern Steppe of Ukraine, including seed length, width, thickness, seed mass, kernel mass, and seed coat cracking force. For this purpose, appropriate laboratory equipment was developed, including two modules for determining the morphological and rheological properties of seeds. The developed methodology for the systematization of sunflower genotypes based on seed phenotypic characteristics includes the following steps: measuring the characteristics of sunflower seeds from various samples (parental components); studying the mutual correlation of characteristics; conducting hierarchical cluster analysis of the data using the Ward's method; determining the optimal number of groups; performing k-means clustering using the vector quantization method; determining the correspondence of ranges of characteristics to the group; training a neural network to group the data by samples and created groups; verifying the adequacy of the neural network on test data. The developed methodology was tested, and the MLP 30-15-3 neural network for grouping data by samples and created groups of sunflower seeds was developed in the Statistica software package. The network's training efficiency was 99.4%, and such of testing and validation was 95.6% and 96.7%, respectively.

Artificial-Intelligence-Based Condition Monitoring of Industrial Collaborative Robots: Detecting Anomalies and Adapting to Trajectory Changes

The increasing use of collaborative robots in smart manufacturing, owing to their flexibility and safety benefits, underscores a critical need for robust predictive maintenance strategies to prevent unexpected faults/failures of the machine. This paper focuses on fault detection and employs multivariate operational data from a universal robot to detect anomalies or early-stage faults using test data from designed anomalous conditions and artificial-intelligence-based anomaly detection techniques called autoencoders. The performance of three autoencoders, namely, a multi-layer-perceptron-based autoencoder, convolutional-neural-network-based autoencoder, and sparse autoencoder, was compared in detecting anomalies. The results indicate that the autoencoders effectively detected anomalies in the examined complex and noisy datasets with more than 93% overall accuracy and an F1 score exceeding 96% for the considered anomalous cases. Moreover, the integration of trajectory change detection and anomaly detection algorithms (i.e., the dynamic time warping algorithm and sparse autoencoder, respectively) was proposed for the local implementation of online condition monitoring. This integrated approach to anomaly detection and trajectory change provides a practical, adaptive, and economical solution for enhancing the reliability and safety of collaborative robots in smart manufacturing environments.

Testing protocols for smoothing datasets of hydraulic variables acquired during unsteady flows

ABSTRACT Flood wave propagation involves complex flow variable dependencies. Continuous in situ hydrograph peak magnitude and timing data provide the most relevant information for understanding these dependencies. New acoustic instruments can produce experimental evidence by extracting usable signals from noisy datasets. This study presents a new screening protocol to smoothen streamflow data from unwanted influences and noise generated by flow perturbations and observational fluctuations. The protocol combines quantitative (statistical fitness parameters) and qualitative (domain expert judgments) evaluations. It is tested with 18 smoothing methods to identify optimal data conditioning candidates. Sensitivity analyses assess the validity, generality, and scalability of the procedures. The goal of this analysis is to set a mathematical foundation for empirical results that can lead to unified, general conclusions on principles or protocols for unsteady flows propagating in open channels, formulating practical guidance for future data acquisition and processing, and using in situ data to better support data-driven modeling efforts.

A hierarchical heterogeneous ant colony optimization based oversampling algorithm using feature similarity for classification of imbalanced data

Imbalanced data classification is one of the challenging problems in machine learning. Oversampling is a promising technique that generates synthetic minority instances to balance the dataset. Inappropriate minority instances generated may deteriorate the performance of the classifier. Majority of the oversampling algorithms create new minority instances by choosing nearest neighbors for random interpolation. However, these methods do not provide new information to the dataset and therefore standard classifiers do not show good performance on such datasets. Therefore, it is necessary to generate diverse minority class instances to increase the performance of the classifier. Since, every feature of each minority class instance contribute valuable information, generating synthetic instances from the features of all minority instances would produce diverse minority instances, thereby increasing the performance of the classifier. This paper proposes a Hierarchical Heterogeneous Ant Colony Optimization based oversampling algorithm using Feature Similarity (HHACO-FSOTe) for generation of synthetic minority instances. Instead of choosing few neighbors for interpolation, the proposal considers all minority instances for generation of synthetic instances. HHACO-FSOTe generates new feature values by computing the minimum absolute difference between the features of a given minority instance and the corresponding features of the remaining minority instances. The features in the dataset are distributed among the ant agents enabling parallelism, thereby reducing the time taken for oversampling. HHACO-FSOTe do not require parameter tuning or training. The proposal is evaluated on 41 low dimensional, 11 high dimensional and 8 noisy datasets. Experiments reveal that HHACO-FSOTe is competent with the state-of-art oversampling techniques. Results were validated using non-parametric statistical tests.

Ultralow Energy Consumption and Fast Neuromorphic Computing Based on La0.1Bi0.9FeO3 Ferroelectric Tunnel Junctions.

Low-power and fast artificial neural network devices represent the direction in developing analogue neural networks. Here, an ultralow power consumption (0.8 fJ) and rapid (100 ns) La0.1Bi0.9FeO3/La0.7Sr0.3MnO3 ferroelectric tunnel junction artificial synapse has been developed to emulate the biological neural networks. The visual memory and forgetting functionalities have been emulated based on long-term potentiation and depression with good linearity. Moreover, with a single device, logical operations of "AND" and "OR" are implemented, and an artificial neural network was constructed with a recognition accuracy of 96%. Especially for noisy data sets, the recognition speed is faster after preprocessing by the device in the present work. This sets the stage for highly reliable and repeatable unsupervised learning.

Dynamic data reconciliation for enhancing the prediction performance of long short-term memory network

Abstract In modern process industries, long short-term memory (LSTM) network is widely used for data-driven modeling. Constrained by measuring instruments and environments, the measured datasets are generally with Gaussian/non-Gaussian distributed measurement noise. The noisy datasets will impact the modeling accuracy of the LSTM network and decrease the prediction performance of it. Aiming at addressing prediction performance impairment of the LSTM network under noisy datasets with Gaussian/non-Gaussian distribution, this study introduces dynamic data reconciliation (DDR) both into LSTM network training and into LSTM network test. Results show that DDR improves not only the data quality based on noisy datasets and the training outputs via the Bayesian formula in the model training step, but also the prediction performance based on offline measured information and the test outputs. The implementation scheme of DDR for Gaussian and non-Gaussian distributed noise is purposely designed. The effectiveness of DDR on the LSTM model is verified in a numerical example and a case involving a set of shared wind power datasets.

BO4IO: A Bayesian optimization approach to inverse optimization with uncertainty quantification

Data-driven inverse optimization (IO) aims to estimate unknown parameters in an optimization model from observed decisions. The IO problem is commonly formulated as a large-scale bilevel program that is notoriously difficult to solve. We propose a derivative-free optimization approach based on Bayesian optimization, BO4IO, to solve general IO problems. The main advantages of BO4IO are two-fold: (i) it circumvents the need of complex reformulations or specialized algorithms and can hence enable computational tractability even when the underlying optimization problem is nonconvex or involves discrete variables, and (ii) it allows approximations of the profile likelihood, which provide uncertainty quantification on the IO parameter estimates. Our extensive computational results demonstrate the efficacy and robustness of BO4IO to estimate unknown parameters from small and noisy datasets. In addition, the proposed profile likelihood analysis effectively provides good approximations of the confidence intervals on the parameter estimates and assesses the identifiability of the unknown parameters.

Transformer for low concentration image denoising in magnetic particle imaging.

Objective.Magnetic particle imaging (MPI) is an emerging tracer-basedin vivoimaging technology. The use of MPI at low superparamagnetic iron oxide nanoparticle concentrations has the potential to be a promising area of clinical application due to the inherent safety for humans. However, low tracer concentrations reduce the signal-to-noise ratio of the magnetization signal, leading to severe noise artifacts in the reconstructed MPI images. Hardware improvements have high complexity, while traditional methods lack robustness to different noise levels, making it difficult to improve the quality of low concentration MPI images.Approach.Here, we propose a novel deep learning method for MPI image denoising and quality enhancing based on a sparse lightweight transformer model. The proposed residual-local transformer structure reduces model complexity to avoid overfitting, in which an information retention block facilitates feature extraction capabilities for the image details. Besides, we design a noisy concentration dataset to train our model. Then, we evaluate our method with both simulated and real MPI image data.Main results.Simulation experiment results show that our method can achieve the best performance compared with the existing deep learning methods for MPI image denoising. More importantly, our method is effectively performed on the real MPI image of samples with an Fe concentration down to 67μgFeml-1.Significance.Our method provides great potential for obtaining high quality MPI images at low concentrations.

Research on steel surface defect detection system based on YOLOv5s-SE-CA model and BEMD image enhancement

ABSTRACT The nondestructive testing capability of steel surface defects has a significant impact on quality of industrial production. However, the single-stage detection network YOLOv5s model for steel surface defect detection is of insufficient feature extraction ability and susceptibility to noise interference in extracting surface defect images. Accordingly, a steel surface defect detection system based on YOLOv5s-SE-CA model and bi-dimensional empirical mode decomposition (BEMD) image enhancement is proposed. Firstly, the BEMD is used for image threshold denoising to improve the quality of surface defect images and achieve preprocessing of the dataset. Secondly, the squeeze and excitation network (SENet) attention mechanism is introduced into the backbone network of YOLOv5s model to adaptively adjust weights of each feature channel. Meanwhile, the coordinate attention (CA) attention mechanism is integrated to improve network model’s ability to focus on surface defects. Therefore, the YOLOv5s-SE-CA model is designed to enhance detection capability of steel surface defects. Experimental verification was carried out using the NEU-DET noisy dataset with specific evaluation indicators of image enhancement detection system. Results showed the proposed system has superior performance when compared to traditional models trained on noisy and denoising datasets and achieved more accurate detection of surface defects in steel.