Noise Cleaning Research Articles

Optimization of parallel SVM algorithm for big data

Parallel Support Vector Machine (SVM) based on big data has achieved some results in data mining, but due to the complexity of the data itself and a large amount of noisy data, its execution efficiency and classification accuracy in the big data environment are very low. In order to eliminate noise, a noise reduction method based on Noise Cleaning (NC) strategy was proposed, and redundant training samples in big data environments were deleted; Introduce an improved Artificial Fish Swarm Algorithm (IAFSA) to obtain the final Parallel SVM algorithm using mutual information and artificial fish swarm algorithm based on MapReduce (MIAFSA-PSVM) classification model. The results indicate that when compared to CMI-PSVM, the execution time of MIAFSA-PSVM algorithm is higher on the NDC dataset with the largest data size, The SVM parameter optimization algorithm based on MapReduce and cuckoo search (CSSVM-MR) and the particle swarm optimization based parallel support vector machine ensemble algorithm (PSO-PSVM) decreased by 40.1%, 79.3%, and 51.7%, respectively. This indicates that GIESVM-MR and MIAFSA-PSVM have strong adaptability to big data environments and high classification accuracy.

Intelligent agent for hyperspectral image classification with noisy labels: a deep reinforcement learning framework

ABSTRACT Label accuracy plays an important role in supervised hyperspectral image classification problems. However, labelling can be prone to various types of noise, mainly including human errors and incorrect pseudo-label predictions. Learning from datasets with noisy labels is a critical challenge in practice. To address this issue, we first propose a deep reinforcement learning method named HyperRefine to tackle the label noise problem for hyperspectral image classification. The method utilizes an agent to learn the accuracy of each sample label and determine its suitability to be used in training the classifier model. The agent is trained using a reinforcement signal based on the reward obtained from a small validation set that reflects performance on the target task, where the validation set is generated from the training set by a momentum-based recruiter. Compared to complex hand-crafted methods of label noise cleaning, HyperRefine is easy to implement, yet highly effective. The robustness of the model to different types of label noises is significantly improved by further combining a novel label refinement approach. Experiments on artificially added label noise and pseudo-label noise hyperspectral datasets show that the proposed method can achieve promising performance.

Which Is More Effective in Label Noise Cleaning, Correction or Filtering?

Most noise cleaning methods adopt one of the correction and filtering modes to build robust models. However, their effectiveness, applicability, and hyper-parameter insensitivity have not been carefully studied. We compare the two cleaning modes via a rebuilt error bound in noisy environments. At the dataset level, Theorem 5 implies that correction is more effective than filtering when the cleaned datasets have close noise rates. At the sample level, Theorem 6 indicates that confident label noises (large noise probabilities) are more suitable to be corrected, and unconfident noises (medium noise probabilities) should be filtered. Besides, an imperfect hyper-parameter may have fewer negative impacts on filtering than correction. Unlike existing methods with a single cleaning mode, the proposed Fusion cleaning framework of Correction and Filtering (FCF) combines the advantages of different modes to deal with diverse suspicious labels. Experimental results demonstrate that our FCF method can achieve state-of-the-art performance on benchmark datasets.

Abstract 7423: BATCH-FLEX: Feature-level equalization of x-batch in heterogeneous cancer data

Abstract Purpose: Develop a shiny application to help integrate cancer datasets and guide researchers in selecting an appropriate method of correction for their technical artifacts. Description: Integrative analysis of heterogeneous expression data remains challenging due to variations in platform, RNA quality, sample processing, and other unknown technical effects. As the field performs omics profiling of samples from cancer patients and murine models, there is the need for harmonizing, identifying, and correcting these technical effects to ensure robust analysis on the treatment and conditional effects of the underlying genetics or biological events. However, selecting and implementing different approaches for removing unwanted batch effects can be a time-consuming and tedious process, especially for more biologically focused investigators. In this project, we present Shiny BATCH-FLEX, a Shiny app to rapidly visualize batch correction by established batch correction methods such as ComBat, Mean Centering, ComBatSeq, and Limma RemoveBatchEffect. With BATCH-FLEX, users can visualize the contribution of variance of a factor before and after correction using principal component analysis, relative log expression plots, heatmaps, and explanatory variables. Users can also save all plots and matrices as a single ZIP file for further downstream analysis. Results: As a proof of concept, we assessed BATCH-FLEX using simulated data generated from a linear model framework introduced by Gagnon-Bartsch and Speed, which assumes that gene expression measurements can be distilled to a combination of the biological signal, systemic nose, and random noise. BATCH-FLEX was able to successfully identify and remove the introduced effect using each of the batch correction methods listed above. Next, we evaluated BATCH-FLEX using a comprehensive collection of bladder cancer data consisting of microarray data from 13 studies spanning 1452 samples. Following the cleaning of study-dependent noise, BATCH-FLEX was successful in revealing the heterogeneity among bladder cancer based on known sample type annotations. Conclusion: We have developed BATCH-FLEX, a tool for oncologic researchers to rapidly assess, select, and implement commonly used batch correction methods. This tool is available at https://github.com/shawlab-moffitt/BATCH-FLEX. Our integrative web portal of a Bladder Cancer Resource for Translational Science (BEACON) will also be shared at the meeting. Citation Format: Joshua Davis, Alyssa Obermayer, Thac Duong, Rebecca Hesterberg, Xuefeng Wang, Mingxiang Teng, G. Daniel Grass, Timothy Shaw. BATCH-FLEX: Feature-level equalization of x-batch in heterogeneous cancer data [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 1 (Regular Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(6_Suppl):Abstract nr 7423.

Generation of transmission wave with low AM noise for sub-GHz CW-EPR spectrometer

This study describes a technique to clean amplitude modulation (AM) noise of RF transmission waves, which is used to observe the sub-GHz CW-EPR spectrum. An RF transmitter amplifier that has the function of cleaning AM noise has been developed. Cleaning of the AM noise was owing to saturation of the output at the amplifier. Three stages of the amplifiers in series could effectively suppress the AM noise to about –176 dBc/Hz and –183 dBc/Hz at offset frequency of 10 kHz and 100 kHz, respectively at the carrier frequency of 750 MHz and the output power of 29 dBm. Since phase modulation (PM) noise is suppressed by phase sensitive detection, the AM noise in the transmission is dominant cause of the noise in the sub-GHz CW-EPR absorption spectrum using a reflection bridge, which depends on the quality factor of the resonator and the power of the RF transmission. The additive phase modulation (PM) noise of this amplifier was –171 dBc/Hz at an offset frequency of 100 kHz, which indicated that the frequency modulation (FM) of the transmission wave was not distorted with this amplifier. Therefore, conventional CW-EPR spectrometers that typically require FM for automatic frequency control or automatic tunning control can use this technique to increase sensitivity.

Research on Ship Main Engine Fuel Consumption Model With Data Integration and Noise Cleaning

Research on Ship Main Engine Fuel Consumption Model With Data Integration and Noise Cleaning

Hybrid resampling and weighted majority voting for multi-class anomaly detection on imbalanced malware and network traffic data

In a large skewed dataset, the data imbalance is severe and the classifier's accuracy is biased towards the majority class. Insufficient data makes it challenging for the classifier to learn the feature of the minority classes. Moreover, some existing techniques of binary classification cannot directly apply to multi-classification. Plain oversampling may generate redundant and irrelevant data for the minority classes while undersampling may eliminate too many features for the majority class. This paper proposes a combination of constrained undersampling, oversampling, noise cleaning, and a weighted majority voting classifier (WMVC) to detect multi-classification obfuscated malware via memory and network traffic anomalies. As a constraint on undersampling and oversampling, the proposed framework divides the total sample number by the class number to obtain the average class size. According to the constraint, one or more majority classes are down-sampled using Random Undersampling, while all the minority classes are up-sampled using Adaptive Synthetic Sampling Approach, which is followed by Tomek Link to remove the noisy data. Then a weighted majority voting classifier aggregated tree-based ensemble algorithms is designed and compared to the XGBoost and Convolutional Neural Network (CNN) classifiers and six state-of-the-art ensemble algorithms. The comparison results show that the performance of the classifiers on balanced data outperforms those on imbalanced data, and the WMVC outperforms XGBoost, CNN, and the six other ensemble algorithms. Our approach can alleviate the classifier's bias towards the majority class while improving its performance for the difficult minority class.

FCNet: Learning Noise-Free Features for Point Cloud Denoising

The acquisition of point clouds is usually accompanied by noise due to imperfect laser scanning or image-based reconstruction techniques. Deep learning-based methods have achieved impressive performance in point cloud denoising. However, the features captured by a denoising network from noisy point clouds are usually contaminated by noise during training. The feature noise will lead to the oscillation of back-propagated gradients, which interferes with parameter optimization and reduces the denoising performance. In this paper, we propose to explicitly clean up feature noise for point cloud denoising from two aspects: feature noise cleaning and network training. From the first aspect, we propose the feature clean network (FCNet for short) to explicitly clean up the feature noise. From the second aspect, we train FCNet by a teacher-student learning model to learn the noise-free features under the guidance of feature domain losses. Specifically, FCNet is designed with emphasis on two modules: non-local self-similarity (NSS) and weighted average pooling (WAP). NSS module smooths features through a non-local filter based on the inherent non-local self-similarity of point clouds. WAP module applies original weights calculated by the statistical outlier removal algorithm to suppress the feature noise induced by outliers. In the teacher-student learning model, we introduce a clean input using the noisy point and its clean neighbors. The teacher network accepts the clean input to capture noise-free features. The student network is trained to imitate the teacher network to learn noise-free features by minimizing the feature loss. The experiments on synthetic and real scanned point clouds show that FCNet outperforms state-of-the-art point cloud denoising methods.

HUMAN ACTION RECOGNITION: IMPROVING THE ACCURACY OF DEEP CONV-LSTM ARCHITECTURE THROUGH NOISE CLEANING PRIOR TO KEY FRAMES SELECTION

Recognizing human actions has numerous practical applications that can address issues and enhance the effectiveness and living standards across various domains. For solving the human action recognition task from videos, the Deep Conv-LSTM model architecture has been used on the UCF101 dataset. To preprocess the frames, a method has been proposed that integrates two algorithms: Noise Cleaning and Dissimilarity-Based Key Frame Selec¬t¬i¬o¬n (KFS). By employing these algorithms cohesively and effectively, image quality has been enhanced and unwanted data have been eliminated. The Uniform Frame Selection, Dissimilarity-Based KFS, and the proposed algorithm are evaluated, and their performances are compared based on the accuracy and data size reduction. The results show that the Dissimilarity-Based KFS algorithm outperforms the Uniform Frame Selection algorithm in accuracy by 2%, and the proposed method shows a 3% and 5% improvement in accuracy compared to the first two algorithms, respectively. Furthermore, the proposed algorithm reduces the data size by 26%, making it computationally efficient.

FORÇA EXPLOSIVA DOS MEMBROS INFERIORES NO EXERCÍCIO DE PULAR CORDA APLICADO ÀS CRIANÇAS E ADOLESCENTES

ABSTRACT Introduction: The physical health of children and adolescents is directly related to school physical training quality. The improvement of adult sports effectiveness results from a concise protocol with standardized exercises. Rope skipping exercise lacks kinematic studies applied to children and adolescents. Objective: Provide a theoretical reference for jump rope exercise training for children and adolescents. Methods: Eight children aged six to eight years old were recruited to jump at a frequency of 80 and 120 jumps per minute controlled by a metronome. The technique was divided into three stages: impulse, suspension, and landing. Control in the knee joint exoskeleton balance was investigated by computerized baropodometry with differentiation of rotational angular velocity by rotational angular acceleration via Butterworth filter. After noise cleaning and multiplication, an adjustable amplification factor returned to the inlet with inertial compensation torque. Results: Initially, the impulsing force is about three times the individual’s gravity. This value and the center of mass acceleration decrease in time. The initial force is proportional to the velocity of its decrease during the impulse, and the opposite occurs with the landing force. Higher intensity is observed when raising the frequency. Conclusion: The fluctuation in the force curve in the landing phase after 20 jumps indicates low muscle strength in lower limbs to maintain activity. A speed of 80 jumps per minute controlled by music in a playful environment is recommended for children. Evidence Level II; Therapeutic Studies – Investigating the results.

An adaptive data cleaning framework: a case study of the water quality monitoring system in China

Robust detection of patterns at low signal-to-noise ratios (SNR) is a fundamental challenge of analyzing high-frequency data, particularly in water quality monitoring. When conducting water quality data analysis, an indispensable step is to clean the data noise. In this paper, a new method named ADAPTIVE-EWT-MFE based on empirical wavelet transform (EWT) and multi-scale fuzzy entropy (MFE) is proposed to implement time series data cleaning. EWT-MFE can decompose the spectrum into different inherent mode functions (IMFs). According to different characteristics of the IMFs, an adaptive and adjustable parameter based on MFE, which reflects the intrinsic characteristics is introduced into the threshold function to improve the performance of noise-cleaning. Finally, this hybrid data cleaning method is designed to filter the high-frequency noise on the IMFs. Results show that our proposed method not only has great potential to improve the better noise-cleaning performance but also does not distort noise-free data.

Weakly-supervised anomaly detection in video surveillance via graph convolutional label noise cleaning

In this paper, we seek to solve the problem of video anomaly detection under the weakly-supervised setting. Different from previous works that usually deal with the problem in a multiple-instance learning manner, we formulate it as a fully-supervised learning task with label noises. Under this new perspective, we can fully leverage the advantages of well-designed action classifiers for anomaly detection as long as the label noises are cleaned. For this purpose, we devise a graph convolutional network for label noise cleaning, which integrates two crucial characteristics for anomaly analysis: feature similarity and temporal consistency. Supervised by both direct and indirect signals, the net propagates supervision information from high-confidence snippets to low-confidence ones and provides cleaned labels for action classifier training. In this way, we design an alternate learning strategy to progressively promote the discrimination of the action classifier. During the test phase, we directly utilize the learned action classifier for anomaly detection in an end-to-end fashion without any intermediate processing. We have conducted extensive experiments on various anomaly datasets of three scales with two main types of action classifiers, and achieved superior or comparable performances compared with state-of-the-art methods. Furthermore, we manually annotate the temporal duration of anomalies in the training data of UCF-Crime, and give out the upper-limit performance of our cleaner net. The annotation can also be used as the ground truth for studying anomaly detection models under multi-level supervisory signals, which will mitigate the present shortage of large-scale anomaly datasets.

РЕЧЕПОДОБНАЯ ПОМЕХА, СТОЙКАЯ К ШУМООЧИСТКЕ, КАК РЕЗУЛЬТАТ СКРЕМБЛИРОВАНИЯ ЗАЩИЩАЕМОЙ РЕЧИ

The aim of this article is to study the possibility of creating a speech-like interference (SLI) based on scrambling of the protected speech signal that is resistant to noise cleaning, restoration and reconstruction of distorted speech through processing of its spectrogram images, including the use of the speaker’s voice database. Methods: applied systems analysis, digital spectral-time analysis, digital processing of signals and images, image analysis of sonograms Results: it is proposed to use as the main indicator of the security of speech information the concept of potential speech intelligibility, methods for assessing potential intelligibility are determined. An algorithm of compensationtype digital noise cleaning using image analysis-synthesis technology has been developed. The requirements for the created SLI are formed, consisting in changing the spetral-temporal envelope of speech, removing («weakening») traces of the harmonic structure of speech, equalizing the capacities of the protected speech signal and the SLI masking it. The approaches to the formation of an SLI resistant to the procedures of digital noise cleaning and reconstruction of distorted sonograms are described, proposals are presented on the methodology for evaluating the effectiveness of interference through the indicator of potential intelligibility. The results obtained will allow to better understand the work of attacking tools, develop methods and means of effectively countering possible threats, and expand the capabilities of existing speech information protection systems. The scientific novelty consists in the development of criteria for evaluating the effectiveness of speech-like interference, taking into account the capabilities of the intruder to restore the intelligibility of distorted speech messages, proposed methods for the formation of scrambler-type SLI that meet such criteria, substantiated proposals to improve existing methods for assessing and monitoring the security of speech information from its leakage through technical channels.

Classification of Calligraphy Writing Types Using Convolutional Neural Network Method (CNN)

Calligraphy is the art of beautiful Arabic writing in which a series of letters are formed in appropriate proportions, maintaining distance and accuracy containing verses from the Qur'an or Hadith. There is a challenge to recognize the type of calligraphy using machine learning. This study aims to classify the types of calligraphy writing for ordinary people who do not understand the differences between each type of calligraphy writing. This study builds a model using the Convolutional Neural Network (CNN) method. The image used will go through a noise cleaning, resizing, and cropping process. This method is to carry out the process of classifying the type of calligraphy using a dataset consisting of 230 of 2 different types of calligraphy, namely the Naskhi and Riq'ah types. 80% is used as training data and 20% for test data. In the modeling process there are two convolutional layers and two MaxPooling layers followed by a Fully connected layer. The CNN modeling results used to test the built data have an average percentage result of 89% accuracy from the training data used. For further research, it can be developed with other types of calligraphy.

Label Noise Cleaning with an Adaptive Ensemble Method Based on Noise Detection Metric.

Real-world datasets are often contaminated with label noise; labeling is not a clear-cut process and reliable methods tend to be expensive or time-consuming. Depending on the learning technique used, such label noise is potentially harmful, requiring an increased size of the training set, making the trained model more complex and more prone to overfitting and yielding less accurate prediction. This work proposes a cleaning technique called the ensemble method based on the noise detection metric (ENDM). From the corrupted training set, an ensemble classifier is first learned and used to derive four metrics assessing the likelihood for a sample to be mislabeled. For each metric, three thresholds are set to maximize the classifying performance on a corrupted validation dataset when using three different ensemble classifiers, namely Bagging, AdaBoost and k-nearest neighbor (k-NN). These thresholds are used to identify and then either remove or correct the corrupted samples. The effectiveness of the ENDM is demonstrated in performing the classification of 15 public datasets. A comparative analysis is conducted concerning the homogeneous-ensembles-based majority vote method and consensus vote method, two popular ensemble-based label noise filters.

Foot 3D Reconstruction and Measurement using Depth Data

Background: The need for shoes with non-standard sizes is increasing, but this is not followed by the competence to measure the foot effectively. The high cost of such an instrument in the market has led to the development of a precise yet affordable measurement system.Objective: This research attempts to solve the measuring problem by employing an automatic instrument utilizing a depth image sensor that is available on the market at an affordable price.Methods: Data from several Realsense sensors that have been preprocessed are combined using transformation techniques and noise cleaning is performed afterward. Finally the 3D model of the foot is ready and hence the length and width can be obtained.Results: The experimental results show that the proposed method produces a measurement error of 0.351 cm in foot length, and 0.355 cm in foot width.Conclusion: The result shows that multiple angles of a static Realsense sensor can produce a good 3D foot model automatically. This proposed system configuration can reduce complexity as well as being an affordable solution.

Automatic A-line coronary plaque classification using combined deep learning and textural features in intravascular OCT images.

We developed a fully automated method for classifying A-line coronary plaques in intravascular optical coherence tomography images using combined deep learning and textural features. The proposed method was trained on 4,292 images from 48 pullbacks giving 80 manually labeled, volumes of interest. Preprocessing steps including guidewire/shadow removal, lumen boundary detection, pixel shifting, and noise reduction were employed. We built a convolutional neural network to extract the deep learning features from the preprocessed image. Traditional textural features were also extracted and combined with deep learning features. Feature selection was performed using the minimum redundancy maximum relevance method. Combined features were utilized as inputs for a random forest classifier. After classification, conditional random field (CRF) method was used for classification noise cleaning. We determined a sub-feature set with the most predictive power. With CRF noise cleaning, sensitivities/specificities were 82.2%/90.8% and 82.4%/89.2% for fibrolipidic and fibrocalcific classes, respectively, with good Dice coefficients. The classification noise cleaning step improved performance metrics by nearly 10-15%. The predicted en face classification maps of entire pullbacks agreed favorably to the manually labeled counterparts. Both assessments suggested that our automated measurements gave clinically relevant results. The proposed method is very promising with regards to both clinical treatment planning and research applications.

Coronary calcification segmentation in intravascular OCT images using deep learning: application to calcification scoring.

.Major calcifications are of great concern when performing percutaneous coronary interventions because they inhibit proper stent deployment. We created a comprehensive software to segment calcifications in intravascular optical coherence tomography (IVOCT) images and to calculate their impact using the stent-deployment calcification score, as reported by Fujino et al. We segmented the vascular lumen and calcifications using the pretrained SegNet, convolutional neural network, which was refined for our task. We cleaned segmentation results using conditional random field processing. We evaluated the method on manually annotated IVOCT volumes of interest (VOIs) without lesions and with calcifications, lipidous, or mixed lesions. The dataset included 48 VOIs taken from 34 clinical pullbacks, giving a total of 2640 in vivo images. Annotations were determined from consensus between two expert analysts. Keeping VOIs intact, we performed 10-fold cross-validation over all data. Following segmentation noise cleaning, we obtained sensitivities of , , and for calcified, lumen, and other tissue classes, respectively. From segmented regions, we automatically determined calcification depth, angle, and thickness attributes. Bland–Altman analysis suggested strong correlation between manually and automatically obtained lumen and calcification attributes. Agreement between manually and automatically obtained stent-deployment calcification scores was good (four of five lesions gave exact agreement). Results are encouraging and suggest our classification approach could be applied clinically for assessment and treatment planning of coronary calcification lesions.

Automated cleaning of identity label noise in a large face dataset with quality control

For face recognition, some very large-scale datasets are publicly available in recent years, which are usually collected from the Internet using search engines, and thus have many faces with wrong identity (ID) labels (outliers). Additionally, the face images in these datasets have different qualities because of uncontrolled situations. The authors propose a novel approach for cleaning the ID label error, handling face images in different qualities. The face ID labels cleaned by their method can train better models for low-quality face recognition since more low-quality images are correctly labelled for training a deep model. In their low-to-high-quality face verification experiments, the deep model trained on their cleaning results of MS-Celeb-1M.v1 face dataset outperforms the same model trained on the same dataset cleaned by the semantic bootstrapping method. They also apply their ID label cleaning method on a subset of the cross-age celebrity dataset (CACD) face dataset, in which their quality-based cleaning can deliver higher precision and recall than a previous method on detecting the ID label errors.

Automated A-line coronary plaque classification of intravascular optical coherence tomography images using handcrafted features and large datasets.

.We developed machine learning methods to identify fibrolipidic and fibrocalcific A-lines in intravascular optical coherence tomography (IVOCT) images using a comprehensive set of handcrafted features. We incorporated features developed in previous studies (e.g., optical attenuation and A-line peaks). In addition, we included vascular lumen morphology and three-dimensional (3-D) digital edge and texture features. Classification methods were developed using expansive datasets (), consisting of both clinical in-vivo images and an ex-vivo dataset, which was validated using 3-D cryo-imaging/histology. Conditional random field was used to perform 3-D classification noise cleaning of classification results. We tested various multiclass approaches, classifiers, and feature selection schemes and found that a three-class support vector machine with minimal-redundancy-maximal-relevance feature selection gave the best performance. We found that inclusion of our morphological and 3-D features improved overall classification accuracy. On a held-out test set consisting of images, we obtained an overall accuracy of 81.58%, with the following (sensitivity/specificity) for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The en-face views of classification results showed that automated classification easily captured the preponderance of a disease segment (e.g., a calcified segment had large regions of fibrocalcific classifications). Finally, we demonstrated proof-of-concept for streamlining A-line classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods, to enable fully automated plaque quantification. The results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and segmentation for live-time treatment planning and for offline assessment of drug and biologic therapeutics.