Noise Filtering Algorithm Research Articles

Blending is all you need: Data-centric ensemble synthetic data

Deep generative models have gained increasing popularity, particularly in fields such as natural language processing and computer vision. Recently, efforts have been made to extend these advanced algorithms to tabular data. While generative models have shown promising results in creating synthetic data, their high computational demands and the need for careful parameter tuning present significant challenges. This study investigates whether a collective integration of refined synthetic datasets from multiple models can achieve comparable or superior performance to that of a single, large generative model. To this end, we developed a Data-Centric Ensemble Synthetic Data model, leveraging principles of ensemble learning. Our approach involved a data refinement process applied to various synthetic datasets, systematically eliminating noise and ranking, selecting, and combining them to create an augmented, high-quality synthetic dataset. This approach improved both the quantity and quality of the data. Central to this process, we introduced the Ensemble k-Nearest Neighbors with Centroid Displacement (EKCD) algorithm for noise filtering, alongside a density score for ranking and selecting data. Our experiments confirmed the effectiveness of EKCD in removing noisy synthetic samples. Additionally, the ensemble model based on the refined synthetic data substantially enhanced the performance of machine learning models, sometimes even outperforming that of real data.

Defect monitoring system of the internal structures of a sodium fast reactor using an artificial intelligence model

This study developed a defect-monitoring system with an artificial intelligence model, YOLOv7, which is tailored for processing image data from an ultrasonic visualization system within sodium fast reactor (SFR) internal structures. For the safety of SFR internal structures, although it is a crucial inspection for defect monitoring, it is difficult to identify structural defects because of the invisible environment. Therefore, we applied the YOLOv7 model in this study; however, we encountered challenges including decreased accuracy with complex defect shapes and complications from data augmentation during pre-training. To solve these problems, we additionally applied the enhanced super-resolution generative adversarial network for higher resolution and a Sobel noise-filtering algorithm to enhance the defect detection accuracy. And we evaluated our system by comparing it with a confidence score. This underscores the effectiveness of the approach in enhancing the defect detection capabilities. Therefore, this defect-monitoring system should be designed to preemptively identify internal structure deformations and enhance SFR safety and maintenance practices.

Noise Filtering Algorithm Using Gaussian Mixture Models for High-Resolution Mass Spectra of Natural Organic Matter.

High-resolution mass spectra of natural organic matter (NOM) contain a large number of noise signals. These signals interfere with the correct molecular composition estimation during nontargeted analysis because formula-assignment programs find empirical formulas for such peaks as well. Previously proposed noise filtering methods that utilize the profile of the intensity distribution of mass spectrum peaks rely on a histogram to calculate the intensity threshold value. However, the histogram profile can vary depending on the user settings. In addition, these algorithms are not automated, so they are handled manually. To overcome the mentioned drawbacks, we propose a new algorithm for noise filtering in mass spectra. This filter is based on Gaussian Mixture Models (GMMs), a machine learning method to find the intensity threshold value. The algorithm is completely data-driven and eliminates the need to work with a histogram. It has no customizable parameters and automatically determines the noise level for each individual mass spectrum. The algorithm performance was tested on mass spectra of natural organic matter obtained by averaging a different number of microscans (transients), and the results were compared with other noise filters proposed in the literature. Finally, the effect of this noise filtering approach on the fraction of peaks with assigned formulas was investigated. It was shown that there is always an increase in the identification rate, but the magnitude of the effect changes with the number of microscans averaged. The increase can be as high as 15%.

A Multi-Level Auto-Adaptive Noise-Filtering Algorithm for Land ICESat-2 Photon-Counting Data

Due to atmospheric scattering, solar radiation, and other factors, the Ice, Cloud, and land Elevation Satellite-2 (ICESat-2) product data suffer from a substantial amount of background noise. This poses a significant challenge when attempting to directly utilize the raw data. Consequently, data denoising becomes an indispensable preprocessing step for its subsequent applications, such as the extraction of forest structure parameters and ground elevation data. While the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm is currently the most widely used method, it remains susceptible to complexities arising from terrain, low signal-to-noise ratio (SNR), and input parameter variations. This paper proposes an efficient Multi-Level Auto-Adaptive Noise Filter (MLANF) algorithm based on photon spatial density. Its purpose is to extract signal photons from ICESat-2 terrestrial data of different ground cover types. The algorithm follows a two-step process. Firstly, random noise photons are removed from the upper and lower regions of the signal photons through a coarse denoising process. Secondly, in the fine denoising step, the K-Nearest Neighbor (KNN) algorithm selects the K photons to calculate the slope along the track. The calculated slope is then used to rotate the direction of the searching neighborhood in the DBSCAN algorithm. The proposed algorithm was tested in eight datasets of four surface types: forest, grassland, desert, and urban, and the extraction results were compared with those from the ATL08 datasets and the DBSCAN algorithm. Based on the ground-truth signal photons obtained by visual inspection, the classification precision, recall, and F-score of our algorithm, as well as two other algorithms, were calculated. The MLANF could achieve a good balance between classification precision (97.48% averaged) and recall (97.96% averaged). Its F-score (97.69% averaged) was higher than that of the other two methods. This demonstrates that the MLANF algorithm successfully obtained a continuous surface profile from ICESat-2 datasets with different surface cover types, significant topographic relief, and low SNR.

Prediction of optimal 3D printing orientation using vertically sparse voxelization and modified support structure tomography

PurposeIncreasingly 3D printing is used for parts of garments or for making whole garments due to their flexibility and comfort and for functionalizing or enhancing the aesthetics of the final garment and hence adding value. Many of these applications rely on complex programming of the 3D printer and are usually provided by the vendor company. This paper introduces a simpler, easier platform for designing 3D-printed textiles, garments and other artifacts, by predicting the optimal orientation of the target objects to minimize the use of plastic filaments.Design/methodology/approachThe main idea is based on the shadow-casting analogy, which assumes that the volume of the support structure is similar to that of the shadow from virtual sunlight. The triangular elements of the target object are converted into 3D pixels with integer-based normal vectors and real-numbered coordinates via vertically sparse voxelization. The pixels are classified into several groups and their noise is suppressed using a specially designed noise-filtering algorithm called slot pairing. The final support structure volume information was rendered as a two-dimensional (2D) figure, similar to a medical X-ray image. Thus, the authors named their method modified support structure tomography.FindingsThe study algorithm showed an error range of no more than 1.6% with exact volumes and 6.8% with slicing software. Moreover, the calculation time is only several minutes for tens of thousands of mesh triangles. The algorithm was verified for several meshes, including the cone, sphere, Stanford bunny and human manikin.Originality/valueSimple hardware, such as a CPU, embedded system, Arduino or Raspberry Pi, can be used. This requires much less computational resources compared with the conventional g-code generation. Also, the global and local support structure is represented both quantitatively and graphically via tomographs.

Real-Time Response Estimation of Structural Vibration with Inverse Force Identification

Virtual sensing is an effective method to identify the inaccessible state of the structural systems by compensating the limitations of the conventional physical sensing techniques. Recently, it becomes popular in structural vibration field and their relevant physical domains such as civil, mechanical, aerospace engineering, thermal dynamics, and acoustics. This study aimed to develop a virtual sensing algorithm of structural vibration for the real-time identification of unmeasured information. First, certain local point vibration responses (such as displacement and acceleration) are measured using physical sensors, and the data sets are extended using a numerical model to cover the unmeasured quantities through the entire spatial domain in the real-time computation process. A modified time integrator is then proposed to synchronize the physical sensors and the numerical model using inverse dynamics. In particular, an efficient inverse force identification method is derived using implicit time integration. The second-order ordinary differential formulation and its projection-based reduced-order modeling are used to avoid two times larger degrees of freedom within the state-space form. Then, the Tikhonov regularization noise-filtering algorithm is employed instead of Kalman filtering. The performance of the proposed method is investigated on both numerical and experimental testbeds under sinusoidal and random excitation loading conditions. In the numerical test, the system could identify the status of the motor housing structure in the speed of 16,181 S a m p l e s / s . The FDE and RMSE values are bounded under 0.1816 and 1503.2. In the case of the experimental test, the algorithm is implemented to the beam structure using a single-board computer, including inverse force identification and unmeasured response prediction. Even in the limited computational environment, the system could identify the applied forces in real time in the speed of 2,173 S a m p l e s / s . Through all experimental cases, FDE and RMSE values are bounded under 0.2925 and 0.1660. The results show that the virtual sensing algorithm can accurately identify unmeasured information, forces, and displacements throughout the vibration model in real time in a very limited computing environment.

Simplified inelastic electron tunneling spectroscopy based on low-noise derivatives

A standard experimental setup for Inelastic Electron Tunneling Spectroscopy (IETS) performs the measurement of the second derivative of the current with respect to the voltage (d^2I/dV^2) using a small AC signal and a lock-in based second harmonic detection. This avoids noise arising from direct differentiation of the current-voltage characteristics (I–V) by standard numerical methods. Here we demonstrate a noise-filtering algorithm based on Tikhonov Regularization to obtain IET spectra (i.e. d^2I/dV^2 vs. V) from measured DC I–V curves. This leads to a simple and effective numerical method for IETS extraction. We apply the algorithm to I–V data from a molecular junction and a metal-insulator-semiconductor tunneling device, demonstrating that the computed first/second derivatives have a workable match with those obtained from our lock-in measurements; the computed IET spectral peaks also correlate well with reported experimental ones. Finally, we present a scheme for automated tuning of the algorithm parameters well-suited for the use of this numerical protocol in real applications.

Improved noise-filtering algorithm for AdaBoost using the inter-and intra-class variability of imbalanced datasets

Boosting methods are known to increase performance outcomes by using multiple learners connected sequentially. In particular, Adaptive boosting (AdaBoost) has been widely used owing to its comparatively improved predictive results for hard-to-learn samples based on misclassification costs. Each weak learner minimizes the expected risk by assigning high misclassification costs to suspect samples. The performance of AdaBoost depends on the distribution of noise samples because the algorithm tends to overfit noisy samples. Various studies have been conducted to address the noise sensitivity issue. Noise-filtering methods used in AdaBoost remove samples defined as noise based on the degree of misclassification to prevent overfitting to noisy samples. However, if the difference in the classification difficulty between classes is considerable, it is easy for samples from classes that are difficult to classify to be defined as noise. This situation is common with imbalanced datasets and can adversely affect performance outcomes. To solve this problem, this study proposes a new noise detection algorithm for AdaBoost that considers differences in the classification difficulty of classes and the characteristics of iteratively recalculated sample weight distributions. Experimental results on ten imbalanced datasets with various degrees of imbalanced ratios demonstrate that the proposed method defines noisy samples properly and improves the overall performance of AdaBoost.

Detection of leek white tip disease under field conditions using hyperspectral proximal sensing and supervised machine learning

Leek white tip disease is one of the key yield-limiting factors in leek production. It leads to socio-economic and environmental costs due to the need for fungicide applications. Precision agriculture aims at reducing these costs, while maintaining or improving farmer revenues. To be able to apply variable rate fungicides, in-field disease detection is necessary. In this work, a disease detection methodology was created that can detect early, pre-symptomatic white tip disease symptoms in field conditions with millimetre resolution. A hyperspectral training library was constructed containing 29,744 spectra of healthy leek, weed plants, diseased leek plants and soil. Soil pixels were removed from hyperspectral images by means of LDA classification, followed by a custom noise filter algorithm. Then, a logistic regression supervised machine learning classifier was trained with five classes: healthy plant material, early (pre-visual) disease, moderate disease, severe disease and fully developed disease. The overall accuracy of the disease detection model was 96.74%. Model diagnostics showed a low likelihood of classifying diseased pixels as healthy (3.59%) and a low likelihood of falsely classifying healthy pixels as diseased (0.77%).

De-noising and enhancement of MRI medical images using Gaussian filter and histogram equalization

Noise in Medical Images reduces the image contrast, resolution, and is observed as an obstacle in image understanding. MRI is used for the treatment of brain, ankle, foot, and prostate cancers. Normally, MR Images consist of Gaussian Noise, Poisson Noise, Speckle Noise, Salt and Pepper Noise, etc. In Medical Image Processing, removing noise and obtaining information from MR Images is a crucial and challenging task. An accurate image is very important for further diagnosis or post-processing tasks such as visualization, segmentation, etc. For removing different types of noise from MR images, different noise filtering algorithms are used. Selecting a particular filtering method is application dependent. Gaussian Filter is the best technique for de-noising of medical images efficiently. To reveal the detailed information hidden in a very small range of grey level or color level from a low contrast image, a powerful tool such as a contrast enhancement algorithm is used for image enhancement to increase the visual appearance or quality of the image. Before Medical Image Analysis, improving the contrast of Medical Images is necessary using contrast enhancement methods. Histogram Equalization (HE) is an important and widely used technique for enhancing the contrast of an image among the different contrast enhancement methods. This paper explains the method of de-noising and image contrast enhancement using Gaussian filter and HE and performed simulations using different databases such as IRMA, KAGGLE, and WANG, etc. and observed that Gaussian filter is used to reduce Gaussian noise efficiently for removal of Gaussian noise by smoothing inner distinct image region and also in Histogram Equalization, the histogram of the resultant image is flat.

A Novel automatic voice recognition system based on text-independent in a noisy environment

Automatic voice recognition system aims to limit fraudulent access to sensitive areas as labs. Our primary objective of this paper is to increase the accuracy of the voice recognition in noisy environment of the Microsoft Research (MSR) identity toolbox. The proposed system enabled the user to speak into the microphone then it will match unknown voice with other human voices existing in the database using a statistical model, in order to grant or deny access to the system. The voice recognition was done in two steps: training and testing. During the training a Universal Background Model as well as a Gaussian Mixtures Model: GMM-UBM models are calculated based on different sentences pronounced by the human voice (s) used to record the training data. Then the testing of voice signal in noisy environment calculated the Log-Likelihood Ratio of the GMM-UBM models in order to classify user's voice. However, before testing noise and de-noise methods were applied, we investigated different MFCC features of the voice to determine the best feature possible as well as noise filter algorithm that subsequently improved the performance of the automatic voice recognition system.

Noise filtering in atomistic stress calculations for crystalline materials

Spatially-varying stress fields can be obtained from atomistic simulations as weighted averages over a phase function that depends on the positions and momenta of the atoms and the interatomic forces between them. However atomistic stress fields exhibit significant nonphysical noise on atomic length scales even under uniform loading conditions. This makes it difficult to obtain accurate stress estimates near atomic-scale defects such as nanocrack tips and dislocation cores. To address this issue, we develop an algorithm to filter noise in the atomistic stress for crystalline materials based on a rigorous stress-invariance condition, which leads to a new class of lattice-dependent weighting functions. Stress fields computed using these weighting functions are identically noise-free under uniform conditions, and have greatly reduced noise in general. The method is demonstrated for three example problems: (1) uniform loading of nickel aluminide, (2) a mode I crack in silicon, and (3) screw and edge dislocation cores in aluminum. The noise filtering algorithm is implemented in MDStressLab++, an open source C++ library for computing atomistic stress fields available online at http://mdstresslab.org.

REVIEW OF NOISE FILTERING ALGORITHM FOR PHOTON DATA

Abstract. As a continuation of Ice, Cloud, and Land Elevation Satellite-1 (ICESat-1)/Geoscience Laser Altimeter System (GLAS), the Ice, Cloud, and Land Elevation Satellite-2 (ICESat-2) , which is equipped with the Advanced Topographic Laser Altimeter (ATLAS) system, was successfully launched in 2018. Since ICESat-1/GLAS has facilitated scientific results in the field of forest structure parameter estimation, how to use the ICESat-2/ATLAS photon cloud data to estimate forest structure parameters has become a hotspot in the field of spaceborne photon data application. However, due to the weak photon characteristics of the ICESat-2/ATLAS system, the system is extremely susceptible to noise, which poses a challenge for its subsequent accurate estimation of forest structural parameters. Aiming to filter out the noise photons, the paper introduces the advantages of the spaceborne lidar system ICESat-2/ATLAS than ICESat-1/GLAS. The paper summarizes the research of the simulated photon-counting lidar (PCL) noise filtering algorithm and noise filtering on spaceborne.

基于连通性检测的图像椒盐噪声滤波算法

基于连通性检测的图像椒盐噪声滤波算法

Adaptive Edge Detection Algorithm Based on Improved Grey Prediction Model

Existing edge detection algorithms suffer from inefficient edge localization, noise sensitivity, and/or relatively poor automatic detection capability. Contemporary edge detection algorithms can be improved by targeting these problems to help bolster their performance. Grey system theory can be used to resolve the small data and poor information issues in the local information of uncertain systems. An automatic edge detection algorithm was developed in this study based on a grey prediction model to remedy these problems. Noise characteristics in grey images are used to deploy a noise-filtering algorithm based on local features. A mask with twenty-four edge direction information points (345°) was established based on edge line texture features. By compressing the amplitude of the sequence, the randomly oscillated grey prediction sequence can be converted into a smooth, new sequence. The discrete grey model (1,1) (DGM(1,1)) was established based on this new grey prediction sequence to obtain the grey prediction maximum value. A grey prediction image with enhanced edges was obtained by replacing the pixel value in the original image with the maximum grey prediction value. A grey prediction subtraction image with edges separated from non-edge points was also obtained by subtracting the original image from the grey prediction image. The optimal separation threshold in the grey prediction subtraction image can be determined via the global adaptive threshold selection method. The neighborhood search method was then deployed to remove stray points and burrs from the image after the target was separated from the background, creating the final edge image. Experiments were performed on a computer-simulated phantom to find that both the subjective visual effects and objective evaluation criteria are better under the proposed method than several other competitive methods. The proposed edge detection algorithm shows excellent edge detection ability and is highly robust to noise, though the grey prediction model needs further improvement to optimize the run time.

Calculating the average distance between horizontal projection peaks

During the analysis and construction of the noise filtering algorithm at the stage of segmentation of character strings, the need to describe a special database or dictionary for storing model and skeletal projections of alphabet characters became clear. However, when formatting entries for alphabetical projection in the dictionary, the following questions arose: how many projection values are needed to describe a single character, and also how this value may change depending on the number of strings analyzed and their characters. The objects of research in the article are the vertical projection of the characters, as well as their width of the projection segments. The subject of research is the change of the average width of the projection segment for a certain type of license plates (character string). The main goal is to calculate and justify the average size of the segment. Data about the average width of the projection segment that describes a character allows to determine the number of projection values (coordinates along the ox axis) to store the projection values of this character from an alphabet. Several language alphabets are considered, where each of them is associated with a unique type. In particular, license plates of the Republic of Belarus and the Slovak Republic are considered. Using an elementary statistical apparatus, calculations and analysis of samples were carried out, taking into account the case of their high variation. As a result, the average values of the segment width and the variance of the segment width were obtained using two types of alphabets as an example. In research an algorithm for using the obtained values in the formation of model projection records is presented. The algorithm takes into account «special» cases of going beyond the segment boundaries. The described steps and calculations can be applied to a larger number of alphabets, which indicates the possibility of describing model projection dictionaries for them, with subsequent widespread use of character strings in noise filtering.

Theory and Statistical Description of the Enhanced Multi-Temporal InSAR (E-MTInSAR) Noise-Filtering Algorithm

In this work, the statistical fundaments of the recently proposed enhanced, multi-temporal interferometric synthetic aperture radar (InSAR) noise-filtering (E-MTInSAR) technique is addressed. The adopted noise-filtering algorithm is incorporated into the improved extended Minimum Cost Flow (EMCF) Small Baseline Subset (SBAS) differential interferometric SAR (InSAR) processing chain, which has extensively been used for the generation of Earth’s surface displacement time-series in several different contexts. Originally, the input of the InSAR EMCF-SBAS processing toolbox consisted of a sequence of multi-looked, small baseline interferograms, which were unwrapped using the space-time EMCF phase unwrapping algorithm. Subsequently, the unwrapped interferograms were inverted through the SBAS algorithm to retrieve the expected InSAR deformation products. The improved processing chain has complemented the original codes with two additional steps. In particular, a new multi-temporal noise-filtering algorithm for sequences of time-redundant multi-looked DInSAR interferograms, followed by a proper interferogram selection step, has been proposed. This research study is aimed at primarily assessing the performance of the E-MTInSAR noise-filtering algorithm from a theoretical perspective. To this aim, the principles of directional statistics and errors propagation are exploited. Experimental results, carried out by applying the E-MTInSAR algorithm to a sequence of SAR data collected over the Los Angeles bay area, have been used to corroborate the academic outcome of this research.

基于开关二级检测的图像椒盐噪声滤波算法

基于开关二级检测的图像椒盐噪声滤波算法

Application of machine vision technology in geometric dimension measurement of small parts

In this paper, the on-line detection of small parts’ dimension measurement based on machine vision is designed, and the key technologies, such as image processing, image registration and stitching, edge detection, sub-pixel location analysis, image feature recognition and clustering, and scale measurement based on image involved in the detection of small part dimension are studied. Firstly, based on the actual usage and the characteristics of the algorithm, the feature-based SIFT algorithm was selected to complete the image registration, and the image edge detection algorithm and data processing method were explored. The histogram equalization improves the grayscale distribution of the stitched image and improves the contrast of the image. The median noise filtering algorithm was used to complete the image noise reduction. The false edge was filtered to get the single pixel edge, and the least square method was used to compensate the missing edge pixels and reduce the measurement error. An accurate image registration transformation matrix was obtained. Then, the weighted average fusion algorithm was used to complete the image fusion. The experimental results show that the image stitching algorithm is accurate and effective, and the measurement accuracy of the system meets the performance requirements.

Operational Anomaly Detection in Flight Data Using a Multivariate Gaussian Mixture Model

This paper presents a robust real-time aircraft health monitoring framework using a machine learning based approach, specifically the multivariate Gaussian mixture model (mGMM), for the detection of in-air operational anomalies of an aircraft system. Sensor fusion and noise filtering algorithms have also been adopted to reduce dimensionality of the feature space while avoiding the elimination of useful information from the original flight data. Random noise in each feature, induced by the aircraft sensors and data acquisition system, is filtered out using a weighted averaging window while maintaining inherent variances. The filtered dataset is then fused according to the underlying physics of each sensed feature to reduce redundant features and subsequently trained using the mGMM. The methodology allows monitoring the behavior of each feature as well as correlations between features, significantly improving detection sensitivity. The high computational efficiency of this approach permits real-time monitoring of an aircraft system.