Noise Detection Method Research Articles

Synthetic Minority Oversampling Technique Based on Adaptive Noise Optimization and Fast Search for Local Sets for Random Forest

The classification is usually degraded due to the imbalanced class distribution. Synthetic minority oversampling technique (SMOTE) has been successful in improving imbalanced classification and has received great praise. Overgeneralization is one of the most challenges in SMOTE. Although multiple SMOTE-based variations are proposed against overgeneralization, they still have the following shortcomings: (a) creating too many synthetic samples in high-density regions; (b) removing suspicious noise directly instead of modifying them; (c) relying on many parameters. This paper proposes a new SMOTE based on adaptive noise optimization and fast search for local sets (SMOTE-ANO-FLS) to overcome the overgeneralization and the shortcomings of existing works. First, SMOTE-ANO-FLS uses the [Formula: see text]-D tree to fast search the local sets for each sample. Second, a new noise detection method based on local sets and the imbalanced ratio is proposed to detect suspicious noise. Third, a new adaptive noise optimization method is proposed to modify detected suspicious noise instead of removing them. Finally, a new probability weight based on local sets is proposed to help create more synthetic minority class samples in borderline and sparse regions. The effectiveness of SMOTE-ANO-FLS is proven by employing 7 oversampling methods and random forest on the extensive synthetic and real data sets.

Adaptive Noise Detector and Partition Filter for Image Restoration

The random-value impulse noise (RVIN) detection approach in image denoising, which is dependent on manually defined detection thresholds or local window information, does not have strong generalization performance and cannot successfully cope with damaged pictures with high noise levels. The fusion of the K-means clustering approach in the noise detection stage is reviewed in this research, and the internal relationship between the flat region and the detail area of the damaged picture is thoroughly explored to suggest an unique two-stage method for gray image denoising. Based on the concept of pixel clustering and grouping, all pixels in the damaged picture are separated into various groups based on gray distance similarity features, and the best detection threshold of each group is solved to identify the noise. In the noise reduction step, a partition decision filter based on the gray value characteristics of pixels in the flat and detail areas is given. For the noise pixels in flat and detail areas, local consensus index (LCI) weighted filter and edge direction filter are designed respectively to recover the pixels damaged by the RVIN. The experimental results show that the accuracy of the proposed noise detection method is more than 90%, and is superior to most mainstream methods. At the same time, the proposed filtering method not only has good noise reduction and generalization performance for natural images and medical images with medium and high noise but also is superior to other advanced filtering technologies in visual effect and objective quality evaluation.

Autoencoder-Based Recommender System Exploiting Natural Noise Removal

Collaborative filtering (CF) is a widely used technique in recommender systems by automatically predicting the user’s latent interests based on many users’ historical rating data. To improve the performance of the CF-based recommender systems, users’ rating data should be pre-processed to avoid noise and enhance data reliability. Many researchers studied anomaly detection to remove malicious noise caused by shilling attacks, but anomalies can still exist in non-attacked real user data, which is called natural noise, as the ratings of users can be impacted by unpredictable factors such as other users’ ratings and anchoring bias. In this paper, we propose an autoencoder-based recommendation system for exploiting the ability of both anomaly detection and CF. The proposed system detects the natural noise in the rating data based on the reconstruction errors after training. By removing the detected natural noise, CF can predict the unrated ratings with noise-free data. Our experiments show that the proposed model showed better performance than the traditional method by reducing the error by up to 5% compared to the method that does not consider natural noise detection and reducing the error by up to 4% compared to the conventional rating classification based natural noise detection methods.

Window filtering algorithm for a low repetition rate pulsed laser coherent combination system.

The multi-dithering method has been well verified in the phase-locking of polarization coherent combination experiments. However, it is difficult to apply to low repetition rate pulsed laser coherent combination, since there exists an overlap in the frequency domain between the pulse laser and the large amplitude-phase noise resulting in traditional filters being unable to effectively separate the phase noise. Aiming to solve the problem, we propose, to the best of our knowledge, a novel method of pulse noise detection, identification, and filtering based on the autocorrelation characteristics between noise signals. The self-designed adaptive window filtering algorithm can effectively filter the pulse signal doped in the phase noise around 0.1 ms. After the pulses are filtered out, the remaining phase noise signal is used as the input signal of the multi-dithering method for phase locking; the phase difference of two pulsed beams (10 kHz) is successfully compensated to zero; and the coherent combination of the closed-loop phase lock is realized. Simultaneously, the phase correction periods are short, the phase lock effect is stable, and the intensity of the final combined pulses rises to the ideal value (0.9Imax). In addition, the adaptive window filtering algorithm we proposed can be applied to the coherent combined system of large array fiber lasers and further lay the foundation for fiber phased array lidar.

Evaluation of the Efficacy of Artificial Neural Network-Based Music Therapy for Depression.

In order to evaluate the therapeutic effect of music therapy on patients with depression, this paper proposes a CNN-based noise detection method with the combination of HHT and FastICA for noise removal, with good data support from the DBN model. DBN-based feature extraction and classification are completed. As the training process of DBN itself requires a large number of training samples, there are also disadvantages such as slow convergence speed and easy to fall into local minima, which lead to a large amount of effort and time, and the learning efficiency is relatively low. A DBN optimization algorithm based on artificial neural network was proposed to evaluate the efficacy of music therapy. First of all, through the comparison of music therapy experimental group and control group, to verify that music therapy is effective for the treatment of depressed patients. Secondly, we propose to optimize the selection of features based on the frequency band energy ratio and the sliding average sample entropy, respectively, and then to classify the EEG of depressed patients under different music perceptions by training the DBN model and continuously adjusting the parameters, combined with the surtax classifier, and the classification accuracy is high. In particular, it can detect the different effects of different music styles, which is of great significance for the selection of appropriate music for the treatment of depressed patients.

PTrustE: A high-accuracy knowledge graph noise detection method based on path trustworthiness and triple embedding

Many knowledge graphs (KGs) constructed automatically or manually contain noises inevitably when adding heterogeneous data nowadays. The existing methods either have difficulty in detecting noisy triples with conflicted relations or ignore to represent triples after noise elimination, which creates obstacles for the downstream tasks to manipulate KGs. To detect various types of noises in KGs and constitute noise-free triples representation, this paper proposes a high-accuracy KG noise detection method based on path trustworthiness and triple embedding (PTrustE). First, PTrustE constructs a correlation-based path trustworthiness network to learn the global and local features in the path from the head entity to the tail entity of the triple. Next, PTrustE integrates all features of the path into the Bi-directional Gated Recurrent Unit to learn the path score matrix and path trustworthiness, to keep the sequential nature of paths. Finally, PTrustE uses the path score matrix for triple representation learning and the path trustworthiness for judging whether the triple is correct or not. Extensive experiments validate the superior performance of PTrustE. Compared with the second-placed INDIGO, PTrustE offers a 7.1% increase in fMRR on FB15K.

An Adaptive Method for Detecting and Removing EEG Noise

To solve the problem of real-time detection and removal of EEG signal noise in anesthesia depth monitoring, we proposed an adaptive EEG signal noise detection and removal method. This method uses discrete wavelet transform to extract the low-frequency energy and high-frequency energy of a segment of EEG signals, and sets two sets of thresholds for the low-frequency band and high-frequency band of the EEG signal. These two sets of thresholds can be updated adaptively according to the energy situation of the most recent EEG signal. Finally, we judge the level of signal interference according to the range of low-frequency energy and high-frequency energy, and perform corresponding denoising processing. The results show that the method can more accurately detect and remove the noise interference in the EEG signal, and improve the stability of the calculated characteristic parameters.

A 1/f Noise Detection Method for IGBT Devices Based on PSO-VMD

The generation of 1/f noise is closely related to the quality defects of IGBT devices. In the process of detecting single-tube noise of IGBT, thermal noise and shot noise show obvious white noise characteristics in the low-frequency range. This paper investigates how to accurately detect the 1/f noise under strong white noise, and thus proposes a particle swarm optimization method known as variational mode decomposition. First, the particle swarm optimization algorithm was used twice to search the optimal parameter combination between the penalty parameter and the decomposition modulus of the VMD model. Then, the parameters of the variational mode decomposition algorithm were set in optimal parameter combination. The frequency center and bandwidth of each IMF component were determined by continuous iteration in the variational framework. Finally, the 1/f noise signal was adaptively separated from background noise. Extensive experimental investigations carried out under different signal-to-noise ratios, compared with the optimal wavelet denoising algorithm, revealed that the PSO-VMD algorithm improved the signal-to-noise ratio by 6.6%, 16.82%, and 42.48%, whereas the mean square error is reduced by 7.12%, 19.80%, and 33.76%.

SOFTWARE AND HARDWARE FOR DETERMINING GAUSSIAN NOISE LEVEL IN IMAGES

Accurate determination of the noise level in digital images is required to obtain their maximum signal-to-noise ratio, which is a necessary condition for the effective performance of the following stages of image processing: visualization, segmentation, recognition, etc. The task of calculating the Gaussian noise level is quite common, because such noise appears in most experimental images taken by video cameras. However, existing high-speed noise detection methods have a significant error, and the fairly accurate LLROI method has a low speed. The LLROI method is based on Low-frequency filtering of the noise component and Low-frequency filtering when selecting the Region Of Interest (ROI). Therefore, it is proposed to determine the level of Gaussian noise by the exact LLROI method and to increase its speed by appropriate hardware and software. Based on the LLROI method, a program in the MATLAB system was created, the structure and Simulink-model of a computer system for determining of Gaussian noise level on digital images were synthesized. Hardware implementation of image filtering units is made by FPGA Artix-7, which allowed us to increase the speed of the system. The results of calculating the Gaussian noise level for test images by the LLROI method using the developed hardware and software proved the errors not to exceed those provided by analogous methods.
 The scientific novelty of the paper is to improve the LLROI method, namely to refine the threshold coefficient, which reduces the errors of calculating the noise level, even for images with clear contours and pronounced textures.
 The practical significance of the developed tools is that they can be used to build high-speed computer systems (or subsystems) designed to increase the signal-to-noise ratio on digital images.

Development of denoising and compression algorithms for AIS-based vessel trajectories

Ship navigation information derived from the Automatic Identification System (AIS) is widely used in the shipping industry. As AIS reports are featured with huge volume and noises, preprocessing of AIS data is essential before its further application. This study aims to develop effective algorithms to denoise and compress raw trajectories derived from AIS reports. Specifically, an effective noise detection method based on statistical theory and sliding window is first proposed to identify glitches in a given trajectory. Linear interpolation is further used to rectify the glitches detected. Then, two modes of algorithms are proposed for trajectory compression: static mode with preset threshold for compression and dynamic mode considering the distance between trajectory points and the coastline in a real-time manner. Numerical experiments show that the noise detection and rectification algorithms and the trajectory compression algorithms are accurate and highly efficient considering compression rate, information loss, and computation time. The main innovation of this research includes developing an accurate and robust trajectory glitch detection and rectification algorithm, proposing two modes of trajectory compression algorithms, and combining the two tasks in a holistic robust framework. Especially, the dynamic compression mode can overcome a major problem encountered in static compression where the compressed trajectories may go across land. It can also deal with more flexible compression requirements and thus should be more applicable in practice.

A novel SMOTE-based resampling technique trough noise detection and the boosting procedure

Most of the classification methods assume that the numbers of class observations are balanced. In such cases, models are predicted by giving biased weight to the the class with more observations. Therefore, the classifiers ignore the class with smaller number of observations and the majority class makes biased predictions. There are some advised performance measures to be used in datasets, as well as recommended approaches to solve class imbalance problem. One of the most widely used methods is resampling method. In this study, the difficulties relevant to random oversampling (ROS) and synthetic minority oversampling technique (SMOTE), which are some of the oversampling methods, are discussed. This study aims to propose a combination of a new noise detection method and SMOTE to overcome those difficulties. Using the boosting procedure in ensemble algorithms, noise detection is possible with the proposed SMOTE with boosting (SMOTEWB) method, which makes use of this information to determine the appropriate number of neighbors for each observation within SMOTE algorithm.

New Robust Regression Method for Outliers and Heavy Sparse Noise Detection via Affine Transformation for Head Pose Estimation and Image Reconstruction in Highly Complex and Correlated Data: Applications in Signal Processing

In this work, we propose a novel method for head pose estimation and face recovery, particularly to solve the potential impacts of noises in signal processing to get an efficient and effective model that is more resilient with annoying effects through adding affine transformation with the low-rank robust subspace regression. Consequently, the corrupted images can be correctly recovered by affine transformations to render more best regression outcomes. Thereby, we need to search so as to get optimal parameters which can be regarded as convex constrained optimization techniques. Afterward, the alternating direction method for multipliers (ADMM) approach is considered and a new set of updated equations is well established so as to update the optimization parameters and affine transformations iteratively in a round-robin manner. Additionally, the convergence of these new updating equations is well scrutinized as well. Thus, the experimental simulations reveal that the proposed method outperforms the state-of-the-art works for head pose estimation and face recovery on some public databases.

Implementasi Pengurangan Noise Pada Citra Rontgen Paru Menggunakan Metode Filter Adaptive-Hierarchical

Salt & Paper Noise Is A Form Of Noise That Is Used To See Black Or White Dots On An Image The Cause Is An Error Or Damage To The Pixels In The Image. X-ray or radiography is the use of ionizing rays to form the image of the object being studied on film, radiography is generally used to see objects that are not transparent, for example, the inside of the human body. X-ray images often have unclear results, because the image is taken using X-Ray rays, the x-ray results usually have a black and white color, and can be clearly seen when placed right in front of a light source. The light source only helps to see the shadow of the image, but does not improve the quality of the image and reduce noise or dots in the image. For this reason, the author intends to try to implement the X-ray results using the Adaptive-Hierarchical filter method with the aim of eliminating noise or dots in the X-ray image and clarifying the existing image shadows. With the Adaptive-Hierarchical Filter Method I Use Two Noise Detection Methods Where Is Detection For Noise In Medium And Smooth Areas Where Noise Detects In Rusal Or Salt Pixels And Paper Detects For Noise In Strong Areas Or You Can Call It Edge Detection On Noise On The Edges Using Only Pixels That Have Been Defined

Reflective Noise Filtering of Large-Scale Point Cloud Using Transformer

Point clouds acquired with LiDAR are widely adopted in various fields, such as three-dimensional (3D) reconstruction, autonomous driving, and robotics. However, the high-density point cloud of large scenes captured with Lidar usually contains a large number of virtual points generated by the specular reflections of reflective materials, such as glass. When applying such large-scale high-density point clouds, reflection noise may have a significant impact on 3D reconstruction and other related techniques. In this study, we propose a method that uses deep learning and multi-position sensor comparison method to remove noise due to reflections from high-density point clouds in large scenes. The proposed method converts large-scale high-density point clouds into a range image and subsequently uses a deep learning method and multi-position sensor comparison method for noise detection. This alleviates the limitation of the deep learning networks, specifically their inability to handle large-scale high-density point clouds. The experimental results show that the proposed algorithm can effectively detect and remove noise due to reflection.

Noise Point Detection From Airborne LiDAR Point Cloud Based on Spatial Hierarchical Directional Relationship

In three-dimensional (3D) airborne light detection and ranging (LiDAR) point-cloud data acquisition, noise point clusters (such as cloud, birds and incomplete scanning ground points) and isolated points are usually generated in the scanning process. Detection and elimination of these noise points directly affect the subsequent processing efficiency of the LiDAR point clouds. In this paper, a noise detection method from airborne LiDAR data based on spatial hierarchical directional relationship and region growing algorithm is proposed. First, the original airborne LiDAR points are divided into regular 3D grids, and the maximum point density unit is searched adaptively to select the initial surface seed points for region growing algorithm. Then, the spherical neighborhood is constructed with the initial seed point as the center, and fourteen main growth directions are generated based on the 3D space topology. Second, candidate seed points in each main direction are determined by the distance threshold. Finally, all LiDAR points are iteratively executed using candidate seed points as new region growing seed points. This paper selects two mountain terrain scenes with different cloud contents as the study area, and the precision, recall rates and F1-score of the proposed method reach 99.8%, 100% and 99.3%, respectively. This method can detect point-cloud clusters and isolated points, thus simplifying the LiDAR point clouds, providing basic support for the subsequent accurate data processing and analysis.

Generalized framework for the design of adaptive fractional-order masks for image denoising

This paper proposes a generalized fractional differential (GFD) mask which incorporates various fractional-order kernels such as power-law kernel, exponentional kernel, and Mittag-Leffler kernel, corresponding to Riemann-Liouville (RL)/Caputo, Caputo-Fabrizio (CF), and Atangana-Baleanu (AB) fractional-order differentials, respectively. Furthermore, a generalized fractional integral (GFI) mask incorporating RL and AB fractional integrals is put forward. The proposed generalized fractional masks provide a more flexible and appropriate tool for image enhancement and image denoising applications, and hence, avoids the construction of different masks for each kernel separately. Additionally, a generalized fractional integral and fractional differential based adaptive algorithm for image denoising (GFIFD-AA) is proposed. The proposedGFIFD-AA incorporates a novel noise detection method (NDM) for the detection of salt and pepper (SP) noise in images, which exhibits a profound advantage in avoiding misclassification of original pixels as noisy pixels when original image itself has some pixels with intensity values 0 or 255. The detected noisy pixels are processed by utilizing the proposed GFI-based adaptive mask (GFI-AM). The noise-free pixels are further updated by utilizing the proposed GFD-based adaptive mask (GFD-AM) so as to enhance the details of the image. Several standard and medical images of different characteristics are examined to evaluate the performance of the proposed approach on images affected by SP noise at various noise densities (i.e., 10%-95%). Furthermore, the performance of the proposed GFI mask is also tested for images affected by Gaussian noise and is compared against conventional fractional-order masks. The simulation results based upon several quantitative parameters validate the effectiveness of the proposed method against conventional methods.

Speckle Noise Detection and Removal for Laser Speech Measurement Systems

Laser speech measurement is a new sound capture technology based on Laser Doppler Vibrometry (LDV). It avoids the need for contact, is easily concealed and is ideal for remote speech acquisition, which has led to its wide-scale adoption for military and security applications. However, lasers are easily affected by complex detection environments. Thus, speckle noise often appears in the measured speech, seriously affecting its quality and intelligibility. This paper examines all of the characteristics of impulsive noise in laser measured speech and proposes a novel automatic impulsive noise detection and removal method. This method first foregrounds noise using decorrelation based on a linear prediction (LP) model that improves the noise-to-signal ratio (NSR) of the measured signal. This makes it possible to detect the position of noise through a combination of the average short-time energy and kurtosis. The method not only precisely locates small clicks (with a duration of just a few samples), but also finds the location of longer bursts and scratches (with a duration of up to a hundred samples). The located samples can then be replaced by more appropriate samples whose coding is based on the LP model. This strategy avoids unnecessary processing and obviates the need to compromise the quality of the relatively large fraction of samples that are unaffected by speckle noise. Experimental results show that the proposed automatic speckle noise detection and removal method outperforms other related methods across a wide range of degraded audio signals.

A New Hybrid Under-sampling Approach to Imbalanced Classification Problems

ABSTRACT Among many machine learning applications, classification is one of the important tasks. Most classification algorithms have been designed under the assumption that the number of samples for each class is approximately balanced. However, if the conventional classification approaches are applied to a class imbalanced dataset, it is likely to cause misclassification and, as a result, may distort classification performance results. Thus, in this study, we consider imbalanced classification problems and adopt an efficient preprocessing technique to improve the classification performances. In particular, we focus on borderline noise and outlier samples that belong to the majority class since they may influence classification performance. For this, we propose a hybrid resampling method, called BOD-based under-sampling, which is based on density-based spatial clustering of applications with noise (DBSCAN) approach as well as noise and outlier detection methods, that is, borderline noise factor (BNF) and outlierness based on neighborhood (OBN) to divide majority class samples into four distinctive categories, i.e., safe, borderline noise, rare, and outlier. Specifically, we first determine the borderline noise samples in the overlapped region using the BNF method. Secondly, we use the OBN method to detect outlier samples and apply the DBSCAN approach to cluster the samples. Based on the results obtained from the sample identification analysis, we then segregate the safe category samples which are not abnormal samples while keeping the rest of the samples as rare samples. Finally, we remove some of safe samples by using the random under-sampling (RUS) method and verify the effectiveness of the proposed algorithm through the comprehensive experimental analysis with considering several class imbalance datasets.

Analysis of the characteristics of noise from substations in buildings

The urbanization has resulted in an increasing number of transformer stations, which has resulted insignificant building noise problems. However, noise problems persist because of inadequate noise characterization and the use of imperfect noise evaluation indexes for centralized substations. Based on this problem, a transformer vibration noise coupling analysis method based on empirical mode decomposition (EMD) and spectrum analysis is proposed in this study. The proposed method accurately and effectively screens and characterizes transformer noise and provides a theoretical basis for transformer noise reduction. To verify the effectiveness of the proposed method, a transformer was reconstructed as an example. It was found that the low-frequency noise from the transformer was mainly caused by vibrations with a frequency below 500 Hz, particularly frequencies of 300 Hz and 100 Hz and 50 Hz. Through the calculation and analysis of eigenvalues, the noise reduction measures focusing on vibration reduction were proposed. In the end, a noise reduction of 10 dB was achieved, which meets the comfort requirements. This method can accurately and effectively identify the characteristics of transformer noise, which makes up for the insufficiency of transformer characteristics analysis in the past. Provide guidance for perfecting transformer noise evaluation index. Practical implication: The noise problem caused by substations is getting more and more serious. Conventional noise detection and noise reduction methods can no longer meet people’s requirements for sound comfort. The coupling analysis method of vibration and noise based on EMD and spectrum analysis proposed in this study can effectively extract the characteristics of transformer noise. It provides theoretical support for the noise reduction transformation of transformers, and solves the problem that the current engineering noise reduction transformation has no theoretical basis. Noise characteristic analysis can make up for the shortcomings of existing acoustic comfort indicators that only use sound pressure level as the evaluation indicator.

Automated Noise Detection in a Database Based on a Combined Method

Data quality has diverse dimensions, from which accuracy is the most important one. Data cleaning is one of the preprocessing steps in data mining which consists of detecting errors and repairing them. Noise is a common type of error, that occur in database. This paper proposes an automated method based on the k-means clustering for noise detection. At first, each attribute (Aj) is temporarily removed from data and the k-means clustering is applied to other attributes. Thereafter, the k-nearest neighbors is used in each cluster. After that a value is predicted for Aj in each record by the nearest neighbors. The proposed method detects noisy attributes using predicted values. Our method is able to identify several noises in a record. In addition, this method can detect noise in fields with different data types, too. Experiments show that this method can averagely detect 92% of the noises existing in the data. The proposed method is compared with a noise detection method using association rules. The results indicate that the proposed method have improved noise detection averagely by 13%.