Denoising Of Data Research Articles

Statistical methods for predicting e-cigarette use events based on beat-to-beat interval (BBI) data collected from wearable devices.

The prevalence of e-cigarette use among young adults in the USA is high (14%). Although the majority of users plan to quit vaping, the motivation to make a quit attempt is low and available support during a quit attempt is limited. Using wearable sensors to collect physiological data (eg, heart rate) holds promise for capturing the right timing to deliver intervention messages. This study aims to fill the current knowledge gap by proposing statistical methods to (1) de-noise beat-to-beat interval (BBI) data from smartwatches worn by 12 young adult regular e-cigarette users for 7days; and (2) summarize the de-noised data by event and control segments. We also conducted a comprehensive review of conventional methods for summarizing heart rate variability (HRV) and compared their performance with the proposed method. The results show that the proposed singular spectrum analysis (SSA) can effectively de-noise the highly variable BBI data, as well as quantify the proportion of total variation extracted. Compared to existing HRV methods, the proposed second order polynomial model yields the highest area under the curve (AUC) value of 0.76 and offers better interpretability. The findings also indicate that the average heart rate before vaping is higher and there is an increasing trend in the heart rate before the vaping event. Importantly, the development of increasing heart rate observed in this study implies that there may be time to intervene as this physiological signal emerges. This finding, if replicated in a larger scale study, may inform optimal timings for delivering messages in future intervention.

A robust deep learning system for screening of obstructive sleep apnea using T-F spectrum of ECG signals

Obstructive sleep apnea (OSA) is a non-communicable sleep-related medical condition marked by repeated disruptions in breathing during sleep. It may induce various cardiovascular and neurocognitive complications. Electrocardiography (ECG) is a useful method for detecting numerous health-related disorders. ECG signals provide a less complex and non-invasive solution for the screening of OSA. Automated and accurate detection of OSA may enhance diagnostic performance and reduce the clinician’s workload. Traditional machine learning methods typically involve several labor-intensive manual procedures, including signal decomposition, feature evaluation, selection, and categorization. This article presents the time-frequency (T-F) spectrum classification of de-noised ECG data for the automatic screening of OSA patients using deep convolutional neural networks (DCNNs). At first, a filter-fusion algorithm is used to eliminate the artifacts from the raw ECG data. Stock-well transform (S-T) is employed to change filtered time-domain ECG into T-F spectrums. To discriminate between apnea and normal ECG signals, the obtained T-F spectrums are categorized using benchmark Alex-Net and Squeeze-Net, along with a less complex DCNN. The superiority of the presented system is measured by computing the sensitivity, specificity, accuracy, negative predicted value, precision, F1-score, and Fowlkes-Mallows index. The results of comparing all three utilized DCNNs reveal that the proposed DCNN requires fewer learning parameters and provides higher accuracy. An average accuracy of 95.31% is yielded using the proposed system. The presented deep learning system is lightweight and faster than Alex-Net and Squeeze-Net as it utilizes fewer learnable parameters, making it simple and reliable.

An Improved Synchrosqueezing S-Transform and Its Application in a GPR Detection Task.

The S-transform is a fundamental time-frequency (T-F) domain analysis method in ground penetrating radar (GPR) data processing and can be used for identifying targets, denoising, extracting thin layers, and high-resolution imaging. However, the S-transform spectrum experiences energy leakage near the instantaneous frequency. This phenomenon causes frequency components to erroneously spread over a wider range, impacting the accuracy and precision of GPR data processing. Synchrosqueezing is an effective method to prevent spectrum leakage. In this work, we introduce the synchrosqueezing generalized phase-shifting S-transform (SS-GPST). Initially, it resolves the compatibility issue between the S-transform and the synchrosqueezing strategy through phase-shifting. Subsequently, the SS-GPST accomplishes spectral energy focusing and resolution enhancement via a generalized parameter and synchrosqueezing. A synthetic signal test shows that the SS-GPST excels over other methods at focusing degree, spectral resolution, and signal reconstruction accuracy and speed. In actual GPR tunnel detection data processing, we assess the adaptability of the SS-GPST from three aspects: spectral energy distribution, thin layer identification, and data denoising. The results indicate: (1) compared to other methods, the SS-GPST accurately expresses spectral components with a strong focusing degree and fewer interference components; (2) high-frequency slices of the SS-GPST accurately detect the top and bottom interfaces of a 3.0-3.5 cm reinforcement protection layer; and (3) due to fewer interference components in the SS-GPST spectrum, reconstructing GPR profiles through the SS-GPST inverse transform is an efficient denoising technique. The SS-GPST demonstrates adaptability to different data processing purposes, offers high-resolution T-F spectra, and shows potential to supersede the S-transform.

Application of 2D Variational Mode Decomposition Method in Seismic Signal Denoising

Seismic data are typical nonlinear and nonstationary data. In the acquisition and processing of seismic data, many factors interfere with it. Seismic data contain both effective waves and random noises, seriously affecting the quality of seismic data and not conducive to the goal of fine interpretation of subsequent seismic data. Therefore, studying new seismic data denoising methods is beneficial for improving the quality of seismic data and plays a very important role in subsequent seismic data interpretation. In this paper, the principle of variational mode decomposition (VMD) and 2D-VMD is introduced in detail, and the seismic profile with a simple signal and fault model is denoised. Compared to traditional empirical mode decomposition (EMD), the 2D-VMD method has the best seismic data denoising effect. The test results of the synthesised signal show that the 2D-VMD method has a signal-to-noise ratio of 47.14 dB after denoising, which is higher than the signal-to-noise ratio after EMD and VMD denoising, indicating that it has a better denoising effect. The VMD and 2D-VMD methods are applied to the denoising of actual seismic data. The application results show that the 2D-VMD method can effectively improve the quality of the seismic data, enhance the continuity and reliability of the seismic data, and is conducive to the fine interpretation of subsequent seismic data.

Seismic data denoising with two-step prediction strategy based on Neural Network

Seismic data denoising (SDD) plays an important role in obtaining high-quality data for subsequent seismic imaging and inversion. However, traditional SDD methods still present several disadvantages such as parameter selection difficulties, high computational costs, and a strong dependence on the experience of processing personnel. Deep learning (DL)-based SDD methods can overcome these issues to a certain extent, having the advantages of high computational efficiency and negligible dependency on the experience of processing personnel. However, existing DL-based SDD methods typically adopt a specific single-network architecture, which can be further improved in terms of training efficiency and denoising quality. In this paper, we propose a two-step prediction denoising method based on the combination of two identical or different network architectures for improving the accuracy and efficiency of DL-based SDD methods. Compared with existing SDD methods, the two-step prediction denoising method is more capable of extracting feature information at multiple scales because multiple network architectures are combined in a single denoising task, thus improving the training efficiency and denoising accuracy. In this study, we implemented the two-step prediction denoising method using an improved denoising convolutional neural network (DnCNN) and a nested U-shaped fully convolutional network (U-Net++). Several synthetic numerical examples based on three different types of noise were denoised, with the results indicating that the proposed method effectively improves the accuracy and efficiency of the DL-based SDD method. Finally, a test using field data was performed to demonstrate the real-world applicability of the novel two-step prediction denoising method.

DAS-VSP coupled noise suppression based on U-Net network

Abstract The emerging distributed fiber-optic acoustic sensing (DAS) technology has broad prospects for application in vertical seismic profiles (VSP). However, the acquired DAS-VSP data often suffers from coupled noise that seriously affects data quality. Traditional methods for suppressing coupled noise are usually time-consuming and not suitable for the large-scale denoising of DAS-VSP data. To address this, a coupled noise suppression method based on the U-Net network is proposed, and a self-attention (SA) block is introduced to enhance the denoising ability of the network. Transfer learning is employed to achieve coupled noise suppression from synthetic data to field data. Denoising results demonstrate that the network can effectively suppress coupled noise in DAS-VSP data while preserving signal energy to a certain extent, exhibiting strong generalization capability. Upon completion of network training, denoising results can be obtained within seconds, making it more convenient and efficient compared to traditional methods.

Whole‐brain deuterium metabolic imaging via concentric ring trajectory readout enables assessment of regional variations in neuronal glucose metabolism

Whole‐brain deuterium metabolic imaging via concentric ring trajectory readout enables assessment of regional variations in neuronal glucose metabolism

Construction health indicator using physically-informed 1D-WGAN-GP joint attention LSTM-DenseNet method

In data-driven prognosis methods, the accuracy of predicting the remaining useful life (RUL) of mechanical systems is predominantly contingent upon the efficacy of system health indicators (HI), typically amalgamated from statistical features derived from collected signals. Nevertheless, the majority of extant HI are beset by two principal shortcomings: (1) during traditional data denoising processes, degradation information from raw data is prone to loss owing to the lack of incorporation of the true physical properties of the data; and (2) the performance evaluation of constructed HI is imbalanced due to the influence of network structures on single models, often resulting in strong performance in only one or two indicators. To overcome such shortcomings, a mechanical health indicator construction method based on physical properties was proposed, termed 1D-WGAN-GP Joint attention LSTM-DenseNet. Firstly, artificial sample data is generated by analyzing the physical properties of the original dataset, which is then used to train the 1D-WGAN-GP model to achieve data denoising. Subsequently, the fusion of the attention LSTM (A-LSTM) network and DenseNet network is utilized to extract crucial feature vectors of HI under varying health conditions from the denoised data. Finally, the extracted feature vectors are used to construct system HI using the Euclidean distance method, and these indicators are used for predicting the system’s RUL. The results indicate that the proposed method outperformed traditional methods in terms of denoising effectiveness. Further, through ablation experiment analysis, the HI constructed by the proposed method demonstrated obvious complementarity in terms of monotonicity, correlation, robustness, and comprehensive evaluation. In RUL prediction applications, the proposed method also exhibited good performance, thereby validating its effectiveness.

Research on sensor data optimization technology for thermal hydraulic experiment of nuclear reactor

The sensor signals collected by the nuclear reactor thermal hydraulic experimental system are mixed with complex noise information. The uncertainty of sensor data directly affects the analysis and evaluation effect of machine learning algorithms on the operating status of the experimental system. Traditional denoising methods have poor adaptability to different types of noisy data, are highly dependent on researchers and have high design cost. This paper proposes a data optimization technology based on a hybrid adaptive real-time denoising (HART) model, which can realize data distribution self-adaptation and algorithm hyperparameter self-optimization according to the characteristics of noisy source data. Through the joint denoising algorithm with Variational Mode Decomposition (VMD) algorithm as the core, the partition denoising of source data is realized. In addition, the practical application of the source data of thermal hydraulic experiments in the nuclear field and the verification of simulation and noise addition experiments have been completed. The results show that the denoising optimization model proposed in this paper has the characteristics of good adaptability, self-optimization and real-time processing for different sensor signals. It can effectively remove noise information, restrain data uncertainty, and provide high-quality data source for experimental data analysis based on machine learning. At the same time, the optimization technology can be further applied to the optimization of sensor data of nuclear power plants (NPP).

A de-noising method of magnetotelluric signals based on the generalized S-transform

The generalized S-transform has excellent time-frequency resolution capability. It can effectively demonstrate the non-stationary characteristics of signals and the dynamic changes in energy. This provides the conditions for denoising in the time-frequency domain. The magnetotelluric (MT) survey method is often affected by various artificial noises, and these noises are very prominent in certain frequency bands. To address this problem, a method based on the generalized S transform is proposed to de-noise MT data. The generalized S transform is used to improve the time-frequency localization of the different signal components, meanwhile the electromagnetic noise can be conveniently suppressed in the time-frequency domain. Simulated signals and measured MT signals contaminated by common sinusoidal and triangular wave interference are processed. The results were compared with those of the Wavelet threshold method. Results show that the presented method is more conducive to the identification and suppression of electromagnetic noise and shows better de-noising results than the Wavelet threshold method. After de-noising, the useful information of the MT signal affected by strong interference is highlighted, which provides favorable conditions for power spectrum and impedance estimation in the future. Further, the response parameter curve calculated from the de-noised data becomes stable. The proposed method provides a new alternative for de-noising of MT signals, especially those with low signal-to-noise ratios.

Unsupervised Denoising and Super-Resolution of Vascular Flow Data by Physics-Informed Machine Learning.

We present an unsupervised deep learning method to perform flow denoising and super-resolution without high-resolution labels. We demonstrate the ability of a single model to reconstruct three-dimensional stenosis and aneurysm flows, with varying geometries, orientations, and boundary conditions. Ground truth data was generated using computational fluid dynamics, and then corrupted with multiplicative Gaussian noise. Auto-encoders were used to compress the representations of the flow domain geometry and the (possibly noisy and low-resolution) flow field. These representations were used to condition a physics-informed neural network. A physics-based loss was implemented to train the model to recover lost information from the noisy input by transforming the flow to a solution of the Navier-Stokes equations. Our experiments achieved mean squared errors in the true flow reconstruction of O(1.0 × 10-4), and root mean squared residuals of O(1.0 × 10-2) for the momentum and continuity equations. Our method yielded correlation coefficients of 0.971 for the hidden pressure field and 0.82 for the derived wall shear stress field. By performing point-wise predictions of the flow, the model was able to robustly denoise and super-resolve the field to 20× the input resolution.

Multiple distresses detection for Asphalt Pavement using improved you Only Look Once Algorithm based on convolutional neural network

ABSTRACT Leveraging the YOLOv7 object detection framework, this study introduces YOLOv7-CSP, a refined algorithm tailored for identifying asphalt pavement distress with enhanced precision. Utilizing advanced image processing for dataset preprocessing, including data augmentation and denoising, YOLOv7-CSP integrates the CSPNeXt structure and CA attention mechanism for improved detection accuracy and efficiency. The algorithm optimizes anchor box selection through Kmeans clustering and employs a secondary labeling method to enhance learning efficiency and dataset quality. Comparative tests reveal YOLOv7-CSP's superior performance, with significant improvements in mAP, F1 score, GFLOPS, and FPS metrics, demonstrating its effectiveness in detecting various pavement distresses. This innovative approach marks a significant advancement in automatic pavement distress recognition, offering a robust solution for highway maintenance decision-making.

Denoising of Geochemical Data using Deep Learning–Implications for Regional Surveys

Regional geochemical surveys generate large amounts of data that can be used for a number of purposes such as to guide mineral exploration. Modern surveys are typically designed to permit quantification of data uncertainty through data quality metrics by using quality assurance and quality control (QA/QC) methods. However, these metrics, such as data accuracy and precision, are obtained through the data generation phase. Consequently, it is unclear how residual uncertainty in geochemical data can be minimized (denoised). This is a limitation to propagating uncertainty through downstream activities, particularly through complex models, which can result from the usage of artificial intelligence-based methods. This study aims to develop a deep learning-based method to examine and quantify uncertainty contained in geochemical survey data. Specifically, we demonstrate that: (1) autoencoders can reduce or modulate geochemical data uncertainty; (2) a reduction in uncertainty is observable in the spatial domain as a decrease of the nugget; and (3) a clear data reconstruction regime of the autoencoder can be identified that is strongly associated with data denoising, as opposed to the removal of useful events in data, such as meaningful geochemical anomalies. Our method to post-hoc denoising of geochemical data using deep learning is simple, clear and consistent, with the amount of denoising guided by highly interpretable metrics and existing frameworks of scientific data quality. Consequently, variably denoised data, as well as the original data, could be fed into a single downstream workflow (e.g., mapping, general data analysis or mineral prospectivity mapping), and the differences in the outcome can be subsequently quantified to propagate data uncertainty.

Sparse Regularization Based on Orthogonal Tensor Dictionary Learning for Inverse Problems

In seismic data processing, data recovery including reconstruction of the missing trace and removal of noise from the recorded data are the key steps in improving the signal-to-noise ratio (SNR). The reconstruction of seismic data and removal of noise becomes a sparse optimization problem that can be solved by using sparse regularization. Sparse regularization is a key tool in the solution of inverse problems. They are used to introduce prior knowledge and make the approximation of ill-posed inverses feasible. It deals with ill-posedness by replacing an ill-posed inverse problem with a well-posed problem that has a solution close to the true solution. In the last 2 decades, interest has shifted from linear toward nonlinear regularization methods even for linear inverse problems. In inverse problems, regularizations serve as stabilizing the solution of ill-posed inverse problems and give a solution that adequately fits measurements without producing unjustifiably complex artifacts. In this paper, we present a novel sparse regularization based on a tensor-based dictionary method for inverse problems (seismic data interpolation and denoising). This regularization avoids the vectorization step for sparse representation of seismic data during the reconstruction process. The key step in sparsifying signals is the choice of sparsity-promoting dictionary learning. The learning-based approach can adaptively sparsify datasets but has high computational complexity and involves no prior-constraint pattern information for the dataset. Many existing dictionary learning methods would be computationally infeasible for the high dimensional seismic data processing. These methods also destroy the essential information as well as it reduces the discriminability and expressibility of the signal, since they deal with vectorization. In this paper, the orthogonal tensor dictionary learning that learns a dictionary from the input data by employing orthogonality and separability is proposed as sparse regularization for the inverse problems. The performance of the proposed method was validated in seismic data interpolation and denoising individually as well as simultaneously. Numerical examples of synthetic and real seismic datasets demonstrate the validity of the proposed method. The SNR of the recovered data confirms that the proposed method is the most effective method than K-singular value decomposition and orthogonal dictionary learning methods.

Weak signal extraction enabled by deep neural network denoising of diffraction data.

The removal or cancellation of noise has wide-spread applications in imaging and acoustics. In applications in everyday life, such as image restoration, denoising may even include generative aspects, which are unfaithful to the ground truth. For scientific use, however, denoising must reproduce the ground truth accurately. Denoising scientific data is further challenged by unknown noise profiles. In fact, such data will often include noise from multiple distinct sources, which substantially reduces the applicability of simulation-based approaches. Here we show how scientific data can be denoised by using a deep convolutional neural network such that weak signals appear with quantitative accuracy. In particular, we study X-ray diffraction and resonant X-ray scattering data recorded on crystalline materials. We demonstrate that weak signals stemming from charge ordering, insignificant in the noisy data, become visible and accurate in the denoised data. This success is enabled by supervised training of a deep neural network with pairs of measured low- and high-noise data. We additionally show that using artificial noise does not yield such quantitatively accurate results. Our approach thus illustrates a practical strategy for noise filtering that can be applied to challenging acquisition problems.

Text Document Clustering Using Modified Particle Swarm Optimization with k-means Model

In the present digital era, vast amounts of data are generated by millions of Internet users in the form of unstructured text documents. The clustering and organizing of text documents play a crucial role in the applications of data analysis and market research. In this research manuscript, a new modified version of metaheuristic-based optimization technique is proposed with k-means for clustering the text documents. In the initial phase, the input data are acquired from the three-benchmark databases such as Reuters-21578, 20-Newsgroup and British Broadcasting Corporation (BBC)-sport. Further, the data denoising is accomplished by using the common techniques: stemming, lemmatization, tokenization, and stop word removal. In addition to this, the denoised data are transformed into feature vectors by utilizing Term Frequency (TF)-Inverse Document Frequency (IDF) technique. The computed feature vectors are given to the Modified Particle Swarm Optimization (MPSO) with k-means to group the closely related text documents by minimizing the similarity in different clusters. The experimental examination showed that the proposed MPSO with k-means model achieved accuracy of 0.85, 0.85 and 0.86 on the Reuters-21578, 20-Newsgroup and BBC-sport databases, which are superior to the comparative models.

Evaluation of Swin Transformer and knowledge transfer for denoising of super-resolution structured illumination microscopy data.

Convolutional neural network (CNN)-based methods have shown excellent performance in denoising and reconstruction of super-resolved structured illumination microscopy (SR-SIM) data. Therefore, CNN-based architectures have been the focus of existing studies. However, Swin Transformer, an alternative and recently proposed deep learning-based image restoration architecture, has not been fully investigated for denoising SR-SIM images. Furthermore, it has not been fully explored how well transfer learning strategies work for denoising SR-SIM images with different noise characteristics and recorded cell structures for these different types of deep learning-based methods. Currently, the scarcity of publicly available SR-SIM datasets limits the exploration of the performance and generalization capabilities of deep learning methods. In this work, we present SwinT-fairSIM, a novel method based on the Swin Transformer for restoring SR-SIM images with a low signal-to-noise ratio. The experimental results show that SwinT-fairSIM outperforms previous CNN-based denoising methods. Furthermore, as a second contribution, two types of transfer learning-namely, direct transfer and fine-tuning-were benchmarked in combination with SwinT-fairSIM and CNN-based methods for denoising SR-SIM data. Direct transfer did not prove to be a viable strategy, but fine-tuning produced results comparable to conventional training from scratch while saving computational time and potentially reducing the amount of training data required. As a third contribution, we publish four datasets of raw SIM images and already reconstructed SR-SIM images. These datasets cover two different types of cell structures, tubulin filaments and vesicle structures. Different noise levels are available for the tubulin filaments. The SwinT-fairSIM method is well suited for denoising SR-SIM images. By fine-tuning, already trained models can be easily adapted to different noise characteristics and cell structures. Furthermore, the provided datasets are structured in a way that the research community can readily use them for research on denoising, super-resolution, and transfer learning strategies.

Football team training algorithm: A novel sport-inspired meta-heuristic optimization algorithm for global optimization

A more efficient optimization algorithm has always been the pursuit of researchers, but the performance of the current optimization algorithm in some complex test functions is not always satisfactory. In order to solve this problem, a new meta-heuristic optimization algorithm—Football Team Training Algorithm (FTTA) is proposed according to the training method of the football team, which simulates the three stages of the training session: Collective Training, Group Training and Individual Extra Training. By the test on two groups of test functions, CEC2005 and CEC2020, the proposed optimization algorithm (FTTA) achieves the best results, which far exceeds the traditional Grey Wolf Optimization (GWO), Whale Optimization Algorithm (WOA) algorithms and so on. In the engineering application, a new hybrid wind speed prediction system is proposed based on FTTA. The FTTA is used to optimize variational mode decomposition (VMD) to improve the effect of data denoising. At the same time, based on unconstrained weighting algorithm, FTTA and combination prediction model build a new hybrid prediction strategy. Through the experiments on four groups of wind speed data in Dalian, the accuracy, stability, advancement, and CPU running speed of the system are verified. It is obvious that the practical application ability of the system is much better than previous methods, which can effectively improve the utilization efficiency of renewable energy.

A Deconvolution–Interpolation Method for Correction and De-Noising of Doppler-Shifted Marine Vibrator Data in the Frequency–Wavenumber Domain

A Deconvolution–Interpolation Method for Correction and De-Noising of Doppler-Shifted Marine Vibrator Data in the Frequency–Wavenumber Domain

Data augmentation and denoising of computed tomography scan images in training deep learning models for rapid COVID-19 detection

Data augmentation and denoising of computed tomography scan images in training deep learning models for rapid COVID-19 detection