Denoising Autoencoder Research Articles

An improved pollution forecasting model with meteorological impact using multiple imputation and fine-tuning approach

Air pollution forecasting is a significant step for air quality pollution management to mitigate pollution’s negative impact on the environment and people’s health. The data-driven forecasting model can help a better understanding of environmental air quality. The existing data-driven forecasting models usually ignore missing values, the correlations between the pollutant and meteorological factors and fail to perform temporal modeling effectively, affecting prediction accuracy. In response to these issues, we present a deep learning-based Convolutional LSTM–SDAE (CLS) model to forecast the particulate matter level, revealing the correlation between particulate matter and meteorological factors. In the proposed architecture, the k nearest neighbor (KNN) imputation technique is employed to recover the air quality dataset’s missing values. The Convolutional Long Short Term Memory (CNN–LSTM) unit identifies the vast dataset’s hidden features and performs pollutants’ temporal modeling. In addition, Bidirectional Gatted Recurrent Unit (BIGRU) is implemented as both encoder and decoder in Sparse Denoising Autoencoder, which reconstructs the CNN–LSTM model’s output in the dynamic fine-tuning layer to get robust prediction results. The experimental results in Talcher, India, and Beijing, China indicate that the model can improve forecasting accuracy and outperforms the other state of art and baseline models.

Research on Movie Recommendation Algorithm Based on Stack De-noising Auto-encoder

Focused on the problems that the randomness of the noise-adding operation in the de-noising auto-encoder (DAE), and the data matrix does not consider the impact of trusted users on the deep preferences of target users, this paper proposes a recommendation algorithm based on stack de-noising auto-encoder (SDAE) which integrates the preferences of trusted users. Firstly, the score vector is used as the input of the auto-encoder, and the mask vector is designed to train the potential preference of the target user. Secondly, the deep preference of the target user and the trusted user is obtained by the weighted fusion of the features of the two hidden layers of the auto-encoder. Thirdly, in order to reduce the impact of noise on the prediction accuracy, the cascaded auto-encoder model is constructed and trained according to the greedy training method layer by layer training. Finally, SDAE model is compared with other models on different data sets. The experimental results show that SDAE model has better recommendation performance.

Robust Computationally-Efficient Wireless Emitter Classification Using Autoencoders and Convolutional Neural Networks.

This paper proposes a novel Deep Learning (DL)-based approach for classifying the radio-access technology (RAT) of wireless emitters. The approach improves computational efficiency and accuracy under harsh channel conditions with respect to existing approaches. Intelligent spectrum monitoring is a crucial enabler for emerging wireless access environments that supports sharing of (and dynamic access to) spectral resources between multiple RATs and user classes. Emitter classification enables monitoring the varying patterns of spectral occupancy across RATs, which is instrumental in optimizing spectral utilization and interference management and supporting efficient enforcement of access regulations. Existing emitter classification approaches successfully leverage convolutional neural networks (CNNs) to recognize RAT visual features in spectrograms and other time-frequency representations; however, the corresponding classification accuracy degrades severely under harsh propagation conditions, and the computational cost of CNNs may limit their adoption in resource-constrained network edge scenarios. In this work, we propose a novel emitter classification solution consisting of a Denoising Autoencoder (DAE), which feeds a CNN classifier with lower dimensionality, denoised representations of channel-corrupted spectrograms. We demonstrate—using a standard-compliant simulation of various RATs including LTE and four latest Wi-Fi standards—that in harsh channel conditions including non-line-of-sight, large scale fading, and mobility-induced Doppler shifts, our proposed solution outperforms a wide range of standalone CNNs and other machine learning models while requiring significantly less computational resources. The maximum achieved accuracy of the emitter classifier is 100%, and the average accuracy is 91% across all the propagation conditions.

Real-Time Outlier Detection Applied to a Doppler Velocity Log Sensor Based on Hybrid Autoencoder and Recurrent Neural Network

This article presents a real-time outlier detection deep-learning (OD-DL)-based method using a hybridized artificial neural network (ANN) approach. We propose an unsupervised ANN scheme that runs in parallel, a denoising autoencoder (DAE) and a recurrent neural network (RNN). The DAE aims to reconstruct relevant/normal input data, whereas it seeks to ignore outliers; the RNN, with a recursive structure, is used to predict time-series data. As measurements arrive, two tasks are performed: 1) the outlier decision, which is based on a reconstruction error and an energy score criteria from the output difference between the DAE and the RNN; and 2) the training procedure for both DAE and RNN. The proposed OD-DL scheme is specifically targeted to address the outlier problem of the data generated by a Doppler velocity log (DVL) sensor installed on board of an autonomous underwater vehicle (AUV) to enhance the AUV navigation system performance. In particular, the DVL data enter into the OD-DL scheme whose output is fed into an AUV navigation system that runs an error-state Kalman filter that integrates the corrected DVL data with the measurements of an inertial measurement unit and a depth meter. The experimental results show that the AUV navigation system with the OD-DL method outperforms in terms of a more accurate estimated position when compared with the case that there is no outlier detection and with the case of a navigation system using a conventional outlier detection method, or other simpler deep-learning methods.

Research on hyper-spectral remote sensing image classification by applying stacked de-noising auto-encoders neural network

Hyper-spectral image can provide precise information on land surface targets identification and classification thanks to its advanced feature on spectral resolution. However, due to its complicated hyper-dimension data structure, greater challenge is put on the conventional image classification methods for hyper-spectral images. To fill this technical knowledge gap, we introduce a deep learning-based feature extraction method for hyper-spectral data classification. Firstly, we used a Stacked De-noising Auto-encoders(SDAE) to extract the in-depth features of hyper-spectral image data: a large amount of unlabeled data was pre-trained to extract the depth characteristics of pixels. We added random noise into the input layer of the network to form a de-noising auto-encoder machine and added treated inputs to reconstruct original data. An L-BFGS (Limited-memory quasi-Newton code) was used to optimize the loss function. In the top layer, deep neural network was fine-tuned by a Softmax regression classifier. All these improvements directed towards the model to obtain the image element abstraction and robust expression in the classification task of the hyper-spectral images. We tested the model performance on a Hyspex imaging spectrometer image and found that the newly introduced model framework outperforms other traditional methods including principal component analysis (PCA), and support vector machine (SVM) classifier, combined PCA-SVM classifier, and Minimum Noise Fraction Rotation (MNF)-SVM classifier.

A fault diagnosis method based on Auxiliary Classifier Generative Adversarial Network for rolling bearing.

Rolling bearing fault diagnosis is one of the challenging tasks and hot research topics in the condition monitoring and fault diagnosis of rotating machinery. However, in practical engineering applications, the working conditions of rotating machinery are various, and it is difficult to extract the effective features of early fault due to the vibration signal accompanied by high background noise pollution, and there are only a small number of fault samples for fault diagnosis, which leads to the significant decline of diagnostic performance. In order to solve above problems, by combining Auxiliary Classifier Generative Adversarial Network (ACGAN) and Stacked Denoising Auto Encoder (SDAE), a novel method is proposed for fault diagnosis. Among them, during the process of training the ACGAN-SDAE, the generator and discriminator are alternately optimized through the adversarial learning mechanism, which makes the model have significant diagnostic accuracy and generalization ability. The experimental results show that our proposed ACGAN-SDAE can maintain a high diagnosis accuracy under small fault samples, and have the best adaptation performance across different load domains and better anti-noise performance.

ClEnDAE: A classifier based on ensembles with built-in dimensionality reduction through denoising autoencoders

High dimensionality is an issue that affects most classification algorithms. This factor implies that the predictive performance of many traditional classifiers decreases considerably as the number of features increases. Therefore, there are numerous proposals that try to mitigate the effects of this issue. This study proposes ClEnDAE, a new classifier based on ensembles whose components incorporate denoising autoencoders (DAEs) to reduce the dimensionality of the input space. On the one hand, the use of ensembles improves the predictive performance by using several components that work jointly. On the other hand, the use of DAEs allows a new higher-level, smaller-sized feature space to be generated, reducing high dimensionality effects. Finally, an experimentation is conducted with the goal of evaluating the behavior of ClEnDAE. The first part of the test compares the performance of ClEnDAE to a model based on basic DAE and to the original untreated data. The second part analyzes the results of ClEnDAE and other traditional methods of dimensionality reduction in order to determine the improvement achieved with the proposed algorithm. In both parts of the experimentation, conclusions show that ClEnDAE offers better predictive performance than the other analyzed models. The main advantage of the ClEnDAE method is the combination of the potential of the ensemble-based methodology, where several components work in parallel, and DAEs, which generate new low-dimensional features that provide more relevant information. Therefore, the classification performance is better than with other classic proposals.

Incipient Fault Diagnosis of Batch Process Based on Deep Time Series Feature Extraction

Intelligence modeling interpretable for incipient fault diagnosis of batch process represents a serious challenge. For the multivariate, nonlinear, and high-dimensionality characteristics of process data, existing fault diagnosis solutions are easily concealed by noise while neglect the low amplitude and noise interference of the incipient faults. In this paper, an intelligent incipient fault diagnosis model based on deep time series feature extraction network is proposed, which integrates denoising autoencoder (DAE), layer normalization and dropout layer added LSTM (LD-LSTM), and stacked autoencoder (SAE) to obtain the efficient features. DAE is applied to restore the data and eliminate noise interference. LD-LSTM is utilized to extract time series features, in which the layer normalization and the dropout layer are added between the layers of the LSTM to solve the problems of slow model convergence and over-fitting. SAE performs the feature extraction to obtain deep time series features and learns more efficient expressions. The softmax function is applied to achieve the incipient fault diagnosis based on the deep time series features. A case study on a fed-batch penicillin fermentation process suggests that the proposed method can obtain an accuracy rate of 98.97%, which has increased an average of 20% than the traditional shallow machine learning. The accuracy of fault recognition and comparative experiments demonstrate the effectiveness and feasibility of this model, and provide a solution for the application of intelligent fault diagnosis in batch industries.

Attention neural collaboration filtering based on GRU for recommender systems

The collaborative filtering method is widely used in the traditional recommendation system. The collaborative filtering method based on matrix factorization treats the user’s preference for the item as a linear combination of the user and the item latent vectors, and cannot learn a deeper feature representation. In addition, the cold start and data sparsity remain major problems for collaborative filtering. To tackle these problems, some scholars have proposed to use deep neural network to extract text information, but did not consider the impact of long-distance dependent information and key information on their models. In this paper, we propose a neural collaborative filtering recommender method that integrates user and item auxiliary information. This method fully integrates user-item rating information, user assistance information and item text assistance information for feature extraction. First, Stacked Denoising Auto Encoder is used to extract user features, and Gated Recurrent Unit with auxiliary information is used to extract items’ latent vectors, respectively. The attention mechanism is used to learn key information when extracting text features. Second, the latent vectors learned by deep learning techniques are used in multi-layer nonlinear networks to learn more abstract and deeper feature representations to predict user preferences. According to the verification results on the MovieLens data set, the proposed model outperforms other traditional approaches and deep learning models making it state of the art.

Extracting and inserting knowledge into stacked denoising auto-encoders

Deep neural networks (DNNs) with a complex structure and multiple nonlinear processing units have achieved great successes for feature learning in image and visualization analysis. Due to interpretability of the “black box” problem in DNNs, however, there are still many obstacles to applications of DNNs in various real-world cases. This paper proposes a new DNN model, knowledge-based deep stacked denoising auto-encoders (KBSDAE), which inserts the knowledge (i.e., confidence and classification rules) into the deep network structure. This model not only can offer a good understanding of the representations learned by the deep network but also can produce an improvement in the learning performance of stacked denoising auto-encoder (SDAE). The knowledge discovery algorithm is proposed to extract confidence rules to interpret the layerwise network (i.e., denoising auto-encoder (DAE)). The symbolic language is developed to describe the deep network and shows that it is suitable for the representation of quantitative reasoning in a deep network. The confidence rule insertion to the deep network is able to produce an improvement in feature learning of DAEs. The classification rules extracted from the data offer a novel method for knowledge insertion to the classification layer of SDAE. The testing results of KBSDAE on various benchmark data indicate that the proposed method not only effectively extracts knowledge from the deep network, but also shows better feature learning performance than that of those typical DNNs (e.g., SDAE).

Multiscale denoising autoencoder for improvement of target detection

ABSTRACT Target detection is one of the most important applications of hyperspectral technology. However, due to spectral variations caused by noise or environment, the within-class variation is enlarged which degrades the performance of detectors, especially when the target size is small. Therefore, improving the detection performance of small targets and noisy targets is a key task. Considering the great feature extraction and representation ability of deep learning models, denoising autoencoder (DAE) is introduced to reconstruct spectrums and exploit the invariant information for target detection. To fully extract the features from the original spectrums, a multiscale denoising autoencoder (MSDAE) model is designed to incorporate complementary informationin in this paper. The final spectrum is fused by reconstructed spectrums from different scales representations, which provides more complex information and more robust features for subsequent spectral identification. Results on simulated hyperspectral images (HSIs) and real-world HSIs demonstrate that the proposed MSDAE model can effectively remove noise interference and lead to great improvements of the target detection. In addition, the proposed method shows significant potential in preserving small targets.

Cell Subtype Classification via Representation Learning Based on a Denoising Autoencoder for Single-Cell RNA Sequencing

Identification of single-cell subtypes is one of the fundamental processes required to understand a heterogeneous population composed of multiple cells, based on single-cell RNA sequencing data. Previously, cell subtype identification was mainly carried out by dimension reduction and clustering approaches that grouped cells with similar expressed profiles together. However, for high robustness to noises and systematic annotation of the subtype in each cell, supervised classification approaches have been widely used. Recently, deep neural network (DNN) models have been widely presented in various fields, including biology. By capturing the composite relationship between sample features and target outcomes, a DNN model enables significant performance improvements in biological data mining analyses. In this paper, we constructed a DNN model, called scDAE for single-cell subtype identification combined with representative feature extraction using a multilayer denoising autoencoder (DAE). The feature sets were learned by the DAE and were further tuned by fully connected layers using a softmax classifier. The model was compared against four state-of-the-art cell subtype identification methods and two conventional machine learning algorithms. From multiple tests, scDAE significantly outperformed competing methods especially on data sets having a large number of cell subtypes and noises. Extracted cell features from the proposed model were clearly clustered with respect to subtype. The results of the experiments indicated that our proposed model is effective in identifying single-cell subtypes and molecular signatures representative of each cell subtype. scDAE is publicly available at https://github.com/cbi-bioinfo/scDAE .

Optimization of a Comprehensive Sequence Forecasting Framework Based on DAE-LSTM Algorithm

Sales forecasting is a multivariate time series forecasting problem. The main challenges of the forecasting task are the high-dimensional influence variables with noise and the complex time series relationships. In this paper, we propose a DAE-LSTM algorithm combined with denoising autoencoder(DAE) and long short-term memory(LSTM) to deal with this problem. The DAE is a non-linear dimensionality reduction method that enhances data robustness. The LSTM is a deep learning algorithm suitable for dealing with multivariate time series problem. Combining the advantages of the above two algorithms, DAE-LSTM is a new multivariate time series prediction algorithm, which transforms unsupervised DAE into supervised data feature extraction method, and adds feature selection function to LSTM algorithm. In addition, in order to build a comprehensive forecasting framework, the hyperparameters of the model are determined by Bayesian optimization method(BOM) and the parameters of the model are solved by adaptive moment estimation(ADAM) optimization algorithm. In the training process, due to the adaptability of the BOM and ADAM, the forecasting model has few parameters that need to be set artificially, only including dimensional scale and time steps determined by data. Finally, the proposed forecasting framework is applied to forecast the sales of online retailers on JD.COM. In comparison with other machine learning algorithms, including multiple linear regression(MLR), support vector regression(SVR), artificial neural network(ANN) and LSTM, the proposed DAE-LSTM achieves the best prediction accuracy.

A Data Augmentation-Based Method for Robust Device-Free Localization in Changing Environments of Passive Radio Frequency Identification System

Device-free localization (DFL) is playing a critical role in many applications which do not require any device attached to the target. Because of environmental changes, the fingerprint database established in the original environments is unable to remain effective when used in the changing environments. Looking at dropout as a prior-knowledge-free data augmentation operation, we propose a convolutional denoising autoencoder (DAE)-based DFL architecture AugRF. Our method combines the merits of the convolutional neural network and the DAE in which convolutional operation is beneficial to feature extraction and dropout operation is equivalent to generating augmented versions of the training data. It is noted that our method can improve the localization performance in changed environments without resampling and retraining processes. Simulations and real-world experiments verify the superiority of the proposed architecture. Our localization system can be implemented with a commercial ultrahigh-frequency radio frequency identification system. When the signal-to-noise ratio of the fingerprint data drops significantly due to environmental changes, AugRF contributes to improving localization accuracy. In addition, the time cost for one positioning meets the requirement of real-time indoor localization.

Deep Spatio-Temporal Representation and Ensemble Classification for Attention Deficit/Hyperactivity Disorder.

Attention deficit/Hyperactivity disorder (ADHD) is a complex, universal and heterogeneous neurodevelopmental disease. The traditional diagnosis of ADHD relies on the long-term analysis of complex information such as clinical data (electroencephalogram, etc.), patients' behavior and psychological tests by professional doctors. In recent years, functional magnetic resonance imaging (fMRI) has been developing rapidly and is widely employed in the study of brain cognition due to its non-invasive and non-radiation characteristics. We propose an algorithm based on convolutional denoising autoencoder (CDAE) and adaptive boosting decision trees (AdaDT) to improve the results of ADHD classification. Firstly, combining the advantages of convolutional neural networks (CNNs) and the denoising autoencoder (DAE), we developed a convolutional denoising autoencoder to extract the spatial features of fMRI data and obtain spatial features sorted by time. Then, AdaDT was exploited to classify the features extracted by CDAE. Finally, we validate the algorithm on the ADHD-200 test dataset. The experimental results show that our method offers improved classification compared with state-of-the-art methods in terms of the average accuracy of each individual site and all sites, meanwhile, our algorithm can maintain a certain balance between specificity and sensitivity.

Grammatical Error Correction with Denoising Autoencoder

A denoising autoencoder sequence-to-sequence model based on transformer architecture proved to be useful for underlying tasks such as summarization, machine translation, or question answering. This paper investigates the possibilities of using this model type for grammatical error correction and introduces a novel method of remark-based model checkpoint output combining. This approach was evaluated by the BEA 2019 shared task. It was able to achieve state-of-the-art F-score results on the test set 73.90 and development set 56.58. This was done without any GEC-specific pre-training, but only by fine-tuning the autoencoder model and combining checkpoint outputs. This proves that an efficient model solving GEC might be trained in a matter of hours using a single GPU.

Assessing Wireless Sensing Potential With Large Intelligent Surfaces

Sensing capability is one of the most highlighted new feature of future 6G wireless networks. This paper addresses the sensing potential of Large Intelligent Surfaces (LIS) in an exemplary Industry 4.0 scenario. Besides the attention received by LIS in terms of communication aspects, it can offer a high-resolution rendering of the propagation environment. This is because, in an indoor setting, it can be placed in proximity to the sensed phenomena, while the high resolution is offered by densely spaced tiny antennas deployed over a large area. By treating an LIS as a radio image of the environment relying on the received signal power, we develop techniques to sense the environment, by leveraging the tools of image processing and machine learning. Once a radio image is obtained, a Denoising Autoencoder (DAE) network can be used for constructing a super-resolution image leading to sensing advantages not available in traditional sensing systems. Also, we derive a statistical test based on the Generalized Likelihood Ratio (GLRT) as a benchmark for the machine learning solution. We test these methods for a scenario where we need to detect whether an industrial robot deviates from a predefined route. The results show that the LIS-based sensing offers high precision and has a high application potential in indoor industrial environments.

Deep auto encoders to adaptive E-learning recommender system

Adaptive learning, supported by Information & Communication Technology (TIC), is an important research area for educational systems which aim to improve the outcomes of students. Thus, the investigation of what should be adapted and how much to adapt constitute a foundation to Adaptive E-learning Systems (AES). In this paper, we compared three classes of Deep Auto Encoders and the popularity model to address the problem of learning and predicting the preferences of student on AES: Collaborative Denoising Auto Encoders (CDAE), Deep Auto Encoders for Collaborative Filtering (DAE-CF), and Deep Auto Encoders for Collaborative Filtering using Content Information (DAE-CI). The results point out that the DAE-CF is more effective providing significant adaptability. Furthermore, we present the concept named as signature of preference to represent a more granular class of adaptability. Therefore, this model may be used in e-learning systems to provide adaptability and help to improve the outcomes of students.

Automatic COVID-19 disease diagnosis using 1D convolutional neural network and augmentation with human respiratory sound based on parameters: cough, breath, and voice.

The issue in respiratory sound classification has attained good attention from the clinical scientists and medical researcher's group in the last year to diagnosing COVID-19 disease. To date, various models of Artificial Intelligence (AI) entered into the real-world to detect the COVID-19 disease from human-generated sounds such as voice/speech, cough, and breath. The Convolutional Neural Network (CNN) model is implemented for solving a lot of real-world problems on machines based on Artificial Intelligence (AI). In this context, one dimension (1D) CNN is suggested and implemented to diagnose respiratory diseases of COVID-19 from human respiratory sounds such as a voice, cough, and breath. An augmentation-based mechanism is applied to improve the preprocessing performance of the COVID-19 sounds dataset and to automate COVID-19 disease diagnosis using the 1D convolutional network. Furthermore, a DDAE (Data De-noising Auto Encoder) technique is used to generate deep sound features such as the input function to the 1D CNN instead of adopting the standard input of MFCC (Mel-frequency cepstral coefficient), and it is performed better accuracy and performance than previous models.ResultsAs a result, around 4% accuracy is achieved than traditional MFCC. We have classified COVID-19 sounds, asthma sounds, and regular healthy sounds using a 1D CNN classifier and shown around 90% accuracy to detect the COVID-19 disease from respiratory sounds.ConclusionA Data De-noising Auto Encoder (DDAE) was adopted to extract the acoustic sound signals in-depth features instead of traditional MFCC. The proposed model improves efficiently to classify COVID-19 sounds for detecting COVID-19 positive symptoms.

IMPLEMENTATION OF AUTOENCODER ON MNIST HANDWRITTEN DIGITS

Autoencoders (AE) are a family of neural networks for which the input is the same as the output. They work by compressing the input into a latent-space representation and then reconstructing the output from this representation. The aim of an Autoencoder is to learn a representation (encoding) for a set of data, typically for dimensionality reduction, by training the network to ignore signal “noise”. In this paper De-noising Autoencoder is implemented by proposing a novel approach on MNIST handwritten digits. This model is validated through training and validation losses, and observing the reconstructed test images when comparing to the original images. The proposed model is found to be working very well.