Stacked Denoising Autoencoder Research Articles

Fault diagnosis based on SPBO-SDAE and transformer neural network for rotating machinery

Fault diagnosis for rotating machinery requires both high diagnosis accuracy and time efficiency. A rotating machinery fault diagnosis method based on intelligent feature self-extraction and transformer neural network is proposed. Firstly, the proposed method employs the student psychology based optimization (SPBO) algorithm to adaptively select hyper parameters, including the number of hidden layer nodes, sparsity coefficient and input data zeroing ratio, of the denoising auto encoder (DAE) network to determine the optimal structure of the stacked denoising auto encoders (SDAE) network. Secondly, the optimized SPBO-SDAE network is used to extract features from high-dimensional original data layer by layer. On this basis, the weight parameters of self-extracted features of SPBO-SDAE network are optimized through the self-attention mechanism of transformer deep neural network. The target features are retained, and the redundant features are filtered. Finally, in order to further validate the performance of the proposed model in the complex conditions, by adding Gaussian noise to the original data, the diagnosis performance of the proposed method is verified through four open data sets. The simulation results indicate that compared with the existing common shallow learning and deep learning methods, the proposed method has great advantages in generalization performance, fault diagnosis accuracy and time efficiency.

Comparison of Deep Learning Architectures for Short-Term Electrical Load Forecasting Based on Multi-Modal Data

Short-term load forecasting is a topic of considerable interest as it is of major importance for specifying and managing power resources and needs. In the literature, Deep Neural Networks have been successfully recently applied in load forecasting using single modalities as an improvement to traditional Artificial Neural Networks (ANN). In this paper, the main objective is to tackle the load forecasting problem with the intention of enhancing the prediction performance by combining and testing different multi-modal deep learning approaches and architectures in order to process and relate information from multiple modalities. The benefits of using multi-modality instead of one modality, applied to the time series modeling problem of short-term load forecasting is therefore investigated. The models are trained and evaluated using the hourly temperature and electrical consumption in addition to auto-regressive variables as the first modality. The day type characteristic, such as: weekends, week days, bank holidays, religious holidays etc., may be considered as a second modality. The approach liability is tested by comparing the empirical results of different Deep architectures namely: Stacked Denoising Auto- Encoders (SDAEs) and Convolutional Neural Network (CNN), with multiple and single modalities where processing the same task of one day ahead load forecasting.

Data-driven power system reliability evaluation based on stacked denoising auto-encoders

The increasing penetration of renewable energy resources in power systems inadvertently leads to a surge in the number of random states. The calculation of optimal load shedding for all the proliferated states is rather overwhelming. It has been considered a bottleneck in the reliability evaluation process. To address this issue, a deep-learning-based approach is proposed to evaluate system reliability more efficiently considering the fluctuation of generation and loads. In the proposed power flow model, a stacked denoising auto-encoder (SDAE) serves as a multilayer neural network for the deep-learning process. Its stacked structure and encoding-decoding recursion enable it to extract high-order features even from non-linear equations. The features of contingency states are crucial to directly acquire the minimum load curtailment without the time-consuming optimal power flow (OPF). The after-trained model is applied to states simulated with the Monte-Carlo method in the RTS-79 system. The numerical result reveals the great advantage of the SDAE-based method in computation time while ensuring high accuracy. As an alternative way to classic OPF algorithms, it can be integrated with the impact-increment method and other state selection approaches.

Floating offshore wind turbine fault diagnosis using stacked denoising autoencoder with temporal information

Currently, the offshore wind turbine has become a hot research area in the wind energy industry. Among different offshore wind turbines, floating offshore wind turbine (FOWT) can harvest abundant wind energy in deepwater areas. However, the harsh working environment will dramatically increase the maintenance cost and downtime of FOWTs. Wind turbine fault diagnosis is being regarded as an indispensable system for maintenance issues. Owing to the complexity of FOWT, it imposes an enormous challenge for effective fault diagnosis. This paper develops a novel FOWT fault diagnosis method based on a stacked denoising autoencoder (SDAE). First, a sliding window technique is adopted for time-series data to preserve temporal information. Then, SDAE is employed to extract the features from high-dimensional data. Based on the extracted features from SDAE, a classifier using multilayer perceptrons (MLP) is developed to determine the health status of the FOWT. To verify the performance of the proposed method, a FOWT simulation benchmark based on the National Renewable Energy Laboratory (NREL) FAST simulator is employed. Results show the superior performance of the proposed method by comparison with other relevant methods.

Neural matrix factorization++ based recommendation system.

In recent years, Recommender System (RS) research work has covered a wide variety of Artificial Intelligence techniques, ranging from traditional Matrix Factorization (MF) to complex Deep Neural Networks (DNN). Traditional Collaborative Filtering (CF) recommendation methods such as MF, have limited learning capabilities as it only considers the linear combination between user and item vectors. For learning non-linear relationships, methods like Neural Collaborative Filtering (NCF) incorporate DNN into CF methods. Though, CF methods still suffer from cold start and data sparsity. This paper proposes an improved hybrid-based RS, namely Neural Matrix Factorization++ (NeuMF++), for effectively learning user and item features to improve recommendation accuracy and alleviate cold start and data sparsity. NeuMF++ is proposed by incorporating effective latent representation into NeuMF via Stacked Denoising Autoencoders (SDAE). NeuMF++ can also be seen as the fusion of GMF++ and MLP++. NeuMF is an NCF framework which associates with GMF (Generalized Matrix Factorization) and MLP (Multilayer Perceptrons). NeuMF achieves state-of-the-art results due to the integration of GMF linearity and MLP non-linearity. Concurrently, incorporating latent representations has shown tremendous improvement in GMF and MLP, which result in GMF++ and MLP++. Latent representation obtained through the SDAEs' latent space allows NeuMF++ to effectively learn user and item features, significantly enhancing its learning capability. However, sharing feature extractions among GMF++ and MLP++ in NeuMF++ might hinder its performance. Hence, allowing GMF++ and MLP++ to learn separate features provides more flexibility and greatly improves its performance. Experiments performed on a real-world dataset have demonstrated that NeuMF++ achieves an outstanding result of a test root-mean-square error of 0.8681. In future work, we can extend NeuMF++ by introducing other auxiliary information like text or images. Different neural network building blocks can also be integrated into NeuMF++ to form a more robust recommendation model.

Fast probability estimation of HVDC successive commutation failure caused by AC grid cascading failures

For line-commutated converter (LCC) based HVDC system, successive commutation failure (SCF) can result in continual cessations of power transmission or even forced blocking of the DC system. AC grid cascading failure near inverters is an important reason for the SCF. The SCF probability is utilized to represent the impact of AC grid cascading failures on DC systems. This paper proposes a fast probability estimation method of SCF based on sequential importance sampling (SIS) and stacked denoising autoencoder (SDAE). Firstly, the probability estimation of SCF caused by cascading failures is transformed into a statistical analysis problem. Secondly, considering AC line failures have different impacts on DC system, a cascading failure sampling method based on SIS is presented to efficiently generate cascading failure samples and evaluate the SCF probability. Thirdly, SDAE based assessment network is established to quickly assess the impact of AC failures on SCF. Topology features related to AC grid structure and short-circuit fault locations are selected as inputs of the assessment network. Finally, real-life hybrid AC/DC grids are adopted to demonstrate the validity of the proposed method.

Ground Target Recognition Using Carrier-Free UWB Radar Sensor With a Semi-Supervised Stacked Convolutional Denoising Autoencoder

This paper proposes a scheme for ground-target recognition based on the carrier-free ultra-wideband (UWB) radar sensor for the first time. Carrier-free UWB system emits an extremely short pulse, which can provide potential advantages over other UWB systems, such as high range resolution, short blind range, enhanced anti-multipath jamming ability. These characteristics above guarantee that UWB echoes contain more comprehensive and detailed information with respect to the target of interest. Feature extraction is fundamental and crucial for target recognition. In this paper a deep network named semi-supervised stacked convolutional denoising autoencoder (SCDAE) is developed to extract discriminative features. As an extension of stacked denoising autoencoders (SDAE), SCDAE replaces fully-connected layers with one-dimensional convolutional layers as middle structures. In order to capture essential signatures exactly and improve classification accuracy, we build up a semi-supervised learning mechanism via binding a label regularization term with SCDAE. Moreover, given that echoes observed at different angles belonging to the same target are different, a new multi-level label coding method is proposed and embedded in SCDAE. Experimental results demonstrate that the proposed algorithm can effectively learn essential representation and improve classification accuracy in the presence of low signal-to-noise ratios (SNRs), making it very suitable for use in a classification scheme.

Online Probabilistic Power Flow with Tie-line Power Transfer: A Stacked Denoising Auto-Encoders Method

Based on stacked denoising auto-encoders (SDAE) and combined with the Monte-Carlo simulation (MCS) method, this paper proposes a hybrid algorithm (SDAE-Based-PPF, SPPF) for online calculation of probabilistic power flow (PPF) with tie-line power transfer. By incorporating the deep structure and reconstructive strategy of SDAE, this paper extracts high-level features of nonlinear power flow equations, thereby constructing SDAE-based power flow models. Then, the parameters of SDAE-based power flow models, with improved training speed and accuracy, are optimized by mini-batch gradient descent algorithm, momentum learning rate and introduced cross-entropy loss function. The SDAE-based power flow models are developed to approximate power flow equations with unidirectional extracted high-level features, so as to enable non-iterative solvability judgment and calculation of power flow, therefore fast and accurate. Benefiting from the parallelizability, speed and accuracy of SDAE, the proposed SDAE-based power flow models, with unsolved random samples generated by the MCS method, are able to calculate all samples simultaneously, and this enables PPF to be calculated online. Finally, numerical results are implemented on three standard IEEE test cases to verify the effectiveness of proposed SDAE-based power flow models and SPPF method.

A Winner‐Take‐All Autoencoder Based Pieceswise Linear Model for Nonlinear Regression with Missing Data

AbstractMissing data is a prevailing problem in predictive analytics. In this paper, a winner‐take‐all (WTA) autoencoder‐based piecewise linear model is developed to solve the nonlinear regression problem under the missing value scenario, which consists of two parts: an overcomplete WTA autoencoder and a gated linear network. The overcomplete WTA autoencoder is a stacked denoising autoencoder (SDAE) designed to play two roles: (1) to estimate the missing values; (2) to realize a sophisticated partitioning by generating a broad set of binary gate control sequences. Besides, an iterative algorithm with renewed teacher signals is developed to train the SDAE. On the other hand, the gated linear network with the generated binary gate control sequences implements a flexible piecewise linear model for nonlinear regression. By composing a quasi‐linear kernel based on the gate control sequences, the piecewise linear model is then identified in the same way as a support vector regression. Experimental results have shown that our proposed hybrid model has a better performance than traditional models. © 2021 Institute of Electrical Engineers of Japan. Published by Wiley Periodicals LLC.

Intelligent sensor validation for sustainable influent quality monitoring in wastewater treatment plants using stacked denoising autoencoders

Wastewater treatment plants (WWTPs) influent conditions can dramatically affect a treatment unit's state and effluent quality. WWTP sensors may record faulty measurements due to abnormal events or the malfunction of the system, leading to serious problems in the system's operation and the violation of effluent discharge standards. Therefore, automatic fault detection and faulty data reconciliation are crucial for an efficient and stable WWTP monitoring. In this study, a holistic framework for sensor validation of WWTP influent conditions is presented considering the non-linearity, measurement noise, and complexity of the WWTP's data. A stacked denoising autoencoder (SDAE) model is proposed to detect, identify, and reconcile faulty data based on data from a real WWTP in South Korea. The proposed SDAE architecture presented a detection rate (DR) between 74% and 98%. The faulty sensor was identified using an SDAE-based sensor validity index (SVI). Data reconciliation showed that the SDAE was the most suitable reconciliation method based on the root mean square error (RMSE) for total nitrogen (TN) influent conditions of 4.04 mg N/L. Finally, faulty, noisy, and reconciled measurements were evaluated in a WWTP model to determine the proposed method's resilience potential.

Damage Identification Based on the Loaded Frequency Changes and Deep Learning for Bridge

Frequency is widely used in damage identification as the dynamic characteristic parameter with the simplest measurement method and the highest measurement accuracy, but the frequency changes are nonunique parameters of damage location for the symmetrical structure. To improve the method of damage location identification based on frequency changes for bridge, the bridge damage identification network based on loaded frequency changes and deep learning theory is proposed by introducing the moving vehicle detection method. A continuous beam example is used for testing, and the test results show that the identification network based on the bridge load frequency changes and the Stacked Denoising Auto-Encoder(SDAE) network can effectively locate the damage of bridge, which enables the frequency changes data to overcome the nonunique limitation of the symmetrical structure, enhances the expression ability of the parameters and improves the effectiveness and the anti-noise property of bridge damage location identification.

Deep Multimodal Learning and Fusion Based Intelligent Fault Diagnosis Approach

Industrial Internet of Things (IoT) connecting society and industrial systems represents a tremendous and promising paradigm shift. With IoT, multimodal and heterogeneous data from industrial devices can be easily collected, and further analyzed to discover device maintenance and health related potential knowledge behind. IoT data-based fault diagnosis for industrial devices is very helpful to the sustainability and applicability of an IoT ecosystem. But how to efficiently use and fuse this multimodal heterogeneous data to realize intelligent fault diagnosis is still a challenge. In this paper, a novel Deep Multimodal Learning and Fusion (DMLF) based fault diagnosis method is proposed for addressing heterogeneous data from IoT environments where industrial devices coexist. First, a DMLF model is designed by combining a Convolution Neural Network (CNN) and Stacked Denoising Autoencoder (SDAE) together to capture more comprehensive fault knowledge and extract features from different modal data. Second, these multimodal features are seamlessly integrated at a fusion layer and the resulting fused features are further used to train a classifier for recognizing potential faults. Third, a two-stage training algorithm is proposed by combining supervised pre-training and fine-tuning to simplify the training process for deep structure models. A series of experiments are conducted over multimodal heterogeneous data from a gear device to verify our proposed fault diagnosis method. The experimental results show that our method outperforms the benchmarking ones in fault diagnosis accuracy.

Denoising and feature extraction of weld seam profiles by stacked denoising autoencoder

Active vision sensing is widely used in intelligent robotic welding for bead detection and tracking. Disturbed by welding noise such as arc light and spatter, it is a hard work to extract the laser stripe and feature values. This paper presents a method for denoising and feature extraction of weld seam profiles with strong welding noise in gas metal arc welding (GMAW) process by using stacked denoising autoencoder (SDAE). This algorithm encodes the images of various butt joints with strong welding noise to several useful intermediate representations, which can be decoded to the image of pure laser stripe in 1-pixel width. The results show little deviations when there are large spatters across the laser stripe. A back propagation neural network (BPNN) is developed to verify the reliability of the intermediate representations gotten from the encoder, in which the intermediate representations are input neurons and the weld seam width is output neuron. The average width error in training dataset and testing dataset is 0.042 mm and 0.061 mm. The results show that this algorithm can extract the weld seam profiles with strong welding noise and extract feature values accurately.

Rolling Bearing Fault Classification Based on Stacked Denoising Auto Encoders

Aiming at the characteristics of large capacity and diversity of rolling bearing fault data, an intelligent rolling bearing fault diagnosis method based on stacked denoising auto encoders was proposed. Firstly, Principal Component Analysis was used to reduce the dimension of the original data, and the redundant information is deleted. Then, three de-noising auto-encoders are created to train the bearing data. Then, a stack de-noising auto-encoder with three hidden layers is stacked with the trained DAE for reverse optimization. Finally, the features are input into soft-max classifier to realize rolling bearing fault diagnosis. The experimental results show that SDAE network can extract fault features effectively, and it is better than back propagation neural network in generalization and fault classification accuracy.

An ECG Denoising Method Based on the Generative Adversarial Residual Network

High-quality and high-fidelity removal of noise in the Electrocardiogram (ECG) signal is of great significance to the auxiliary diagnosis of ECG diseases. In view of the single function of traditional denoising methods and the insufficient performance of signal details after denoising, a new method of ECG denoising based on the combination of the Generative Adversarial Network (GAN) and Residual Network is proposed. The method adopted in this paper is based on the GAN structure, and it restructures the generator and discriminator. In the generator network, residual blocks and Skip-Connecting are used to deepen the network structure and better capture the in-depth information in the ECG signal. In the discriminator network, the ResNet framework is used. In order to optimize the noise reduction process and solve the lack of local relevance considering the global ECG problem, the differential function and overall function of the maximum local difference are added in the loss function in this paper. The experimental results prove that the method used in this article has better performance than the current excellent S-Transform (S-T) algorithm, Wavelet Transform (WT) algorithm, Stacked Denoising Autoencoder (S-DAE) algorithm, and Improved Denoising Autoencoder (I-DAE) algorithm. Experiments show that the Root Mean Square Error (RMSE) of this method in the Massachusetts Institute of Technology and Beth Israel Hospital (MIT-BIH) noise pressure database is 0.0102, and the Signal-to-Noise Ratio (SNR) is 40.8526 dB, which is compared with that of the most advanced experimental methods. Our method improves the SNR by 88.57% on average. Besides the three noise intensities for comparison experiments, additional noise reduction experiments are also performed under four noise intensities in our paper. The experimental results verify the scientific nature of the model, which is that our method can effectively retain the important information conveyed by the original signal.

Research on Movie Recommendation Algorithm Based on Stack De-noising Auto-encoder

Focused on the problems that the randomness of the noise-adding operation in the de-noising auto-encoder (DAE), and the data matrix does not consider the impact of trusted users on the deep preferences of target users, this paper proposes a recommendation algorithm based on stack de-noising auto-encoder (SDAE) which integrates the preferences of trusted users. Firstly, the score vector is used as the input of the auto-encoder, and the mask vector is designed to train the potential preference of the target user. Secondly, the deep preference of the target user and the trusted user is obtained by the weighted fusion of the features of the two hidden layers of the auto-encoder. Thirdly, in order to reduce the impact of noise on the prediction accuracy, the cascaded auto-encoder model is constructed and trained according to the greedy training method layer by layer training. Finally, SDAE model is compared with other models on different data sets. The experimental results show that SDAE model has better recommendation performance.

SDAE-based feature selection method for biological Omics data

The advancement of Omics technology has led to a surge in molecular and cell profiling data for mechanism study. Large amount of data and complex data structure pose a great challenge to data analysis. Modern machine learning methods such as deep learning are expected to take advantage such big data for accurate disease prediction or other related tasks. However, large feature number may bring large amount redundant information and adversely affect the accuracy of a classifier. To this end, feature selection methods can remove redundant information and help the model achieve higher accuracy by selecting informative features. In this paper, we propose a two-step deep learning-based method combining stacked denoising autoencoders (SDAE) with SVM-RFE to accomplish the task of feature selection. We compared our method with other related methods and the results showed that our approach achieved a better performance than other methods when using the TCGA datasets.

A Hybrid Deep Network Framework for Android Malware Detection

Android is a growing target for malicious software (malware) because of its popularity and functionality. Malware poses a serious threat to users’ privacy, money, equipment and file integrity. A series of data-driven malware detection methods were proposed. However, there exist two key challenges for these methods: (1) how to learn effective feature representation from raw data; (2) how to reduce the dependence on the prior knowledge or human labors in feature learning. Inspired by the success of deep learning methods in the feature representation learning community, we propose a malware detection framework which starts with learning rich-features by a novel unsupervised feature learning algorithm Merged Sparse Auto-Encoder (MSAE). In order to extract more compact and discriminative feature from the rich-features to further boost the malware detection capability, a hybrid deep network learning algorithm Stacked Hybrid Learning MSAE and SDAE (SHLMD) is established by further incorporating a classical deep learning method Stacked Denoising Auto-encoders (SDAE). After that, we feed the feature learned by MSAE and SHLMD respectively to classification algorithms, e.g., Support Vector Machine (SVM) or K-NearestNeighbor (KNN), to train a malware detection model. Evaluation results on two real-world datasets demonstrate that SHLMD achieves 94.46 and 90.57 percent accuracy respectively, which outperforms the classical unsupervised feature representation learning Sparse Auto-encoder (SAE). MSAE performs similarly to SAE. SHLMD can further improve the performance of MSAE and the supervised fine-tuned method SDAE. Besides, we compare the performance of our methods with that of state-of-the-art detection approaches, including classical deep-learning-based methods. Extensive experiments show that our proposed methods are effective enough to detect Android malware.

Research on hyper-spectral remote sensing image classification by applying stacked de-noising auto-encoders neural network

Hyper-spectral image can provide precise information on land surface targets identification and classification thanks to its advanced feature on spectral resolution. However, due to its complicated hyper-dimension data structure, greater challenge is put on the conventional image classification methods for hyper-spectral images. To fill this technical knowledge gap, we introduce a deep learning-based feature extraction method for hyper-spectral data classification. Firstly, we used a Stacked De-noising Auto-encoders(SDAE) to extract the in-depth features of hyper-spectral image data: a large amount of unlabeled data was pre-trained to extract the depth characteristics of pixels. We added random noise into the input layer of the network to form a de-noising auto-encoder machine and added treated inputs to reconstruct original data. An L-BFGS (Limited-memory quasi-Newton code) was used to optimize the loss function. In the top layer, deep neural network was fine-tuned by a Softmax regression classifier. All these improvements directed towards the model to obtain the image element abstraction and robust expression in the classification task of the hyper-spectral images. We tested the model performance on a Hyspex imaging spectrometer image and found that the newly introduced model framework outperforms other traditional methods including principal component analysis (PCA), and support vector machine (SVM) classifier, combined PCA-SVM classifier, and Minimum Noise Fraction Rotation (MNF)-SVM classifier.

Fault Diagnosis of Permanent Magnet Synchronous Motor Based on Stacked Denoising Autoencoder.

As a complex field-circuit coupling system comprised of electric, magnetic and thermal machines, the permanent magnet synchronous motor of the electric vehicle has various operating conditions and complicated condition environment. There are various forms of failure, and the signs of failure are crossed or overlapped. Randomness, secondary, concurrency and communication characteristics make it difficult to diagnose faults. Meanwhile, the common intelligent diagnosis methods have low accuracy, poor generalization ability and difficulty in processing high-dimensional data. This paper proposes a method of fault feature extraction for motor based on the principle of stacked denoising autoencoder (SDAE) combined with the support vector machine (SVM) classifier. First, the motor signals collected from the experiment were processed, and the input data were randomly damaged by adding noise. Furthermore, according to the experimental results, the network structure of stacked denoising autoencoder was constructed, the optimal learning rate, noise reduction coefficient and the other network parameters were set. Finally, the trained network was used to verify the test samples. Compared with the traditional fault extraction method and single autoencoder method, this method has the advantages of better accuracy, strong generalization ability and easy-to-deal-with high-dimensional data features.