Stacked Sparse Autoencoder Research Articles

Short-term wind speed forecasting based on the Jaya-SVM model

Wind energy is an emerging environmentally friendly energy source. However, due to the uncertainty and volatility of wind speed, wind energy cannot be effectively exploited, and it is essential to build an accurate wind speed forecasting model. In this paper, a Jaya algorithm-based support vector machine (Jaya-SVM) model is proposed for short-term wind speed forecasting. Different from the typical SVM regression, the most representative features of input data are selected and the hyper-parameters of SVM are optimized by using the Jaya optimization algorithm. To examine the performance of the Jaya-SVM model, seven other wind speed forecasting models, Least Absolute Shrinkage and Selection Operator, Extreme Gradient Boosting model, Multi-Layer Perceptron Regression model, Deep Belief Network, Gaussian Process Regression, Stacked Sparse Autoencoder and Granular Computing method are employed for comparison. The wind speed data collected in Jilin, China is fully utilized. The computational results demonstrate that the Jaya-SVM model generates the best results among all eight models in terms of MAE, MSE, MAPE, R2 and reliability, having the capacity of accurate wind speed forecasting.

Feature Dimension Reduction Using Stacked Sparse Auto-Encoders for Crop Classification with Multi-Temporal, Quad-Pol SAR Data

Crop classification in agriculture is one of important applications for polarimetric synthetic aperture radar (PolSAR) data. For agricultural crop discrimination, compared with single-temporal data, multi-temporal data can dramatically increase crop classification accuracies since the same crop shows different external phenomena as it grows up. In practice, the utilization of multi-temporal data encounters a serious problem known as a “dimension disaster”. Aiming to solve this problem and raise the classification accuracy, this study developed a feature dimension reduction method using stacked sparse auto-encoders (S-SAEs) for crop classification. First, various incoherent scattering decomposition algorithms were employed to extract a variety of detailed and quantitative parameters from multi-temporal PolSAR data. Second, based on analyzing the configuration and main parameters for constructing an S-SAE, a three-hidden-layer S-SAE network was built to reduce the dimensionality and extract effective features to manage the “dimension disaster” caused by excessive scattering parameters, especially for multi-temporal, quad-pol SAR images. Third, a convolutional neural network (CNN) was constructed and employed to further enhance the crop classification performance. Finally, the performances of the proposed strategy were assessed with the simulated multi-temporal Sentinel-1 data for two experimental sites established by the European Space Agency (ESA). The experimental results showed that the overall accuracy with the proposed method was raised by at least 17% compared with the long short-term memory (LSTM) method in the case of a 1% training ratio. Meanwhile, for a CNN classifier, the overall accuracy was almost 4% higher than those of the principle component analysis (PCA) and locally linear embedded (LLE) methods. The comparison studies clearly demonstrated the advantage of the proposed multi-temporal crop classification methodology in terms of classification accuracy, even with small training ratios.

Scattering feature dimension reduction of multitemporal fully PolSAR image based on Stacked Sparse AutoEncoder

Scattering feature dimension reduction of multitemporal fully PolSAR image based on Stacked Sparse AutoEncoder

Deep-Learning-Enabled Security Issues in the Internet of Things

In order to explore the application value of deep learning denoising autoencoder (DAE) in Internet-of-Things (IoT) fusion security, in this study, a hierarchical intrusion security detection model stacked DAE supporting vector machine (SDAE-SVM) is constructed based on the three-layer neural network of self-encoder. The sample data after dimension reduction are obtained by layer by layer pretraining and fine-tuning. The traditional deep learning algorithms [stacked noise autoencoder (SNAE), stacked autoencoder (SAE), stacked contractive autoencoder (SCAE), stacked sparse autoencoder (SSAE), deep belief network (DBN)] are introduced to carry out the comparative simulation with the model in this study. The results show that when the encoder in the model is a 4-layer network structure, the accuracy rate (Ac) of the model is the highest (97.83%), the false-negative rate (Fn) (1.27%) and the false-positive rate (Fp) (3.21%) are the lowest. When the number of nodes in the first hidden layer is about 110, the model accuracy is about 98%. When comparing the model designed in this study with the common feature dimension reduction methods, the Ac, Fn, and Fp of this model are the best, which are 98.12%, 3.21%, and 1.27%, respectively. When compared with other deep learning algorithms of the same type, the recognition rate, Ac, error rate, and rejection rate show good results. In multiple data sets, the recognition rate, Ac, error rate, and rejection rate of the model in this study are always better than the traditional deep learning algorithms. In conclusion, when deep learning SDAE is applied to IoT convergence-based intrusion security detection, the detection load can be reduced, the detection effect can be improved, and the operation is more secure and stable.

Gated Recurrent Unit Networks for Remaining Useful Life Prediction

Abstract Remaining useful life prediction is a key procedure for prognostics and health management. However, traditional data-driven methods rely on handcrafted feature selection from the whole range of time series data, which may not obtain the temporal information for complex systems. This study proposes a gated recurrent unit networks based approach to predict remaining useful life. First, time window approach is applied on sample preparation for multiple sensor data. In particular, unsupervised stacked sparse autoencoder is utilized to automatically extract nonlinear features, then the selected features are fed into gated recurrent unit based recurrent neural networks to predict remaining useful life. The effectiveness of the proposed method is demonstrated on the commercial modular aero-propulsion system simulation data from NASA. Experimental results validate that the proposed approach achieves better prediction performance than other methods.

Brain Tumor Detection by Using Stacked Autoencoders in Deep Learning.

Brain tumor detection depicts a tough job because of its shape, size and appearance variations. In this manuscript, a deep learning model is deployed to predict input slices as a tumor (unhealthy)/non-tumor (healthy). This manuscript employs a high pass filter image to prominent the inhomogeneities field effect of the MR slices and fused with the input slices. Moreover, the median filter is applied to the fused slices. The resultant slices quality is improved with smoothen and highlighted edges of the input slices. After that, based on these slices' intensity, a 4-connected seed growing algorithm is applied, where optimal threshold clusters the similar pixels from the input slices. The segmented slices are then supplied to the fine-tuned two layers proposed stacked sparse autoencoder (SSAE) model. The hyperparameters of the model are selected after extensive experiments. At the first layer, 200 hidden units and at the second layer 400 hidden units are utilized. The testing is performed on the softmax layer for the prediction of the images having tumors and no tumors. The suggested model is trained and checked on BRATS datasets i.e., 2012(challenge and synthetic), 2013, and 2013 Leaderboard, 2014, and 2015 datasets. The presented model is evaluated with a number of performance metrics which demonstrates the improved performance.

Multiparametric deep learning tissue signatures for a radiological biomarker of breast cancer: Preliminary results.

PurposeDeep learning is emerging in radiology due to the increased computational capabilities available to reading rooms. These computational developments have the ability to mimic the radiologist and may allow for more accurate tissue characterization of normal and pathological lesion tissue to assist radiologists in defining different diseases. We introduce a novel tissue signature model based on tissue characteristics in breast tissue from multiparametric magnetic resonance imaging (mpMRI). The breast tissue signatures are used as inputs in a stacked sparse autoencoder (SSAE) multiparametric deep learning (MPDL) network for segmentation of breast mpMRI.MethodsWe constructed the MPDL network from SSAE with 5 layers with 10 nodes at each layer. A total cohort of 195 breast cancer subjects were used for training and testing of the MPDL network. The cohort consisted of a training dataset of 145 subjects and an independent validation set of 50 subjects. After segmentation, we used a combined SAE‐support vector machine (SAE‐SVM) learning method for classification. Dice similarity (DS) metrics were calculated between the segmented MPDL and dynamic contrast enhancement (DCE) MRI‐defined lesions. Sensitivity, specificity, and area under the curve (AUC) metrics were used to classify benign from malignant lesions.ResultsThe MPDL segmentation resulted in a high DS of 0.87 ± 0.05 for malignant lesions and 0.84 ± 0.07 for benign lesions. The MPDL had excellent sensitivity and specificity of 86% and 86% with positive predictive and negative predictive values of 92% and 73%, respectively, and an AUC of 0.90.ConclusionsUsing a new tissue signature model as inputs into the MPDL algorithm, we have successfully validated MPDL in a large cohort of subjects and achieved results similar to radiologists.

Combustion stability monitoring through flame imaging and stacked sparse autoencoder based deep neural network

Combustion instability is a well-known problem in the combustion processes and closely linked to lower combustion efficiency and higher pollutant emissions. Therefore, it is important to monitor combustion stability for optimizing efficiency and maintaining furnace safety. However, it is difficult to establish a robust monitoring model with high precision through traditional data-driven methods, where prior knowledge of labeled data is required. This study proposes a novel approach for combustion stability monitoring through stacked sparse autoencoder based deep neural network. The proposed stacked sparse autoencoder is firstly utilized to extract flame representative features from the unlabeled images, and an improved loss function is used to enhance the training efficiency. The extracted features are then used to identify the classification label and stability index through clustering and statistical analysis. Classification and regression models incorporating the stacked sparse autoencoder are established for the qualitative and quantitative characterization of combustion stability. Experiments were carried out on a gas combustor to establish and evaluate the proposed models. It has been found that the classification model provides an F1-score of 0.99, whilst the R-squared of 0.98 is achieved through the regression model. Results obtained from the experiments demonstrated that the stacked sparse autoencoder model is capable of extracting flame representative features automatically without having manual interference. The results also show that the proposed model provides a higher prediction accuracy in comparison to the traditional data-driven methods and also demonstrates as a promising tool for monitoring the combustion stability accurately.

Tool Wear Prediction via Multidimensional Stacked Sparse Autoencoders With Feature Fusion

Tool wear prediction is of critical importance to maintain the desired part quality and improve productivity. Inspired by the successful application of deep learning in many condition monitoring tasks. In this article, a novel modeling framework is presented, which includes multiple stacked sparse autoencoders and a nonlinear regression function for tool wear prediction. Multiple stacked sparse autoencoders consists of two main structures. One model is designed with multidimensional stacked sparse autoencoders, which can learn more features from different feature domains in the raw vibration signal, and another single-dimensional stacked sparse autoencoders is used for feature fusion and deeper features learning. And a modified loss function is applied that improves the learning ability. In addition, due to the good properties of tool wear process in nonstationarity and complex nonlinear, a nonlinear regression function is utilized to enhance the progressive tool wear prediction tasks. A dataset from a real manufacturing process is used to evaluate the performance of the proposed modeling framework. Experimental results show that tool wear can be predicted accurately and stably by the proposed tool wear predictive model, which outperforms the already developed methods.

A Deep Learning Architecture for P300 Detection with Brain-Computer Interface Application

In this paper, a brain-computer interface (BCI) system for character recognition is proposed based on the P300 signal. A P300 speller is used to spell the word or character without any muscle movement. P300 detection is the first step to detect the character from the electroencephalogram (EEG) signal. The character is recognized from the detected P300 signal. In this paper, sparse autoencoder (SAE) and stacked sparse autoencoder (SSAE) based feature extraction methods are proposed for P300 detection. This work also proposes a fusion of deep-features with the temporal features for P300 detection. A SSAE technique extracts high-level information about input data. The combination of SSAE features with the temporal features provides abstract and temporal information about the signal. An ensemble of weighted artificial neural network (EWANN) is proposed for P300 detection to minimize the variation among different classifiers. To provide more importance to the good classifier for final classification, a higher weightage is assigned to the better performing classifier. These weights are calculated from the cross-validation test. The model is tested on two different publicly available datasets, and the proposed method provides better or comparable character recognition performance than the state-of-the-art methods.

Discriminative Feature Learning With Distance Constrained Stacked Sparse Autoencoder for Hyperspectral Target Detection

Target detection (TD) is one of the major tasks in hyperspectral image (HSI) processing, and its performance is greatly affected by the background. Feature extraction (FE) has been an effective way to mine discriminative information, especially FE based on deep learning, which can learn the intrinsic properties of data to further improve the detection performance. Unlike supervised networks, unsupervised stacked sparse autoencoders (SSAEs) can learn deep and nonlinear features without any labeled data. However, SSAEs usually require a supervised fine-tuned model to obtain better discrimination, which is not feasible for TD, since the prior information is generally insufficient. In this letter, we introduce a distance constraint that is added to the SSAE to form a new distance constrained SSAE (DCSSAE) network. Specifically, the distance constraint maximizes the distinction between the target pixels and other background pixels in the feature space. Then, using the discriminative features learned from the DCSSAE, a simple detector using radial basis function kernel is derived for background suppression. Experiments on two HSIs demonstrate that the deep spectral features learned from the DCSSAE are more distinguishable, and our proposed detector, namely, the DCSSAE detector, outperforms several popular detectors, especially in background suppression.

Snapshot High Dynamic Range Imaging via Sparse Representations and Feature Learning

Bracketed High Dynamic Range (HDR) imaging architectures acquire a sequence of Low Dynamic Range (LDR) images in order to either produce a HDR image or an “optimally” exposed LDR image, achieving impressive results under static camera and scene conditions. However, in real world conditions, ghost-like artifacts and noise effects limit the quality of HDR reconstruction. We address these limitations by introducing a post-acquisition snapshot HDR enhancement scheme that generates a bracketed sequence from a small set of LDR images, and in the extreme case, directly from a single exposure. We achieve this goal via a sparse-based approach where transformations between differently exposed images are encoded through a dictionary learning process, while we learn appropriate features by employing a stacked sparse autoencoder (SSAE) based framework. Via experiments with real images, we demonstrate the improved performance of our method over the state-of-the-art, while our single-shot based HDR formulation provides a novel paradigm for the enhancement of LDR imaging and video sequences.

P300 based character recognition using sparse autoencoder with ensemble of SVMs

In this study, a brain–computer interface (BCI) system known as P300 speller is used to spell the word or character without any muscle activity. For P300 signal classification, feature extraction is an important step. In this work, deep feature learning techniques based on sparse autoencoder (SAE) and stacked sparse autoencoder (SSAE) are proposed for feature extraction. Deep feature provides the abstract information about the signal. This work proposes fusion of deep features with the temporal features, which provides abstract and temporal information about the EEG signal. These deep feature and temporal feature are partially complement of each other to represent the EEG signal. For classification of the EEG signal, an ensemble of support vector machines (ESVM) is adopted as it helps to reduce the classifiers variability. In classifier ensemble system, the score of individual classifier is not at the same level. To transform these scores into a common level, min–max normalization is proposed prior to combining them. Min-max normalization scales the classifiers’ score between 0 and 1. The experiments are conducted on three standard public datasets, dataset IIb of BCI Competition II, dataset II of the BCI Competition III and BNCI Horizon dataset. The experimental results show that the proposed method yields better or comparable performance compared to earlier reported techniques.

Multi-objective data-ensemble wind speed forecasting model with stacked sparse autoencoder and adaptive decomposition-based error correction

Accurate wind speed prediction is essential for proper use of wind energy resource. In this paper, a novel hybrid multi-step wind speed forecasting model is developed, which consists of sparse feature extraction, bidirectional deep learning, multi-objective optimization, and adaptive decomposition-based error correction. Apart from the traditional average-based resolution transformation method, a two-layer stacked sparse autoencoder (SSAE) is proposed to extract the hidden representation of original 3 s high-resolution wind speed data. Trained by the data generated from different resolution transformation methods, two bidirectional long short-term memory (BiLSTM) networks serve as base predictors and provide 10-step forecasting results. The results of base predictors are reasonably ensembled by multi-objective multi-universe optimization (MOMVO). Moreover, to reduce the predictable components in error series further, a correction model based on empirical wavelet transform (EWT) and outlier robust extreme learning machine model (ORELM) is constructed to reduce the forecasting error further. The effectiveness of the proposed hybrid model is comprehensively evaluated by a series of experiments. The experimental results demonstrate that: (a) the proposed model is well trained, with great convergence, and an average RMSE of 0.2618 m/s in 10-step forecasting; (b) the proposed model outperforms other existing models in all experimental sites and forecasting steps; (c) the multi-objective optimization algorithm can rationally integrate base predictors to obtain better performance in each step; (d) the proposed residual error correction model can generate more than 78% improvement of RMSE, significantly better than compared correction methods.

Fingerprint Image Segmentation using Deep Features and SVM

The task of fingerprint segmentation is the most important step in an automated fingerprint identification system. It is essential to separate the fingerprint foreground with ridge and valley structure from the background, which usually contains unwanted data hindering an accurate feature extraction. In the proposed method, fingerprint segmentation is treated as a classification problem by classifying the given input image into foreground class or background class. Here, we have used an unsupervised learning algorithm by using Stacked Sparse Autoencoder (SSAE) to learn the deep features which can very well distinguish the background region from foreground one. Finally, these deep features are given to the SVM classifier. The experimental results prove that the proposed method meets the state-of-the-art results in a wide range of applications.

Variety Identification of Orchids Using Fourier Transform Infrared Spectroscopy Combined with Stacked Sparse Auto-Encoder.

The feasibility of using the fourier transform infrared (FTIR) spectroscopic technique with a stacked sparse auto-encoder (SSAE) to identify orchid varieties was studied. Spectral data of 13 orchids varieties covering the spectral range of 4000–550 cm−1 were acquired to establish discriminant models and to select optimal spectral variables. K nearest neighbors (KNN), support vector machine (SVM), and SSAE models were built using full spectra. The SSAE model performed better than the KNN and SVM models and obtained a classification accuracy 99.4% in the calibration set and 97.9% in the prediction set. Then, three algorithms, principal component analysis loading (PCA-loading), competitive adaptive reweighted sampling (CARS), and stacked sparse auto-encoder guided backward (SSAE-GB), were used to select 39, 300, and 38 optimal wavenumbers, respectively. The KNN and SVM models were built based on optimal wavenumbers. Most of the optimal wavenumbers-based models performed slightly better than the all wavenumbers-based models. The performance of the SSAE-GB was better than the other two from the perspective of the accuracy of the discriminant models and the number of optimal wavenumbers. The results of this study showed that the FTIR spectroscopic technique combined with the SSAE algorithm could be adopted in the identification of the orchid varieties.

Learning deep neural networks for node classification

Deep Neural Network (DNN) has made great leaps in image classification and speech recognition in recent years. However, employing DNN for node classification such as in social network remains to be a non-trivial problem. Moreover, the current advanced method of implementing node classification tasks usually takes two steps, i.e. firstly, the embedding vector of the node is obtained through network embedding and then the classifier such as SVM is leveraged to do the task. Distinctly, this may only get the suboptimal solution of the problem. To settle the above issues, a novel Deep Neural Network method for node classification named DNNNC is proposed in the framework of Deep Learning. Specifically, we first get the positive pointwise mutual information (PPMI) matrix from the given adjacency matrix. Then, the data is fed to deep neural network composed of deep stacked sparse autoencoders and softmax layer, which could learn the node representation while encoding the rich nonlinear structural and semantic information and could be well trained for node classification under the DNN framework. Extensive experiments are conducted on real-world network datasets for node classification task and have shown that the proposed model DNNNC outperforms the state-of-the-art method in the view of superior performance.

Breast Cancer Diagnosis Using Feature Ensemble Learning Based on Stacked Sparse Autoencoders and Softmax Regression.

Nowadays, the most frequent cancer in women is breast cancer (malignant tumor). If breast cancer is detected at the beginning stage, it can often be cured. Many researchers proposed numerous methods for early prediction of this Cancer. In this paper, we proposed feature ensemble learning based on Sparse Autoencoders and Softmax Regression for classification of Breast Cancer into benign (non-cancerous) and malignant (cancerous). We used Breast Cancer Wisconsin (Diagnostic) medical data sets from the UCI machine learning repository. The proposed method is assessed using various performance indices like true classification accuracy, specificity, sensitivity, recall, precision, f measure, and MCC. Simulation and result proved that the proposed approach gives better results in terms of different parameters. The prediction results obtained by the proposed approach were very promising (98.60% true accuracy). In addition, the proposed method outperforms the Stacked Sparse Autoencoders and Softmax Regression based (SSAE-SM) model and other State-of-the-art classifiers in terms of various performance indices. Experimental simulations, empirical results, and statistical analyses are also showing that the proposed model is an efficient and beneficial model for classification of Breast Cancer. It is also comparable with the existing machine learning and soft computing approaches present in the related literature.

Deep Learning Based Retrieval of Forest Aboveground Biomass from Combined LiDAR and Landsat 8 Data

Estimation of forest aboveground biomass (AGB) is crucial for various technical and scientific applications, ranging from regional carbon and bioenergy policies to sustainable forest management. However, passive optical remote sensing, which is the most widely used remote sensing data for retrieving vegetation parameters, is constrained by spectral saturation problems and cloud cover. On the other hand, LiDAR data, which have been extensively used to estimate forest structure attributes, cannot provide sufficient spectral information of vegetation canopies. Thus, this study aimed to develop a novel synergistic approach to estimating biomass by integrating LiDAR data with Landsat 8 imagery through a deep learning-based workflow. First the relationships between biomass and spectral vegetation indices (SVIs) and LiDAR metrics were separately investigated. Next, two groups of combined optical and LiDAR indices (i.e., COLI1 and COLI2) were designed and explored to identify their performances in biomass estimation. Finally, five prediction models, including K-nearest Neighbor, Random Forest, Support Vector Regression, the deep learning model, i.e., Stacked Sparse Autoencoder network (SSAE), and multiple stepwise linear regressions, were individually used to estimate biomass with input variables of different scenarios, i.e., (i) all the COLI1 (ACOLI1), (ii) all the COLI2 (ACOLI2), (iii) ACOLI1 and all the optical (AO) and LiDAR variables (AL), and (iv) ACOLI2, AO and AL. Results showed that univariate models with the combined optical and LiDAR indices as explanatory variables presented better modeling performance than those with either optical or LiDAR data alone, regardless of the combination mode. The SSAE model obtained the best performance compared to the other tested prediction algorithms for the forest biomass estimation. The best predictive accuracy was achieved by the SSAE model with inputs of combined optical and LiDAR variables (i.e., ACOLI1, AO and AL) that yielded an R2 of 0.935, root mean squared error (RMSE) of 15.67 Mg/ha, and relative root mean squared error (RMSEr) of 11.407%. It was concluded that the presented combined indices were simple and effective by integrating LiDAR-derived structure information with Landsat 8 spectral data for estimating forest biomass. Overall, the SSAE model with inputs of Landsat 8 and LiDAR integrated information resulted in accurate estimation of forest biomass. The presented modeling workflow will greatly facilitate future forest biomass estimation and carbon stock assessments.

Real-time monitoring of high-power disk laser welding statuses based on deep learning framework

The laser welding quality is determined by its welding statuses, and online welding statuses are depicted by the real-time signals captured from the welding process. A multiple-sensor system is designed to obtain information as comprehensive as possible for welding statuses monitoring. The multiple-sensor system includes an auxiliary illumination visual sensor system, an ultraviolet and visible band visual sensor system, a spectrometer and two photodiodes. The signals captured by different sensors are analyzed via signal or digital image processing algorithms, and distinct features are extracted from these signals to depict the online welding statuses. A deep learning framework based on stacked sparse autoencoder (SSAE) is established to model the relationship between the multi-sensor features and their corresponding welding statuses, and Genetic algorithm (GA) is applied to optimize the parameters of the SSAE framework (SSAE-GA). The proposed framework achieves higher accuracy and stronger robustness in monitoring welding status by comparing with the backpropagation neural network, support vector machine and random forest. Three new experiments with different welding parameters are implemented to validate the effectiveness and generalization of our proposed method. This study provides a novel and accurate method for high-power disk laser welding status monitoring.