Multi-layer Autoencoder Research Articles

Privacy-preserving approach for IoT networks using statistical learning with optimization algorithm on high-dimensional big data environment

In the present digital scenario, the explosion of Internet of Things (IoT) devices makes massive volumes of high-dimensional data, presenting significant data and privacy security challenges. As IoT networks enlarge, certifying sensitive data privacy while still employing data analytics authority is vital. In the period of big data, statistical learning has seen fast progressions in methodological practical and innovation applications. Privacy-preserving machine learning (ML) training in the development of aggregation permits a demander to firmly train ML techniques with the delicate data of IoT collected from IoT devices. The current solution is primarily server-assisted and fails to address collusion attacks among servers or data owners. Additionally, it needs to adequately account for the complex dynamics of the IoT environment. In a large-sized big data environment, privacy protection challenges are additionally enlarged. The data dimensional can have vague meaningful patterns, making it challenging to certify that privacy-preserving models do not destroy the efficacy and accuracy of statistical methods. This manuscript presents a Privacy-Preserving Statistical Learning with an Optimization Algorithm for a High-Dimensional Big Data Environment (PPSLOA-HDBDE) approach. The primary purpose of the PPSLOA-HDBDE approach is to utilize advanced optimization and ensemble techniques to ensure data confidentiality while maintaining analytical efficacy. In the primary stage, the linear scaling normalization (LSN) method scales the input data. Besides, the sand cat swarm optimizer (SCSO)-based feature selection (FS) process is employed to decrease the high dimensionality problem. Moreover, the recognition of intrusion detection takes place by using an ensemble of temporal convolutional network (TCN), multi-layer auto-encoder (MAE), and extreme gradient boosting (XGBoost) models. Lastly, the hyperparameter tuning of the three models is accomplished by utilizing an improved marine predator algorithm (IMPA) method. An extensive range of experimentations is performed to improve the PPSLOA-HDBDE technique’s performance, and the outcomes are examined under distinct measures. The performance validation of the PPSLOA-HDBDE technique illustrated a superior accuracy value of 99.49% over existing models.

Reducing the Dimensionality of Data Arrays Using Multi-Layer Autoencoders in the Task of Classifying Mobile Applications

The problem of reducing the dimension of the initial data arrays to improve the efficiency of mobile application traffic processing is considered. The relevance of the study is due to the need to optimize the volume of transmitted and stored data when working in conditions of limited computing resources, as well as to increase the speed and quality of analytical operations. To solve this problem, multi-layer autoencoders are used, capable of forming compact representations of the source data with minimal losses in their informativeness. The approach is based on the idea of training neural network models that extract the most significant features from the source arrays and are able to restore them with a given level of accuracy. Methods used. During the experiments, various architectures of multilayer autocoders were used, differing in the number of layers and dimensions of hidden representations. The research was conducted on real data sets collected from mobile applications with a wide range of functionality. The analysis was carried out by varying the internal parameters of the networks and evaluating the results through an integral statistical indicator reflecting the degree of compression. This indicator allows you to identify how much the spread of attributes changes when passing data through the autoencoder. Results. To evaluate the filtering properties of multilayer autoencoders, an integral compression indicator is proposed that characterizes the change in the spread of attributes of mobile applications when passing them through an autoencoder of a given structure. The indicator is calculated as the ratio of the standard deviation of the attributes at the input and at the output, which allows you to assess the degree of data compression and the degree of information preservation after processing. It is shown that an increase in the integral compression index indicates a more significant compression of the initial data. It was found that filtering is practically independent of the type of application and lies within 10-20 % for three-layer autoencoders, whereas for five-layer auto-encoders, preference is given to encoders with a minimum dimension of the inner layer. The main novelty of the work lies in the development of an integral statistical indicator that not only reflects the degree of compression of mobile application data, but also takes into account the preservation of the original information structure. Unlike existing approaches, this indicator allows for a systematic comparison of various architectures of autoencoders, taking into account not only the reduction in dimension, but also the quality of recovery of the original information. This creates the basis for a more objective assessment of the effectiveness of multilayer autoencoders in specific application conditions. Practical significance. The proposed methodology may be useful for developers and researchers working on optimizing systems for collecting, storing and processing mobile application data. In conditions of limited computing resources, which are typical for mobile devices and embedded systems, the use of multilayer autoencoders aimed at achieving a given balance between compression and preservation of information provides a significant reduction in the volume of transmitted data. The results of the study can be implemented into existing analytical platforms, monitoring systems and classification of mobile applications.

A novel asymmetrical autoencoder with a sparsifying discrete cosine Stockwell transform layer for gearbox sensor data compression

The lack of an efficient compression model remains a challenge for the wireless transmission of gearbox data in non-contact gear fault diagnosis problems. In this paper, we present a signal-adaptive asymmetrical autoencoder with a transform domain layer to compress sensor signals. First, a new discrete cosine Stockwell transform (DCST) layer is introduced to replace linear layers in a multi-layer autoencoder. A trainable filter is implemented in the DCST domain by utilizing the multiplication property of the convolution. A trainable hard-thresholding layer is applied to reduce redundant data in the DCST layer to make the feature map sparse. In comparison to the linear layer, the DCST layer reduces the number of trainable parameters and improves the accuracy of data reconstruction. Second, training the autoencoder with a sparsifying DCST layer only requires a small number of datasets. The proposed method is superior to other autoencoder-based methods on the University of Connecticut (UoC) and Southeast University (SEU) gearbox datasets, as the average quality score is improved by 2.00% at the lowest and 32.35% at the highest with a limited number of training samples.

Industrial Application of Data-Driven Process Monitoring with an Automatic Selection Strategy for Modeling Data

The increasing scale of industrial processes has significantly motivated the development of data-driven fault detection and diagnosis techniques. The selection of representative fault-free modeling data from operation history is an important prerequisite to establishing a long-term effective process monitoring model. However, industrial data are characterized by a high dimension and multimode, and are also contaminated with both outliers and frequent random disturbances, making automatic modeling data selection a great challenge in industrial applications. In this work, an information entropy-based automatic selection strategy for modeling data is proposed, based on which a general real-time process monitoring framework is developed for a large-scale industrial methanol to olefin unit with multiple operating conditions. Modeling data representing normal operating conditions are automatically selected with only a few manually defined normal samples. A long-term effective process monitoring model is then established based on a multi-layer autoencoder, through which unexpected disturbances in real-time operation can be detected early and the root cause can be preliminarily diagnosed by contribution plots. The adjustment of operating conditions has also been considered through a model update strategy. Details of the proposed data selection strategy and modeling process have been provided to facilitate the industrial application of process monitoring systems by other researchers or companies.

Research on feature extraction and classification of load's reactive power based on data-driven

Driven by the advancement of science and technology, the extensive access of high proportion of renewable distributed energy and power electronic equipment has changed the reactive power characteristics of the user side of the urban distribution network. In order to deeply explore the current status of reactive power characteristics on the urban user side, this paper takes the typical users of a regional power grid in Shanghai as the research object, whose power factor data are collected, and the power factor change curves of users are analyzed by combining deep learning and multidimensional clustering. Firstly, the collected data are preliminarily processed to remove the abnormal data in the total sample, and the incomplete data is filled by the interpolation method. Then, aiming at avoiding the influence of the data with an overhigh dimension on the clustering effect, this paper uses a multi-layer autoencoder deep neural network in deep learning to reduce the dimension of the typical daily power factor change curve set of input and extracts its deep features through unsupervised learning. Finally, the K-means clustering algorithm is used to conduct clustering analysis on the daily power factor change curve of typical users with different power consumption types, including large industry, commerce, resident, and charging service industry, etc. So as to summarize various users' reactive power consumption characteristics and reactive power distribution rules. The analysis results show that the shape of the power factor curves of some users vary significantly in different months, and even the factor fluctuates greatly in the same day.

Inter-subject cognitive workload estimation based on a cascade ensemble of multilayer autoencoders

Machine learning approaches can build a computational model to predict cognitive workload levels by using electroencephalogram (EEG) feature inputs at the same time instant. However, when the EEG signals are recorded on different people, the accuracy of such a model could be impaired due to its incapability of fitting varied statistical distributions across different individuals. To this end, we propose an individual-independent workload estimator, a cascade ensemble of multilayer autoencoders to tackle the individual difference within the EEG features. It could assess the workload levels of an unseen subject by adapting the EEG data recorded from non-overlapped existing subjects. We first construct a deep stacked denoising autoencoder to abstract EEG features from a specific individual. Its shallow weights are optimized with individual-specific geometrical information of the features. Then, to find generalizable feature properties, we introduce Q-statistics to measure the independence between base learners. Finally, a regularized extreme learning machine is used as a cascade meta-classifier to fuse and filter high-level EEG abstractions and determine workload levels. We employ databases from two different experiments to validate our approach. The proposed framework can lead to acceptable accuracy and computational complexity compared to several existing workload classifiers.

A Self-Tuning Cyber-Attacks’ Location Identification Approach for Critical Infrastructures

The integration of the communications network and the Internet of Things in today’s critical infrastructures facilitates intelligent and online monitoring of these systems. However, although critical infrastructure’s digitalization brings tremendous advantages and opportunities for remote access and control, it significantly increases cyber-attack’s vulnerability. Therefore, efficient and proper detection and localization of cyber-attack are paramount for the critical infrastructure’s reliable and secure operation. This article proposes a deep learning-based cyber-attack detection and location identification system for critical infrastructures by constructing new representations and model the system behavior using multilayer autoencoders. The results show that the new representations capture the physical relationships among the measurements and have more discriminant power in distinguishing the location of the attack. Furthermore, the proposed method has outperformed conventional machine learning models under various cyber-attack scenarios using real-world data from the gas pipeline and water distribution supervisory control and data acquisition systems.

Real-Time Data Transmission Scheduling Algorithm for Wireless Sensor Networks Based on Deep Q-Learning

In the industrial environment, the data transmission of Wireless Sensor Networks (WSNs) usually has strict deadline requirements. Improving the reliability and real-time performance of data transmission has become one of the critical issues in WSNs research. One of the main methods to improve the network performance of WSNs is to schedule the transmission process. An effective scheduling algorithm can meet the requirements of a strict industrial environment for network performance, which is of great research significance. Aiming at the problem of concurrent data transmission in WSNs, a real-time data transmission scheduling algorithm based on deep Q-learning is proposed. The algorithm comprehensively considers the influence of the remaining deadline, remaining hops, and unassigned time-slot nodes in the data transmission process, defines the reward function and action selection strategy of Q-learning, and guides the system state information transfer process. At the same time, deep learning and Q-learning are combined to solve the problem of disaster maintenance caused by the large scale of the system state. A multi-layer Stacked Auto Encoder (SAE) network model establishes the state-action mapping relationship, and the Q-learning algorithm updates it. Finally, according to the trained SAE network model, the data transmission scheduling strategy of the system in different states is obtained. The network performance of the proposed data transmission scheduling algorithm is analyzed and evaluated by simulation experiments. The simulation results show that compared with the commonly used heuristic algorithms, the proposed algorithm improves real-time performance and can better meet the data transmission requirements of high reliability and real-time WSNs.

Research on Electronic Information Feature Extraction and Classification Recognition Algorithm Based on Deep Learning

In order to make up for the shortcomings of traditional electronic information characteristics and classification recognition algorithms and improve the application ability of the technology, a special extraction algorithm based on the performance of deep multilayer autoencoders is required, and the training of deep learning network model is not supervised with pretraining and the supervision of the margin-based Fisher guidelines. The regularization means in the process of data generating pretraining and carving will prevent the prefining training. Experimental results on multiple data distributions further determine the efficiency of the algorithm. By analyzing the depth learning model, the image information in electronic information is used as an example, and its extraction technology is discussed, and the direction is indicated further. The experimental results show that the acceptance performance of the DMFA algorithm is improved in some cases compared with similar algorithms. And in Ionosphere, PIMA, IRIS Data Set this algorithm acquires the improvement effect, but the results obtained in other data sets are not ideal, thereby obtaining the most use of electronic information feature extraction and classification recognition.

High accuracy multilayer autoencoder trained classification method for diagnosis of Parkinson’s disease using vocal signals

Dysarthria and dysphonia are two commonly occurring disorders of speech in Parkinson’s disease patient, which are observed in approximately 90 % of the disease cases. It has been reported that these disorders give early sign of Parkinson’s disease. Hence, effective development of diagnostic tools for detecting early biomarkers can help in controlling the symptoms of disease. In this paper, we have proposed a classification model using Multi- layer Autoencoder for feature space reduction. Comparisons among 6 types of classification methods with various feature space sizes were conducted to find out the best performing classification algorithm and feature space size based on accuracy, specificity, and sensitivity. Mel Frequency Cepstral method was used to extract set of features from voice signals of both healthy group and unhealthy group. The model showed promising results in classification when combined with Multi-Layer Autoencoder and SVM classifier.

Automated Grading of Breast Cancer Histopathology Images Using Multilayered Autoencoder

Breast cancer (BC) is the most widely recognized cancer in women worldwide. By 2018, 627,000 women had died of breast cancer (World Health Organization Report 2018). To diagnose BC, the evaluation of tumours is achieved by analysis of histological specimens. At present, the Nottingham Bloom Richardson framework is the least expensive approach used to grade BC aggressiveness. Pathologists contemplate three elements, 1. mitotic count, 2. gland formation, and 3. nuclear atypia, which is a laborious process that witness's variations in expert's opinions. Recently, some algorithms have been proposed for the detection of mitotic cells, but nuclear atypia in breast cancer histopathology has not received much consideration. Nuclear atypia analysis is performed not only to grade BC but also to provide critical information in the discrimination of normal breast, non-invasive breast (usual ductal hyperplasia, atypical ductal hyperplasia) and pre-invasive breast (ductal carcinoma in situ) and invasive breast lesions. We proposed a deep-stacked multi-layer autoencoder ensemble with a softmax layer for the feature extraction and classification process. The classification results show the value of the multi-layer autoencoder model in the evaluation of nuclear polymorphisms. The proposed method has indicated promising results, making them more fit in breast cancer grading.

Joint network embedding of network structure and node attributes via deep autoencoder

Network embedding aims to learn a low-dimensional vector for each node in networks, which is effective in a variety of applications such as network reconstruction and community detection. However, the majority of the existing network embedding methods merely exploit the network structure and ignore the rich node attributes, which tend to generate sub-optimal network representation. To learn more desired network representation, diverse information of networks should be exploited. In this paper, we develop a novel deep autoencoder framework to fuse topological structure and node attributes named FSADA. We firstly design a multi-layer autoencoder which consists of multiple non-linear functions to capture and preserve the highly non-linear network structure and node attribute information. Particularly, we adopt a pre-processing procedure to pre-process the original information, which can better facilitate to extract the intrinsic correlations between topological structure and node attributes. In addition, we design an enhancement module that combines topology and node attribute similarity to construct pairwise constraints on nodes, and then a graph regularization is introduced into the framework to enhance the representation in the latent space. Our extensive experimental evaluations demonstrate the superior performance of the proposed method.

A stacked auto-encoder with scaled conjugate gradient algorithm for Malayalam ASR

Automatic speech recognition (ASR) is entitled to automate natural speech perception and the processing mechanism through analysis in the linguistic and acoustic features of the speech signal. ASR for children is highly challenging due to their developing physical aspects and rapidly changing articulation features. Therefore, ASR for children is still at its infant level. In this work, a stacked multilayer auto-encoder (AE) network is designed for ASR of the Malayalam vowel, articulated by children in the age group of five to ten. The proposed network structured with an unsupervised pre-training followed by supervised training. The pre-training coupled with two layers of sparse auto-encoders and scaled conjugate gradient (SCG) algorithm used for back-propagation. The auto-encoders are used to pre-train the network in an unsupervised (self- supervised) manner with 40,500 features that include Mel frequency cepstral coefficients (MFCC) and its derivatives, spectrogram formants and zero crossing rate (ZCR). In the softmax layer, the pre-trained network retrained in a supervised manner with bottleneck features. Fine-tuning has been applied in the trained network to enhance its performance. The unsupervised and supervised layers are stacked together to form a comprehensive network. The designed network has shown an average accuracy of 97% in training and 89.5% accuracy in the test data-set.

Quantum system compression: A Hamiltonian guided walk through Hilbert space

We present a systematic study of quantum system compression for the evolution of generic many-body problems. The necessary numerical simulations of such systems are seriously hindered by the exponential growth of the Hilbert space dimension with the number of particles. For a \emph{constant} Hamiltonian system of Hilbert space dimension $n$ whose frequencies range from $f_{\min}$ to $f_{\max}$, we show via a proper orthogonal decomposition, that for a run-time $T$, the dominant dynamics are compressed in the neighborhood of a subspace whose dimension is the smallest integer larger than the time-bandwidth product $\delf=(f_{\max}-f_{\min})T$. We also show how the distribution of initial states can further compress the system dimension. Under the stated conditions, the time-bandwidth estimate reveals the \emph{existence} of an effective compressed model whose dimension is derived solely from system properties and not dependent on the particular implementation of a variational simulator, such as a machine learning system, or quantum device. However, finding an efficient solution procedure \emph{is} dependent on the simulator implementation{\color{black}, which is not discussed in this paper}. In addition, we show that the compression rendered by the proper orthogonal decomposition encoding method can be further strengthened via a multi-layer autoencoder. Finally, we present numerical illustrations to affirm the compression behavior in time-varying Hamiltonian dynamics in the presence of external fields. We also discuss the potential implications of the findings for machine learning tools to efficiently solve the many-body or other high dimensional Schr{\"o}dinger equations.

Deep Autoencoder Imaging Method for Electrical Impedance Tomography

Electrical impedance tomography (EIT) is an effective technique for real-time monitoring, visualization, and analysis of industrial process in a noninvasive manner. However, due to the nonlinear and “soft-field” nature of its inverse problem, image reconstruction of EIT is always limited in image resolution and, in particular, the accuracy of identifying object boundaries. In order to solve the above problems, a novel multilayer autoencoder (MLAE) image reconstruction network that consists of a feature extraction module and an image reconstruction module is proposed. In the proposed method, hierarchical structures are applied to increase the forward information flow and the selected appropriate hidden layers can solve the disappearance of the reverse gradient flow. The training process of MLAE containing self-supervised pretraining and supervised fine-tuning can provide better complex nonlinear mapping and improve the model performance. The experimental and analytical results prove that the MLAE image reconstruction method can obtain higher quality images than the typical algorithms and certain methods based on deep learning.

Deep Learning Approach to Power Demand Forecasting in Polish Power System

The paper presents a new approach to predicting the 24-h electricity power demand in the Polish Power System (PPS, or Krajowy System Elektroenergetyczny—KSE) using the deep learning approach. The prediction system uses a deep multilayer autoencoder to generate diagnostic features and an ensemble of two neural networks: multilayer perceptron and radial basis function network and support vector machine in regression model, for final 24-h forecast one-week advance. The period of the data that is the subject of the experiments is 2014–2019, which has been divided into two parts: Learning data (2014–2018), and test data (2019). The numerical experiments have shown the advantage of deep learning over classical approaches of neural networks for the problem of power demand prediction.

Screening for chronic conditions with reproductive factors using a machine learning based approach

A large proportion of cases with chronic conditions including diabetes or pre-diabetes, hypertension and dyslipidemia remain undiagnosed. To include reproductive factors (RF) might be able to improve current screening guidelines by providing extra effectiveness. The objective is to study the relationships between RFs and chronic conditions’ biomarkers. A cross-sectional study was conducted. Demographics, RFs and metabolic biomarkers were collected. The relationship of the metabolic biomarkers were shown by correlation analysis. Principal component analysis (PCA) and autoencoder were compared by cross-validation. The better one was adopted to extract a single marker, the general chronic condition (GCC), to represent the body’s chronic conditions. Multivariate linear regression was performed to explore the relationship between GCC and RFs. In total, 1,656 postmenopausal females were included. A multi-layer autoencoder outperformed PCA in the dimensionality reduction performance. The extracted variable by autoencoder, GCC, was verified to be representative of three chronic conditions (AUC for patoglycemia, hypertension and dyslipidemia were 0.844, 0.824 and 0.805 respectively). Linear regression showed that earlier age at menarche (OR = 0.9976) and shorter reproductive life span (OR = 0.9895) were associated with higher GCC. Autoencoder performed well in the dimensionality reduction of clinical metabolic biomarkers. Due to high accessibility and effectiveness, RFs have potential to be included in screening tools for general chronic conditions and could enhance current screening guidelines.

Impact identification using nonlinear dimensionality reduction and supervised learning

Real-time monitoring systems that can automatically locate and identify impacts as they occur have become increasingly attractive for ensuring safety and preventing catastrophic accidents in aerospace structures. In most cases, a set of piezoelectric transducers distributed over the structure captures strain–time data, which are preprocessed to extract relevant features that are fed to a supervised learning algorithm to detect, locate, and quantify impacts. The best results achieved to date in feature extraction for impact identification have been obtained with the use of principal component analysis (PCA). However, this technique cannot handle complex nonlinear data. The primary contribution of this study is the implementation of a novel impact identification algorithm that uses a supervised learning algorithm called linear approximation with maximum entropy (LME) in conjunction with different linear and nonlinear dimensionality reduction techniques, including PCA, kernel PCA, Isomap, local linear embedding (LLE), and multilayer autoencoders. The performance of LME with the different reduction techniques is tested with two experimental applications. The results show that the techniques that do not employ graphs, such as PCA, kernel PCA, and autoencoders, perform better, and the method that provides the best results is LME in conjunction with autoencoders. It is further demonstrated that LME with autoencoders works better than the algorithms available in the literature for similar problems.

Neural Networks Principal Component Analysis for estimating the generative multifactor model of returns under a statistical approach to the Arbitrage Pricing Theory. Evidence from the Mexican Stock Exchange

A nonlinear principal component analysis (NLPCA) represents an extension of the standard principal component analysis (PCA) that overcomes the limitation of the PCA’s assumption about the linearity of the model. The NLPCA belongs to the family of nonlinear versions of dimension reduction or the extraction techniques of underlying features, including nonlinear factor analysis and nonlinear independent component analysis, where the principal components are generalized from straight lines to curves. The NLPCA can be achieved via an artificial neural network specification where the PCA classic model is generalized to a nonlinear mode, namely, Neural Networks Principal Component Analysis (NNPCA). In order to extract a set of nonlinear underlying systematic risk factors, we estimate the generative multifactor model of returns in a statistical version of the Arbitrage Pricing Theory (APT), in the context of the Mexican Stock Exchange. We used an auto-associative multilayer perceptron neural network or autoencoder, where the ‘bottleneck’ layer represented the nonlinear principal components, or in our context, the scores of the underlying factors of systematic risk. This neural network represents a powerful technique capable of performing a nonlinear transformation of the observed variables into the nonlinear principal components, and to execute a nonlinear mapping that reproduces the original variables. We propose a network architecture capable of generating a loading matrix that enables us to make a first approach to the interpretation of the extracted latent risk factors. In addition, we used a two stage methodology for the econometric contrast of the APT involving first, a simultaneous estimation of the system of equations via Seemingly Unrelated Regression (SUR), and secondly, a cross-section estimation via Ordinary Least Squared corrected by heteroskedasticity and autocorrelation by means of the Newey-West heteroskedasticity and autocorrelation consistent covariances estimates (HEC). The evidence found shows that the reproductions of the observed returns using the estimated components via NNPCA are suitable in almost all cases; nevertheless, the results in an econometric contrast lead us to a partial acceptance of the APT in the samples and periods studied.

GSAE: an autoencoder with embedded gene-set nodes for genomics functional characterization

BackgroundBioinformatics tools have been developed to interpret gene expression data at the gene set level, and these gene set based analyses improve the biologists’ capability to discover functional relevance of their experiment design. While elucidating gene set individually, inter-gene sets association is rarely taken into consideration. Deep learning, an emerging machine learning technique in computational biology, can be used to generate an unbiased combination of gene set, and to determine the biological relevance and analysis consistency of these combining gene sets by leveraging large genomic data sets.ResultsIn this study, we proposed a gene superset autoencoder (GSAE), a multi-layer autoencoder model with the incorporation of a priori defined gene sets that retain the crucial biological features in the latent layer. We introduced the concept of the gene superset, an unbiased combination of gene sets with weights trained by the autoencoder, where each node in the latent layer is a superset. Trained with genomic data from TCGA and evaluated with their accompanying clinical parameters, we showed gene supersets’ ability of discriminating tumor subtypes and their prognostic capability. We further demonstrated the biological relevance of the top component gene sets in the significant supersets.ConclusionsUsing autoencoder model and gene superset at its latent layer, we demonstrated that gene supersets retain sufficient biological information with respect to tumor subtypes and clinical prognostic significance. Superset also provides high reproducibility on survival analysis and accurate prediction for cancer subtypes.