Autoencoder Method Research Articles

Autoencoders and their applications in machine learning: a survey

Autoencoders have become a hot researched topic in unsupervised learning due to their ability to learn data features and act as a dimensionality reduction method. With rapid evolution of autoencoder methods, there has yet to be a complete study that provides a full autoencoders roadmap for both stimulating technical improvements and orienting research newbies to autoencoders. In this paper, we present a comprehensive survey of autoencoders, starting with an explanation of the principle of conventional autoencoder and their primary development process. We then provide a taxonomy of autoencoders based on their structures and principles and thoroughly analyze and discuss the related models. Furthermore, we review the applications of autoencoders in various fields, including machine vision, natural language processing, complex network, recommender system, speech process, anomaly detection, and others. Lastly, we summarize the limitations of current autoencoder algorithms and discuss the future directions of the field.

Multi-Kernel Graph Attention Deep Autoencoder for MiRNA-Disease Association Prediction.

Accumulating evidence indicates that microRNAs (miRNAs) can control and coordinate various biological processes. Consequently, abnormal expressions of miRNAs have been linked to various complex diseases. Recognizable proof of miRNA-disease associations (MDAs) will contribute to the diagnosis and treatment of human diseases. Nevertheless, traditional experimental verification of MDAs is laborious and limited to small-scale. Therefore, it is necessary to develop reliable and effective computational methods to predict novel MDAs. In this work, a multi-kernel graph attention deep autoencoder (MGADAE) method is proposed to predict potential MDAs. In detail, MGADAE first employs the multiple kernel learning (MKL) algorithm to construct an integrated miRNA similarity and disease similarity, providing more biological information for further feature learning. Second, MGADAE combines the known MDAs, disease similarity, and miRNA similarity into a heterogeneous network, then learns the representations of miRNAs and diseases through graph convolution operation. After that, an attention mechanism is introduced into MGADAE to integrate the representations from multiple graph convolutional network (GCN) layers. Lastly, the integrated representations of miRNAs and diseases are input into the bilinear decoder to obtain the final predicted association scores. Corresponding experiments prove that the proposed method outperforms existing advanced approaches in MDA prediction. Furthermore, case studies related to two human cancers provide further confirmation of the reliability of MGADAE in practice.

NPCA: a linear dimensionality reduction method using a multilayer perceptron.

Background: Linear dimensionality reduction techniques are widely used in many applications. The goal of dimensionality reduction is to eliminate the noise of data and extract the main features of data. Several dimension reduction methods have been developed, such as linear-based principal component analysis (PCA), nonlinear-based t-distributed stochastic neighbor embedding (t-SNE), and deep-learning-based autoencoder (AE). However, PCA only determines the projection direction with the highest variance, t-SNE is sometimes only suitable for visualization, and AE and nonlinear methods discard the linear projection. Results: To retain the linear projection of raw data and generate a better result of dimension reduction either for visualization or downstream analysis, we present neural principal component analysis (nPCA), an unsupervised deep learning approach capable of retaining richer information of raw data as a promising improvement to PCA. To evaluate the performance of the nPCA algorithm, we compare the performance of 10 public datasets and 6 single-cell RNA sequencing (scRNA-seq) datasets of the pancreas, benchmarking our method with other classic linear dimensionality reduction methods. Conclusion: We concluded that the nPCA method is a competitive alternative method for dimensionality reduction tasks.

Music personalization imputation method based on deep transfer transfer learning

Abstract The current music recommendation is less efficient and cannot meet the needs of most people. This paper combines deep learning and migration learning to research music recommendations. It mainly utilizes the hybrid recommendation method of label recommendation and noise reduction autoencoder to achieve accurate music recommendations after extracting the features of music and the user’s preference features. In addition, in order to explore the recommendation effect of the model in this paper, the deviation between the measured score and the actual score of this model is compared and analyzed as well as the similarity difference of music with different features is explored in terms of HR, Recall, and NDCG index performance. The results show that the number of songs with a deviation of 2 or less in the predicted scores of this recommender system all account for 88% of the total number of songs, which is better than other models. In terms of HR, Recall, and NDCG, this paper’s model has better recommendation performance than other models, and the similarity of music recommendation between the same album and different albums is more than 0.65. This study is of great significance for eliminating the information barriers to enhance and explore the value of music data information.

Segmentation‐enhanced gamma spectrum denoising based on deep learning

AbstractGamma spectrum denoising can reduce the adverse effects of statistical fluctuations of radioactivity, gamma ray scattering, and electronic noise on the measured gamma spectrum. Traditional denoising methods are intricate and require analytical expertise in gamma spectrum analysis. This paper proposes a segmentation‐enhanced Convolutional Neural Network‐Stacked Denoising Autoencoder (CNN‐SDAE) method based on convolutional feature extraction network and stacked denoising autoencoder to achieve gamma spectrum denoising, which adopts the idea of data segmentation to enhance the learning ability of the neural network. By dividing the complete gamma spectrum into multiple segments and then using the segmentation‐enhanced CNN‐SDAE method for denoising, the method can achieve adaptive denoising without manually setting the threshold. The experimental results show that our method can effectively achieve gamma spectrum denoising while retaining the characteristics of the gamma spectrum. Compared with traditional methods, the denoising speed and effectiveness have been significantly improved, and the proposed method demonstrates an approximately 1.72‐fold enhancement in smoothing performance than the empirical mode decomposition method. Furthermore, in terms of retaining gamma spectrum characteristics, it also achieves a performance improvement of approximately three orders of magnitude than the wavelet method.

Comparing the Performance of Deep Denoising Sparse Autoencoder with Other Defense Methods Against Adversarial Attacks for Arabic letters

The aim of this paper is to compare how effectively the Deep Denoising Sparse Autoencoder (DDSA) method performs compared to other defense strategies - like adversarial training, defensive distillation and feature squeezing - in dealing with adversarial attacks for Arabic letters. We strive to evaluate both the accuracy and robustness as well as efficiency of these methods by examining a test set from the Arabic Handwritten Characters Dataset while considering adversarial attacks. Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini and Wagner (C&W) are all part of this. Our research findings demonstrate that DDSA surpasses the rest of the defense methods in terms of classification accuracy and robustness. This exceptional performance is due to the distinctive attributes of DDSA, which concentrate on acquiring distinguishing features and integrating spatial information to improve defense against adversarial perturbations. While it necessitates more computational resources, DDSA's superior performance validates the additional expenses, particularly in critical applications where misclassification may have severe implications.

MTS-LOF: Medical Time-Series Representation Learning via Occlusion-Invariant Features.

Medical time series data are indispensable in healthcare, providing critical insights for disease diagnosis, treatment planning, and patient management. The exponential growth in data complexity, driven by advanced sensor technologies, has presented challenges related to data labeling. Self-supervised learning (SSL) has emerged as a transformative approach to address these challenges, eliminating the need for extensive human annotation. In this study, we introduce a novel framework for Medical Time Series Representation Learning, known as MTS-LOF. MTS-LOF leverages the strengths of Joint-Embedding SSL and Masked Autoencoder (MAE) methods, offering a unique approach to representation learning for medical time series data. By combining these techniques, MTS-LOF enhances the potential of healthcare applications by providing more sophisticated, context-rich representations. Additionally, MTS-LOF employs a multi-masking strategy to facilitate occlusion-invariant feature learning. This approach allows the model to create multiple views of the data by masking portions of it. By minimizing the discrepancy between the representations of these masked patches and the fully visible patches, MTS-LOF learns to capture rich contextual information within medical time series datasets. The results of experiments conducted on diverse medical time series datasets demonstrate the superiority of MTS-LOF over other methods. These findings hold promise for significantly enhancing healthcare applications by improving representation learning. Furthermore, our work delves into the integration of Joint-Embedding SSL and MAE techniques, shedding light on the intricate interplay between temporal and structural dependencies in healthcare data. This understanding is crucial, as it allows us to grasp the complexities of healthcare data analysis.

Locally Masked Convolutional Models for Anomalous Sound Detection by Deep Learning

Anomalous Sound Detection by Deep Learning could be used to automatically detect malfunctions in mechanical products from their operating sounds. With many normal sounds but very few anomalous sounds available for training, one-class classification methods applied to sound spectrograms are the most suitable methods. Among them, the Masked Autoencoder for Distribution Estimation (MADE) method showed state-of-the-art results with transient anomalous sounds but poor results with sustained anomalous sounds. The loss of the spectro-temporal structure in the inputs possibly explaining this issue, we have modified a method recently designed for 2D image completion, taking advantage of the full spectro-temporal structure of the inputs. When evaluated on industrial data, this new method outperforms a baseline Autoencoder method and the original MADE method.

Улучшение точности прогноза состояния Мирового океана за счет оптимального расположения измерителей

The paper examines the impact of sensor placement on the accuracy of the Global ocean state forecasting. A comparison is made between various sensor placement methods, including the arrangement obtained by the Concrete Autoencoder method. To evaluate how sensor placement affects forecast accuracy, a simulation was conducted that emulates a scenario where the initial state of the global ocean significantly deviates from the ground truth. In the experiment, initial conditions for the ocean and ice model were altered, while atmospheric forcing was retained from the control experiment. Subsequently, the model was integrated with the assimilation of data about the ground truth state at the sensor locations. The results showed that the sensor placement obtained using deep learning methods is superior in forecast accuracy to other considered arrays with a comparable number of sensors.

The Hybrid Pathway to Flexible Power Turbines, Part I: Novel Autoencoder Methods for the Automated Optimization of Thermal Probes and Fast Sparse Data Reconstruction, Enabling Real-Time Thermal Analysis

Abstract The global drive toward renewable energy is imposing challenging operating requirements on power turbines. Flexible load-leveling applications must accept more frequent and demanding start-stop cycles. Full transient analyses are too computationally expensive for real-time simulation across all operating regimes so monitoring relies on sparse physical measurements. Alone, these sparse data lack the fidelity for real-time prediction of a complex thermal field. A new hybrid methodology is proposed, coupling data across a range of fidelities to bridge the limitations in the individual analyses. Combining several fidelity methods in parallel, low-order models, corrected by real-time physical measurements, is calibrated with high-fidelity simulations. The multifaceted hybrid approach enables the real-time speed of low-order analysis at high resolution. This paper series develops the critical enabling features of the hybrid method. Real-time cross-fidelity data transition is fundamental to the hybrid methodology. A novel neural network auto-encoder method is presented, facilitating complex thermal profile reconstruction. Uncovering a compressed latent space, auto-encoders leverage underlying data features for fast simulation. Coupled with a dynamic mask and top-k selection, thermal probe placement can be automatically optimized. The auto-encoder method is demonstrated on a turbine casing, reconstructing over 500 h of transient operation in real-time, whilst reducing the required number of measurements by half.

Generative and Contrastive Self-Supervised Learning for Graph Anomaly Detection

Anomaly detection from graph data has drawn much attention due to its practical significance in many critical applications including cybersecurity, finance, and social networks. Existing data mining and machine learning methods are either shallow methods that could not effectively capture the complex interdependency of graph data or graph autoencoder methods that could not fully exploit the contextual information as supervision signals for effective anomaly detection. To overcome these challenges, in this paper, we propose a novel method, Self-Supervised Learning for Graph Anomaly Detection (SL-GAD). Our method constructs different contextual subgraphs (views) based on a target node and employs two modules, generative attribute regression and multi-view contrastive learning for anomaly detection. While the generative attribute regression module allows us to capture the anomalies in the attribute space, the multi-view contrastive learning module can exploit richer structure information from multiple subgraphs, thus abling to capture the anomalies in the structure space, mixing of structure, and attribute information. We conduct extensive experiments on six benchmark datasets and the results demonstrate that our method outperforms state-of-the-art methods by a large margin.

Customization of latent space in semi-supervised Variational AutoEncoder

We propose a novel semi-supervised learning method of Variational AutoEncoder (VAE), which yields a customized latent space through our EXplainable encoder Network (EXoN). The customization involves a manual design of the interpolation and structural constraint, such as proximity, which enhances the interpretability of the latent space. To improve the classification performance, we introduce a new semi-supervised classification method called SCI (Soft-label Consistency Interpolation). Combining the classification loss and the Kullback–Leibler divergence is crucial in constructing an explainable latent space. Additionally, the variability of the generated samples is determined by an active latent subspace, which effectively captures distinctive characteristics. We conduct experiments using the MNIST, SVHN, and CIFAR-10 datasets, and the results demonstrate that our approach yields an explainable latent space while significantly reducing the effort required to analyze representation patterns within the latent space.

Self-supervised pre-training with contrastive and masked autoencoder methods for dealing with small datasets in deep learning for medical imaging

Deep learning in medical imaging has the potential to minimize the risk of diagnostic errors, reduce radiologist workload, and accelerate diagnosis. Training such deep learning models requires large and accurate datasets, with annotations for all training samples. However, in the medical imaging domain, annotated datasets for specific tasks are often small due to the high complexity of annotations, limited access, or the rarity of diseases. To address this challenge, deep learning models can be pre-trained on large image datasets without annotations using methods from the field of self-supervised learning. After pre-training, small annotated datasets are sufficient to fine-tune the models for a specific task. The most popular self-supervised pre-training approaches in medical imaging are based on contrastive learning. However, recent studies in natural image processing indicate a strong potential for masked autoencoder approaches. Our work compares state-of-the-art contrastive learning methods with the recently introduced masked autoencoder approach “SparK” for convolutional neural networks (CNNs) on medical images. Therefore, we pre-train on a large unannotated CT image dataset and fine-tune on several CT classification tasks. Due to the challenge of obtaining sufficient annotated training data in medical imaging, it is of particular interest to evaluate how the self-supervised pre-training methods perform when fine-tuning on small datasets. By experimenting with gradually reducing the training dataset size for fine-tuning, we find that the reduction has different effects depending on the type of pre-training chosen. The SparK pre-training method is more robust to the training dataset size than the contrastive methods. Based on our results, we propose the SparK pre-training for medical imaging tasks with only small annotated datasets.

Digital twin-enhanced predictive maintenance for indoor climate: A parallel LSTM-autoencoder failure prediction approach

Recently, the emergency of predictive maintenance (PdM) in the building industry has expanded from facilities to indoor climates, as air quality is highly relevant to residential health, comfort, and work efficiency. Besides, digital twin (DT) is considered as an effective solution for PdM deployment because it significantly enhances data-driven insights and enables proactive interventions. However, most existing studies on indoor climate focus on condition monitoring or anomaly detection rather than failure prediction, which has higher requirements for data and algorithms. In this study, the remaining useful life (RUL) and time shift (TS) methods are employed to split the prediction problem into the combination of a supervised and an unsupervised subtask, followed by the development of a parallel prediction model integrating the long short-term memory network (LSTM) and autoencoder (AE) methods. Besides, a DT-enabled PdM framework has been proposed for indoor climates, validated through the establishment of an online platform designed to reconstruct the 3D building model and demonstrate real-time monitoring and alert information of indoor climates. Experiments show the effectiveness of the proposed model under different warning times and fault severity through a comparison study with other 4 benchmark models based on a practical dataset collected from different buildings in Singapore, while the practical online platform serves as an illustrative case for future DT-enhanced PdM solutions.

Abstract 16476: Association Between Atrial Fibrillation and Deep Learning Derived Retinal Features Trained on Electrocardiogram versus Coronary Artery Calcium

Introduction: This study aimed to assess the ability of a deep learning algorithm, Reti-AF, developed from retinal photos, to predict atrial fibrillation (AF) incidence. Its predictive performance was evaluated using the UK Biobank and compared to Reti-CAC, a similar algorithm trained on coronary artery calcium (CAC) scores. Hypothesis: We hypothesised that Reti-AF and Reti-CAC could potentially predict the incidence of AF. Methods: Due to the lack of atrial fibrillation (AF) patients in the UK Biobank, a Masked AutoEncoder method was used with a Vision Transformer (ViT) model. The model was first trained on 636,992 fundus images from a health screening dataset, then fine-tuned with an additional 6,068 retinal images from the UK Biobank. The survival analysis of the UK Biobank's longitudinal data was performed to assess the model's predictive performance for AF, with hazard ratios calculated. Results: In longitudinal validation among participants without baseline AF, Reti-AF was significantly associated with AF incidence (HR=5.50, 95% CI, 1.98-2.57, highest tertile of Reti-AF versus the first tertile) in a univariable model. However, this significance was not maintained in a multivariable model (HR=1.23, 95% CI, 0.93-1.30). When Reti-CAC was applied to predict AF, it showed a significant association with AF incidence in both univariable (HR=9.30, 95% CI, 6.68-12.96, highest tertile of Reti-CAC versus the first tertile) and multivariable (HR=1.84, 95% CI, 1.26-2.70) models after adjustment for age, gender, smoking, diabetes, hyperlipidemia, hypertension, and baseline chronic kidney disease. Conclusions: The study found that combining deep learning with retinal photos and coronary artery calcium (CAC) as a reference, led to more accurate prediction of atrial fibrillation (AF) incidence compared to using AF. Furthermore, Reti-CAC was shown to be capable of predicting both cardiovascular disease onset and AF incidence.

Wind Turbine Blade Cracking Detection under Imbalanced Data Using a Novel Roundtrip Auto-Encoder Approach

Imbalanced data cause low recognition of wind turbine blade cracking. Existing data-level augmentation methods, including sampling and generative strategies, may yield lots of high-confidence but low-value samples, which fail to improve the detection of blade cracking. Therefore, this paper designs a novel RTAE (roundtrip auto-encoder) method. Based on the idea of the roundtrip approach, we design two generator networks and two discriminator networks to ensure the cycle mapping between cracking samples and latent variables. Further, by leveraging cycle consistency loss, generated samples fit the distribution of historical cracking samples well. Thus, these generated samples effectively realize data augmentation and improve recognition of blade cracking. Additionally, we apply an auto-encoder method to reduce the dimension of historical samples and thus the complexity of the generator network and discriminator network. Through the analysis of real wind turbine blade cracking data, the recognition of cracking samples is improved by 19.8%, 23.8% and 22.7% for precision, recall and F1-score.

Long-term PM2.5 concentration prediction based on improved empirical mode decomposition and deep neural network combined with noise reduction auto-encoder- A case study in Beijing

Effective prediction of PM2.5 long-term concentration can help reduce exposure risks, but few current studies based on machine learning have been able to credibly predict concentration changing trends after 12 h. To address this issue, this study employed the complementary ensemble empirical mode decomposition (CEEMD) method combined with sample entropy (SE) index, as well as bidirectional long short-term memory (BiLSTM) model stacked denoising auto-encoder (AE) method for better decomposition and prediction of long-term trends in PM2.5 time series data. The new CEEMD-AE-BiLSTM model has the best long-term prediction results at different sites, and the R2 value is 0.871, and the RMSE is 9.28 μg/m3 at T+12 moment in the 1002A site. The new model effectively captures a significant portion of the long-term concentration trends characteristically and has been proven to be an effective tool for long-term predictions. This is attributed to the robust noise resistance of the AE method and the appropriate parameter setting of the CEEMD method.

Hierarchical graph augmented stacked autoencoders for multi-view representation learning

With recent success of deep neural networks, stacked autoencoder networks have received a lot of attention for robust unsupervised representation learning. However, recent autoencoder methods cannot make full use of multi-view information and thus fail to further improve many real-world applications by exploring the geometric structures of multi-view data. To address the above-mentioned issue, we introduce hierarchical graph augmented stacked autoencoders (HGSAE) for unsupervised multi-view representation learning. Specifically, a hierarchical graph structure is first adapted to stacked autoencoders to learn view-specific representations, aiming to preserve the geometric information of multi-view data through local and non-local graph regularizations. A general or common representation can then be learned by reconstructing each single view using fully connected neural networks. By doing this, the proposed method not only preserves the geometric information in multi-view data but also automatically balances the complementarity/consistency among different views. Extensive experiments on six popular unsupervised representation learning datasets demonstrate the effectiveness of our method when compared with recent state-of-the-art autoencoder methods.

Reconstructing echoes completely submerged in background noise by a stacked denoising autoencoder method for low-power EMAT testing

Low-power electromagnetic-acoustic transducer (EMAT) is crucially important for safety-critical equipment in industry, especially for potential explosives and inflammable petrochemical equipment and facilities. When the excitation power is very low, the corresponding echoes are overwhelmed in noise and related measurement would be inaccurate. To solve this problem, this paper presents a new echo reconstruction method based on a deep stacked denoising autoencoder (DSDAE) for nondestructive evaluation. First, the uses of reference signals and new data structure are to improve the training efficiency. A hybrid method based on variational mode decomposition and wavelet transform is used to obtain clean reference signals as inputs of the deep network. Then, the modified network structure and loss function aim to improve the ability of feature extraction and reconstruct clean echoes from low-power EMAT signals. To validate the effectiveness of the proposed method, the experiments of self-excitation and receiving A-scan inspections of stepped specimens with different thicknesses are conducted at some excitation voltages, as low as 25 V. The results indicate that the proposed DSDAE shows better and more stable denoising performance than some popular processing methods for different specimens and excitation voltages. It greatly improves the signal-to-noise ratio of the reconstructed signal to 20 dB. When applying to thickness measurement of specimens, its relative error is lower than 0.3%, which provides a practical and accurate tool for low-power EMAT testing.

Weighted target feature regression neural networks based soft sensing for industrial process

AbstractThere are often nonlinear and time‐varying characteristics in industrial processes. These characteristics cause difficulty in measuring product quality online. To address these issues, a weighted target feature regression neural network (WTFAER) was proposed for soft sensor modelling in this paper. The Pearson correlation coefficient was calculated to assign corresponding weights to process variables and design a weighted objective function. A target feature regression network (TFAER) was constructed using target correlation autoencoder with fully connected layer. After that, the weighted reconstructed information was applied to the TFAER model to extract deep quality‐related features and realize feature reuse. A deep network was formed by layered stacking to fully exploit the deep features for quality prediction. To make the proposed method domain adaptive, a maximum mean squared deviation (MMD) based regularization term was introduced in the loss function. Through the simulation experiments of debutanizer column and industrial polyethylene process, and compared with stacked autoencoder (SAE), variable‐wise weighted stacked autoencoder (VW‐SAE) and stacked target‐related autoencoder (STAE) methods, the effectiveness and generalization performance of the proposed modelling method were verified.