Conventional Autoencoder Research Articles

Graph-Variational Convolutional Autoencoder-Based Fault Detection and Diagnosis for Photovoltaic Arrays

Solar energy is a critical renewable energy source, with solar arrays or photovoltaic systems widely used to convert solar energy into electrical energy. However, solar array systems can develop faults and may exhibit poor performance. Diagnosing and resolving faults within these systems promptly is crucial to ensure reliability and efficiency in energy generation. Autoencoders and their variants have gained popularity in recent studies for detecting and diagnosing faults in solar arrays. However, traditional autoencoder models often struggle to capture the spatial and temporal relationships present in photovoltaic sensor data. This paper introduces a deep learning model that combines a graph convolutional network with a variational autoencoder to diagnose faults in solar arrays. The graph convolutional network effectively learns from spatial and temporal sensor data, significantly improving fault detection performance. We evaluated the proposed deep learning model on a recently published solar array dataset for an integrated power probability table mode. The experimental results show that the model achieves a fault detection rate exceeding 95% and outperforms the conventional autoencoder models. We also identified faulty components by analyzing the model’s reconstruction error for each feature, and we validated the analysis through the Kolmogorov–Smirnov test and noise injection techniques.

Enhancing performance of end-to-end communication system using Attention Mechanism-based Sparse Autoencoder over Rayleigh fading channel

Deep learning has revolutionized communication systems by introducing innovative approaches to address channel impairments through end-to-end models. Autoencoders, a type of deep learning architecture, are adept at learning compact data representations. However, conventional autoencoders in end-to-end models can suffer from overfitting, which limits their effectiveness in noisy communication environments. To address this issue, we propose a Sparse Autoencoder-based (SAE) model that enforces sparsity and promotes the extraction of robust features. Despite its effectiveness, the SAE model may still lack the ability to focus on the most relevant features of the input data. To overcome this limitation, we further introduce an Attention Mechanism-based Sparse Autoencoder (ASA) model. This model integrates the feature extraction capabilities of a sparse autoencoder with an attention mechanism that selectively highlights informative features of the signal. Through simulations, we demonstrate that both proposed models significantly improve M-PSK and M-QAM communication system performance. When trained at 7 dB, both proposed models exhibit significant performance improvements at higher testing average SNRs. Our results show that the SAE model outperforms the conventional Maximum Likelihood Detection (MLD) model and baseline autoencoder systems but suffers from error floor issues. The SAE model suffers from an error floor at average SNRs beyond 16 dB for BPSK and 14 dB for higher-order modulation schemes. As the value of M increases, the performance gap between the MLD and the proposed SAE model narrows. The ASA model, however, effectively mitigates the error floor observed in the SAE model for all values of M and across all modulation schemes. This research highlights the benefits of integrating an attention mechanism with SAE, resulting in enhanced robustness and reliability in communication systems characterized by improved accuracy and reduced error rates.

Temporal denoising and deep feature learning for enhanced defect detection in thermography using stacked denoising convolution autoencoder

Thermal wave imaging uses the temporal temperature distribution over the object’s surface for subsurface analysis. However, the noise generated during experimentation corrupts this temporal history and hampers the detection of defect signatures. As denoising of the temporal thermal history enhances the defect detectability, this study offers a stacked denoising convolution autoencoder (SDCAE) in frequency-modulated thermal wave imaging with one-dimensional convolution layers to reduce noise in temporal thermal evolution and train high-level features resulting in improved defect signs. Experimental results on mild steel and carbon fiber reinforced polymer specimens with different sizes of defects at various depths demonstrate that integrating temporal denoising and deep feature learning techniques into a single novel framework significantly improved defect detectability. In addition, defect signal-to-noise ratios of the denoised thermal data and latent space of the proposed model compared to conventional autoencoder and dimensionality reduction techniques recommend the superiority of the proposed method.

Multi-scale feature reconstruction network for industrial anomaly detection

Unsupervised anomaly detection techniques, which operate without prior knowledge of anomalies, have garnered significant attention in industrial inspection due to their adaptability and generalization. Therefore, knowledge-based computer vision techniques have been broadly applied to identify unusual image patterns. However, real-time industrial applications present challenges such as limited anomalous samples, inadequate defect knowledge, and complex background textures. These factors lead to difficulties in accurately identifying defect regions, and conventional auto-encoder networks often struggle to overcome these issues.To address these limitations, we propose a multi-scale feature reconstruction (MSFR) network specifically designed for domain shift scenarios. Our approach employs a pyramidal vision transformer network (PVTN) to reconstruct multi-scale feature maps, capturing discriminative features at various scales. Additionally, a pre-trained module extracts multi-level features at the same scale, and a dedicated feature matching module enhances accuracy by improving the alignment probability between features. The MSFR strategy surpasses conventional auto-encoders by filtering pixel-level information at multiple depths. Empirical evaluations were conducted using benchmark datasets such as MVTec AD and AeBAD-S. Furthermore, an extensive ablation study demonstrates the effectiveness and viability of the proposed MSFR approach for industrial anomaly detection tasks. The experimental results show that the proposed model significantly outperforms recent approaches, making it highly suitable for real-world industrial applications, particularly in manufacturing.

Asymmetric double-winged multi-view clustering network for exploring diverse and consistent information

In unsupervised scenarios, deep contrastive multi-view clustering (DCMVC) is becoming a hot research spot, which aims to mine the potential relationships between different views. Most existing DCMVC algorithms focus on exploring the consistency information for the deep semantic features, while ignoring the diverse information on shallow features. To fill this gap, we propose a novel multi-view clustering network termed CodingNet to explore the diverse and consistent information simultaneously in this paper. Specifically, instead of utilizing the conventional auto-encoder, we design an asymmetric structure network to extract shallow and deep features separately. Then, by approximating the similarity matrix on the shallow feature to the zero matrix, we ensure the diversity for the shallow features, thus offering a better description of multi-view data. Moreover, we propose a dual contrastive mechanism that maintains consistency for deep features at both view-feature and pseudo-label levels. Our framework’s efficacy is validated through extensive experiments on six widely used benchmark datasets, outperforming most state-of-the-art multi-view clustering algorithms.

Multi-Branch Network for Color Image Denoising Using Dilated Convolution and Attention Mechanisms.

Image denoising is regarded as an ill-posed problem in computer vision tasks that removes additive noise from imaging sensors. Recently, several convolution neural network-based image-denoising methods have achieved remarkable advances. However, it is difficult for a simple denoising network to recover aesthetically pleasing images owing to the complexity of image content. Therefore, this study proposes a multi-branch network to improve the performance of the denoising method. First, the proposed network is designed based on a conventional autoencoder to learn multi-level contextual features from input images. Subsequently, we integrate two modules into the network, including the Pyramid Context Module (PCM) and the Residual Bottleneck Attention Module (RBAM), to extract salient information for the training process. More specifically, PCM is applied at the beginning of the network to enlarge the receptive field and successfully address the loss of global information using dilated convolution. Meanwhile, RBAM is inserted into the middle of the encoder and decoder to eliminate degraded features and reduce undesired artifacts. Finally, extensive experimental results prove the superiority of the proposed method over state-of-the-art deep-learning methods in terms of objective and subjective performances.

Enhancing Yarn Quality Wavelength Spectrogram Analysis: A Semi-Supervised Anomaly Detection Approach with Convolutional Autoencoder

Abnormal detection plays a pivotal role in the routine maintenance of industrial equipment. Malfunctions or breakdowns in the drafting components of spinning equipment can lead to yarn defects, thereby compromising the overall quality of the production line. Fault diagnosis of spinning equipment entails the examination of component defects through Wavelet Spectrogram Analysis (WSA). Conventional detection techniques heavily rely on manual experience and lack generality. To address this limitation, this current study leverages machine learning technology to formulate a semi-supervised anomaly detection approach employing a convolutional autoencoder. This method trains deep neural networks with normal data and employs the reconstruction mode of a convolutional autoencoder in conjunction with Kernel Density Estimation (KDE) to determine the optimal threshold for anomaly detection. This facilitates the differentiation between normal and abnormal operational modes without the necessity for extensive labeled fault data. Experimental results from two sets of industrial data validate the robustness of the proposed methodology. In comparison to conventional Autoencoder and prevalent machine learning techniques, the proposed approach demonstrates superior performance across evaluation metrics such as Accuracy, Recall, Area Under the Curve (AUC), and F1-score, thereby affirming the feasibility of the suggested model.

Deep-Representation-Learning-Based Classification Strategy for Anticancer Peptides

Cancer, with its complexity and numerous origins, continues to provide a huge challenge in medical research. Anticancer peptides are a potential treatment option, but identifying and synthesizing them on a large scale requires accurate prediction algorithms. This study presents an intuitive classification strategy, named ACP-LSE, based on representation learning, specifically, a deep latent-space encoding scheme. ACP-LSE can demonstrate notable advancements in classification outcomes, particularly in scenarios with limited sample sizes and abundant features. ACP-LSE differs from typical black-box approaches by focusing on representation learning. Utilizing an auto-encoder-inspired network, it embeds high-dimensional features, such as the composition of g-spaced amino acid pairs, into a compressed latent space. In contrast to conventional auto-encoders, ACP-LSE ensures that the learned feature set is both small and effective for classification, giving a transparent alternative. The suggested approach is tested on benchmark datasets and demonstrates higher performance compared to the current methods. The results indicate improved Matthew’s correlation coefficient and balanced accuracy, offering insights into crucial aspects for developing new ACPs. The implementation of the proposed ACP-LSE approach is accessible online, providing a valuable and reproducible resource for researchers in the field.

Robust reduced-order machine learning modeling of high-dimensional nonlinear processes using noisy data

Autoencoder-based reduced-order machine learning models have been developed for modeling and predictive control of nonlinear chemical processes with high dimensionality such as discretization of reaction–diffusion processes. However, in the presence of data noise, autoencoders may over-fit the training data and subsequently learn an inaccurate low-dimensional representation of the process variables. This leads to an inaccurate prediction model when the models are integrated with model predictive control (MPC). To address this issue, this work develops a novel machine-learning-based reduced-order modeling method by integrating SpectralDense layers into autoencoders and incorporating them with recurrent neural networks. We demonstrate that the new architecture of autoencoders using SpectralDense layers is more robust against over-fitting than conventional autoencoders in the presence of data noise, which improves the prediction accuracy in MPC. A diffusion–reaction process simulation example is used to demonstrate that the robust autoencoders outperform those using conventional layers for reduced-order modeling in predictive control.

Orientation-invariant autoencoders learn robust representations for shape profiling of cells and organelles

Cell and organelle shape are driven by diverse genetic and environmental factors and thus accurate quantification of cellular morphology is essential to experimental cell biology. Autoencoders are a popular tool for unsupervised biological image analysis because they learn a low-dimensional representation that maps images to feature vectors to generate a semantically meaningful embedding space of morphological variation. The learned feature vectors can also be used for clustering, dimensionality reduction, outlier detection, and supervised learning problems. Shape properties do not change with orientation, and thus we argue that representation learning methods should encode this orientation invariance. We show that conventional autoencoders are sensitive to orientation, which can lead to suboptimal performance on downstream tasks. To address this, we develop O2-variational autoencoder (O2-VAE), an unsupervised method that learns robust, orientation-invariant representations. We use O2-VAE to discover morphology subgroups in segmented cells and mitochondria, detect outlier cells, and rapidly characterise cellular shape and texture in large datasets, including in a newly generated synthetic benchmark.

Autoencoders and their applications in machine learning: a survey

Autoencoders have become a hot researched topic in unsupervised learning due to their ability to learn data features and act as a dimensionality reduction method. With rapid evolution of autoencoder methods, there has yet to be a complete study that provides a full autoencoders roadmap for both stimulating technical improvements and orienting research newbies to autoencoders. In this paper, we present a comprehensive survey of autoencoders, starting with an explanation of the principle of conventional autoencoder and their primary development process. We then provide a taxonomy of autoencoders based on their structures and principles and thoroughly analyze and discuss the related models. Furthermore, we review the applications of autoencoders in various fields, including machine vision, natural language processing, complex network, recommender system, speech process, anomaly detection, and others. Lastly, we summarize the limitations of current autoencoder algorithms and discuss the future directions of the field.

Anomalous Change Detection in Drilling Process Using Variational Autoencoder with Temperature Near Drill Edge

The different flexibility and diversity requirements for respective manufacturing units have made modern cutting tool management much more crucial and complicated, as a greater variety of tools and more frequent tool changes are required to enhance production efficiency and avoid unplanned manufacturing downtime. Developing in-process anomalous change detection methods has been identified as an essential challenge. Machine learning techniques have been widely applied in tool condition monitoring and anomalous change detection. As anomaly data is rare in manufacturing processes, supervised machine learning approaches (such as regression and classification) are not applied to the anomalous change detection problem. Rather, self-supervised machine learning (a representative type of unsupervised machine learning) is applied. This study describes a variational autoencoder (VAE) neural network and proposes a VAE-based method for tool condition monitoring and change detection in a drilling process using the temperature near a drill edge. The proposed VAE evaluates the drill tool condition based on the reconstruction error between the input temperature and its estimate per a drill unit process through the trained network. Computational simulations demonstrate that the proposed VAE network model can avoid overfitting to the anomaly data and that its expressive power is greater than that of the conventional autoencoder model.

HAT-GAE: Self-supervised graph autoencoders with hierarchical adaptive masking and trainable corruption

In recent years, self-supervised autoencoders have developed into a powerful representation learning framework in computer vision and natural language processing. However, their application to graph data has resulted in limited performance due to the non-Euclidean and complex structure of graphs compared to images or text, as well as the limitations of conventional autoencoder architectures. In this paper, we investigate factors impacting the performance of autoencoders on graph data and propose a novel autoencoder model for graph representation learning. Our model incorporates a hierarchical adaptive masking mechanism to incrementally increase the difficulty of training to mimic the process of human cognitive learning and a trainable corruption scheme to enhance the robustness of learned representations. Through extensive experimentation on ten benchmark datasets, we demonstrate the superiority of our proposed method over state-of-the-art graph representation learning models.

ECG Signal Denoising Method Based on Disentangled Autoencoder

The electrocardiogram (ECG) is widely used in medicine because it can provide basic information about different types of heart disease. However, ECG data are usually disturbed by various types of noise, which can lead to errors in diagnosis by doctors. To address this problem, this study proposes a method for denoising ECG based on disentangled autoencoders. A disentangled autoencoder is an improved autoencoder suitable for denoising ECG data. In our proposed method, we use a disentangled autoencoder model based on a fully convolutional neural network to effectively separate the clean ECG data from the noise. Unlike conventional autoencoders, we disentangle the features of the coding hidden layer to separate the signal-coding features from the noise-coding features. We performed simulation experiments on the MIT-BIH Arrhythmia Database and found that the algorithm had better noise reduction results when dealing with four different types of noise. In particular, using our method, the average improved signal-to-noise ratios for the three noises in the MIT-BIH Noise Stress Test Database were 27.45 db for baseline wander, 25.72 db for muscle artefacts, and 29.91 db for electrode motion artefacts. Compared to a denoising autoencoder based on a fully convolutional neural network (FCN), the signal-to-noise ratio was improved by an average of 12.57%. We can conclude that the model has scientific validity. At the same time, our noise reduction method can effectively remove noise while preserving the important information conveyed by the original signal.

Auto-Encoder Based Orthogonal Time Frequency Space Modulation and Detection With Meta-Learning

To tackle a Doppler sensitivity problem of orthogonal frequency division multiplexing (OFDM), orthogonal time frequency space (OTFS) has been investigated, where information is carried over delay-Doppler domain. In this paper, to improve communication reliability in doubly dispersive channel, an auto-encoder (AE)-based OTFS modulation and detection scheme is developed, where the transmit OTFS waveform and its associated detection scheme at the receiver are jointly optimized in a deep learning framework. However, the conventional AE architecture which takes one-hot encoded input vector is hard to be reused in OTFS due to its enormous input dimensionality that increases exponentially on the number of grid points in delay-Doppler domain. To overcome it, we divide the delay-Doppler grid into multiple subblocks and associate the one-hot encoded vector with each subblock. Then, by concatenating them, one multi-hot vector is formed and exploited as the input vector for the proposed AE-based OTFS modulation and detection. We also develop a meta-learning scheme to effectively train the AE-based OTFS transceiver for newly updated channel profile.

Self-supervised Discriminative Representation Learning by Fuzzy Autoencoder

Representation learning based on autoencoders has received great concern for its potential ability to capture valuable latent information. Conventional autoencoders pursue minimal reconstruction error, but in most machine learning tasks such as classification and clustering, the discrimination of feature representation is also important. To address this limitation, an enhanced self-supervised discriminative fuzzy autoencoder (FAE) is innovatively proposed, which focuses on exploring information within data to guide the unsupervised training process and enhancing feature discrimination in a self-supervised manner. In FAE, fuzzy membership is applied to provide a means of self-supervised, which allows FAE can not only utilize AE’s outstanding representation learning capabilities but can also transform the original data into another space with improved discrimination. First, the objective function corresponding to FAE is proposed by reconstruction loss and clustering oriented loss simultaneously. Subsequently, Mini-Batch Gradient Descent is applied to infer the objective function and the detailed process is illustrated step by step. Finally, empirical studies on clustering tasks have demonstrated the superiority of FAE over the state of the art.

Dimension Reduction Technique Based on Supervised Autoencoder for Intrusion Detection of Industrial Control Systems

Industrial control systems (ICSs) are closely related to human life. In recent years, many ICSs have been connected to the Internet rather than being physically isolated, which has improved business efficiency while also increasing the risks of being attacked. The security issues of ICSs have gotten a lot of interest in the research community because attack events that aim at ICSs can cause catastrophic damage. An intrusion detection system (IDS) serves as an important tool for providing protection. Many IDS studies using machine learning and deep learning have been proposed. However, high-dimensional data may cause overfitting, resulting in inferior performance. To improve the classification performance, we suggest a dimension reduction technique based on the supervised autoencoder (SupervisedAE) and principal components analysis (PCA) in this study. To obtain more discriminative latent representations, compared with the conventional autoencoder, the SupervisedAE absorbs the label information during the training process. In this way, the improved autoencoder model is trained with reconstruction error and classification error simultaneously. Based on the latent representations extracted from the SupervisedAE, we add the PCA algorithm. The additional PCA algorithm reduces the dimension of features further. We conduct a series of experiments utilizing the suggested technique on a public power system data set to evaluate the performance. Compared with various dimension reduction methods, including autoencoder variants, the technique proposed in this study shows higher performance. In the meanwhile, it outperforms some existing detection methods in terms of accuracy and F1 score.

Feature autoencoder for detecting adversarial examples

Deep neural networks (DNNs) have gained widespread adoption in computer vision. Unfortunately, state-of-the-art DNNs are vulnerable to adversarial example (AE) attacks, where an adversary introduces imperceptible perturbations to a test example for defrauding DNNs. The obstacles have urged intensive research on improving the DNN robustness via adversarial training, that is, the clean data set is blended with adversarial examples to carry out training. However, the adversarial example attack technologies are open-ended, and the adversarial training is insufficient to focus on improving robustness performance. To circumvent this limitation, we mitigate adversarial example attacks from another perspective, which aims at detecting adversarial examples. Feature autoencoder detector (FADetector), a novel defense framework that exploits feature knowledge is proposed. One of the hallmarks of FADetector is to not involve adversarial examples to train the detector. Our extensive evaluation on MNIST and CIFAR-10 data sets demonstrates that our defense outperforms the conventional autoencoder detectors in terms of detection accuracy.

Dimension reduced turbulent flow data from deep vector quantisers

Analysing large-scale data from simulations of turbulent flows is memory intensive, requiring significant resources. This major challenge highlights the need for data compression techniques. In this study, we apply a physics-informed Deep Learning technique based on vector quantisation to generate a discrete, low-dimensional representation of data from simulations of three-dimensional turbulent flows. The deep learning framework is composed of convolutional layers and incorporates physical constraints on the flow, such as preserving incompressibility and global statistical characteristics of the velocity gradients. The accuracy of the model is assessed using statistical, comparison-based similarity and physics-based metrics. The training data set is produced from Direct Numerical Simulation of an incompressible, statistically stationary, isotropic turbulent flow. The performance of this lossy data compression scheme is evaluated not only with unseen data from the stationary, isotropic turbulent flow, but also with data from decaying isotropic turbulence, a Taylor–Green vortex flow, and a turbulent channel flow. Defining the compression ratio (CR) as the ratio of original data size to the compressed one, the results show that our model based on vector quantisation can offer CR with a mean square error (MSE) of , and predictions that faithfully reproduce the statistics of the flow, except at the very smallest scales where there is some loss. Compared to the recent study of Glaws. et al. [Deep learning for in situ data compression of large turbulent flow simulations. Phys Rev Fluids. 2020;5(11):114602], which was based on a conventional autoencoder (where compression is performed in a continuous space), our model improves the CR by more than 30%, and reduces the MSE by an order of magnitude. Our compression model is an attractive solution for situations where fast, high quality and low-overhead encoding and decoding of large data are required.

Explaining classifiers by constructing familiar concepts

Interpreting a large number of neurons in deep learning is difficult. Our proposed ‘CLAssifier-DECoder’ architecture (ClaDec) facilitates the understanding of the output of an arbitrary layer of neurons or subsets thereof. It uses a decoder that transforms the incomprehensible representation of the given neurons to a representation that is more similar to the domain a human is familiar with. In an image recognition problem, one can recognize what information (or concepts) a layer maintains by contrasting reconstructed images of ClaDec with those of a conventional auto-encoder(AE) serving as reference. An extension of ClaDec allows trading comprehensibility and fidelity. We evaluate our approach for image classification using convolutional neural networks. We show that reconstructed visualizations using encodings from a classifier capture more relevant classification information than conventional AEs. This holds although AEs contain more information on the original input. Our user study highlights that even non-experts can identify a diverse set of concepts contained in images that are relevant (or irrelevant) for the classifier. We also compare against saliency based methods that focus on pixel relevance rather than concepts. We show that ClaDec tends to highlight more relevant input areas to classification though outcomes depend on classifier architecture. Code is at https://github.com/JohnTailor/ClaDec