Autoencoder Architecture Research Articles

β-Variational autoencoders and transformers for reduced-order modelling of fluid flows

Variational autoencoder architectures have the potential to develop reduced-order models for chaotic fluid flows. We propose a method for learning compact and near-orthogonal reduced-order models using a combination of a β-variational autoencoder and a transformer, tested on numerical data from a two-dimensional viscous flow in both periodic and chaotic regimes. The β-variational autoencoder is trained to learn a compact latent representation of the flow velocity, and the transformer is trained to predict the temporal dynamics in latent-space. Using the β-variational autoencoder to learn disentangled representations in latent-space, we obtain a more interpretable flow model with features that resemble those observed in the proper orthogonal decomposition, but with a more efficient representation. Using Poincaré maps, the results show that our method can capture the underlying dynamics of the flow outperforming other prediction models. The proposed method has potential applications in other fields such as weather forecasting, structural dynamics or biomedical engineering.

Intelligent methods for natural data analysis: application to space weather

The paper describes methods for detecting anomalies in geophysical monitoring data. This work studies a highly relevant class of problems in this area, aimed at creating methods for space weather forecasting. The negative impact of space weather anomalies on human health and practically all modern infrastructure objects requires the development of methods and the creation of effective means of detecting anomalies. Threshold wavelet filtering methods widely used for data analysis and anomaly detection allow one to obtain fairly accurate estimates using a “greed” strategy, even in the case of incomplete noise data. With this approach, the signal is estimated by isolating coherent structures. But these methods have high computational complexity, failing to provide accurate estimates when the signal-to-noise ratio is low. For such signals, we propose using adaptive probabilistic thresholds. Threshold function parameters are introduced that make it possible to estimate the variability of the process, suppress noise, and detect nonstationary features of different time-frequency structures. The paper also considers ways to combine threshold wavelet filtering with neural networks of the NARX and Autoencoder architectures. Schemes for the implementation of such approaches in the problems of detecting space weather anomalies are proposed. Using the problem of detecting ionospheric anomalies as an example, the efficiency of combining threshold wavelet filtering with the NARX network is shown. The efficiency of network sharing an Autoencoder with adaptive wavelet threshold filtering is shown in the problem of anomaly detection in cosmic ray flux intensity data.

Enhancing Cardiac Anomaly Detection through Deep Learning Autoencoder: An In-Depth Analysis Using the PTB Diagnostic ECG Database

Cardiovascular diseases are the leading cause of mortality worldwide, necessitating advancements in early anomaly detection from electrocardiogram (ECG) signals. This study introduces a novel convolutional neural network (CNN)-based autoencoder architecture that significantly outperforms traditional Multi-Layer Perceptron (MLP) models in detecting ECG anomalies. Our method capitalizes on unsupervised learning to discern between normal and pathological heartbeats with an accuracy of 71.16% and an F1 score of 73%. We address the challenge of imbalanced datasets by implementing a refined thresholding strategy for anomaly classification. Comparative analysis reveals that our model achieves superior precision, particularly in delineating true anomalies within ECG data. The proposed autoencoder architecture holds promise for clinical applications, offering a robust tool for enhancing diagnostic accuracy in cardiac care. Our research contributes to the growing body of knowledge in medical diagnostics, paving the way for improved patient outcomes through advanced deep learning techniques.

Detecting communities with multiple topics in attributed networks via self-supervised adaptive graph convolutional network

Attributed networks which have topology information and attribute information simultaneously are very common in the real world, e.g., online social networks and co-authorship networks. Community detection in attribute networks is a very valuable research topic. Although there is already a lot of work focused on it, but most of existing approaches still face two key challenges: topology information and attribute information fusion, and multiple topics community detection. To effectively address these challenges, in this paper we devise an approach called SSAGCN, which adopts the self-supervised learning paradigm using the autoencoder architecture. Specifically, SSAGCN comprises three main parts: adaptive graph convolutional network (AGCN) encoder, modularity maximization and dual decoder. AGCN uses two GCNs with shared parameters and the attention mechanism to fuse topology information and attribute information automatically. To drive AGCN encoder to uncover community structure, we select the modularity maximization as the optimization objective. The dual decoder is applied to reconstruct both topology structure and network attributes. By introducing the joint training strategy, SSAGCN is able to discover multiple topics community through the end-to-end manner. Extensive experiments are conducted on nine benchmark attributed networks, and the results illustrate not only the superiority of SSAGCN over state-of-the-art approaches, but also its good ability of community-topic analysis. For the reproducibility, we release the source code at https://github.com/GDM-SCNU/SSAGCN.

Deep learning based asymmetrical autoencoder for PAPR reduction of CP-OFDM systems

Adoption of Orthogonal Frequency Division Multiplexing (OFDM) in 5th Generation New Radio (5G NR) as a multicarrier modulation technique allows high data rate transmission with lower complexity. However, problem such as peak to average power ratio (PAPR) has decreased the transmitter efficiency. Recently, several attempts have been made to reduce the high PAPR in OFDM utilizing deep learning (DL) based on autoencoder architecture. However, the proposed autoencoder using symmetrical autoencoder (SAE) is followed by high computational complexity at both transmitter and receiver as well as BER performance degradation. Since 5G NR focuses on massive 5G internet of things eco-system, a flexible carrier spacing is used to support diverse spectrum bands which is afterward named as Cyclic Prefix OFDM (CP-OFDM). In this study, we aimed to contribute to this growing area of research by exploring the potential of our proposed asymmetrical autoencoder (AAE) to reduce high PAPR in the CP-OFDM system. Four AAE models have been developed in this study and the performance of the models were evaluated based on comprehensive conditions such as data training at different corruption levels, cyclic prefix length, upsampling factors and loss function levels. The investigation of AAE in 5G CP-OFDM system has shown superior performance using a 5×1 AAE model that can reduce a substantial amount of PAPR, BER degradation and computational complexity compared to conventional SAE. This study lays the groundwork for future research into the asymmetrical approach of autoencoder especially in 5G and beyond networks.

Deep Anomaly Detection with Attention (DADA): A Novel Approach for Identifying Multipath Interference in Radar Signals

Multipath interference in radar signals caused by sea, ground, and other environments poses significant challenges to the target detection, tracking, and classification capabilities of radar systems. Existing methods for radar signal identification require labeled samples and focus mainly on the classification of normal signals. However, in practice, anomalous samples (multipath interference signals) may be scarce and highly imbalanced (i.e., mostly normal samples). To address this problem, we propose a deep anomaly detection with attention (DADA) for semisupervised detection of multipath radar signals. The method transforms radar signals into time–frequency images and is trained exclusively on normal samples. The autoencoder architecture is extended with a feature extractor network to capture latent sample features. CBAM attention is introduced to improve feature extraction. By learning the distribution of normal samples in high-dimensional image space and low-dimensional feature space, a two-dimensional feature space representing normal samples is constructed. A one-class SVM then learns the boundary of normal samples for anomaly detection. Extensive experiments on radar signal datasets validate the effectiveness of the proposed approach.

Multiple Kernel Synthesis of Head CT Using a Task-Based Loss Function

In CT imaging of the head, multiple image series are routinely reconstructed with different kernels and slice thicknesses. Reviewing the redundant information is an inefficient process for radiologists. We address this issue with a convolutional neural network (CNN)-based technique, synthesiZed Improved Resolution and Concurrent nOise reductioN (ZIRCON), that creates a single, thin, low-noise series that combines the favorable features from smooth and sharp head kernels. ZIRCON uses a CNN model with an autoencoder U-Net architecture that accepts two input channels (smooth- and sharp-kernel CT images) and combines their salient features to produce a single CT image. Image quality requirements are built into a task-based loss function with a smooth and sharp loss terms specific to anatomical regions. The model is trained using supervised learning with paired routine-dose clinical non-contrast head CT images as training targets and simulated low-dose (25%) images as training inputs. One hundred unique de-identified clinical exams were used for training, ten for validation, and ten for testing. Visual comparisons and contrast measurements of ZIRCON revealed that thinner slices and the smooth-kernel loss function improved gray-white matter contrast. Combined with lower noise, this increased visibility of small soft-tissue features that would be otherwise impaired by partial volume averaging or noise. Line profile analysis showed that ZIRCON images largely retained sharpness compared to the sharp-kernel input images. ZIRCON combined desirable image quality properties of both smooth and sharp input kernels into a single, thin, low-noise series suitable for both brain and skull imaging.

A graph convolutional autoencoder approach to model order reduction for parametrized PDEs

The present work proposes a framework for nonlinear model order reduction based on a Graph Convolutional Autoencoder (GCA-ROM). In the reduced order modeling (ROM) context, one is interested in obtaining real-time and many-query evaluations of parametric Partial Differential Equations (PDEs). Linear techniques such as Proper Orthogonal Decomposition (POD) and Greedy algorithms have been analyzed thoroughly, but they are more suitable when dealing with linear and affine models showing a fast decay of the Kolmogorov n-width. On one hand, the autoencoder architecture represents a nonlinear generalization of the POD compression procedure, allowing one to encode the main information in a latent set of variables while extracting their main features. On the other hand, Graph Neural Networks (GNNs) constitute a natural framework for studying PDE solutions defined on unstructured meshes. Here, we develop a non-intrusive and data-driven nonlinear reduction approach, exploiting GNNs to encode the reduced manifold and enable fast evaluations of parametrized PDEs. We show the capabilities of the methodology for several models: linear/nonlinear and scalar/vector problems with fast/slow decay in the physically and geometrically parametrized setting. The main properties of our approach consist of (i) high generalizability in the low-data regime even for complex behaviors, (ii) physical compliance with general unstructured grids, and (iii) exploitation of pooling and un-pooling operations to learn from scattered data.

Improved limited view ultrasound tomography via machine learning.

Tomographic reconstruction is used extensively in medicine, non-destructive testing and geology. In an ideal situation where measurements are taken at all angles around an object, known as full view configuration, a full reconstruction of the object can be produced. One of the major issues faced in tomographic imaging is when measurements cannot be taken freely around the object under inspection. This may be caused by the size and geometry of the object or difficulty accessing from particular directions. The resulting limited view transducer configuration leads to a large deterioration in image quality, thus it is very beneficial to employ a compensation algorithm. At present, the most effective compensation algorithms require a large amount of computing power or a bespoke case-by case approach, often with numerous arbitrary constants which must be tuned for a specific application. This work proposes a machine learning based approach to perform the limited view compensation. The model is based around an autoencoder architecture. It is trained on an artificial dataset, taking advantage of the ability to generate arbitrary limited view images given a full view input. The approach is evaluated on ten laser-scanned corrosion maps and the results compared to positivity regularisation - a limited view compensation algorithm similar in the speed of execution and generalisation potential. The algorithms are compared for root mean squared error (RMSE) across the image, and maximum absolute error (MAE). Furthermore, they are visually compared for subjective quality. Compared to the conventional algorithm, the ML-based approach improves on the MAE in eight out of the ten cases. The conventional approach performs better on mean RMSE, which is explained by the ML outputting inaccurate background level, which is not a critical ability. Most importantly, the visual inspection of outputs shows the ML approach reconstructs the images better, especially in the case of irregular corrosion patches. Compared to limited view images, the ML method improves both the RMSE and MAE by 41%.

DSANet: Dynamic and Structure-Aware GCN for Sparse and Incomplete Point Cloud Learning.

Learning 3-D structures from incomplete point clouds with extreme sparsity and random distributions is a challenge since it is difficult to infer topological connectivity and structural details from fragmentary representations. Missing large portions of informative structures further aggravates this problem. To overcome this, a novel graph convolutional network (GCN) called dynamic and structure-aware NETwork (DSANet) is presented in this article. This framework is formulated based on a pyramidic auto-encoder (AE) architecture to address accurate structure reconstruction on the sparse and incomplete point clouds. A PointNet-like neural network is applied as the encoder to efficiently aggregate the global representations of coarse point clouds. On the decoder side, we design a dynamic graph learning module with a structure-aware attention (SAA) to take advantage of the topology relationships maintained in the dynamic latent graph. Relying on gradually unfolding the extracted representation into a sequence of graphs, DSANet is able to reconstruct complicated point clouds with rich and descriptive details. To associate analogous structure awareness with semantic estimation, we further propose a mechanism, called structure similarity assessment (SSA). This method allows our model to surmise semantic homogeneity in an unsupervised manner. Finally, we optimize the proposed model by minimizing a new distortion-aware objective end-to-end. Extensive qualitative and quantitative experiments demonstrate the impressive performance of our model in reconstructing unbroken 3-D shapes from deficient point clouds and preserving semantic relationships among different regional structures.

CORE: CORrelation-Guided Feature Enhancement for Few-Shot Image Classification.

Few-shot classification aims to adapt classifiers trained on base classes to novel classes with a few shots. However, the limited amount of training data is often inadequate to represent the intraclass variations in novel classes. This can result in biased estimation of the feature distribution, which in turn results in inaccurate decision boundaries, especially when the support data are outliers. To address this issue, we propose a feature enhancement method called CORrelation-guided feature Enrichment that generates improved features for novel classes using weak supervision from the base classes. The proposed CORrelation-guided feature Enhancement (CORE) method utilizes an autoencoder (AE) architecture but incorporates classification information into its latent space. This design allows the CORE to generate more discriminative features while discarding irrelevant content information. After being trained on base classes, CORE's generative ability can be transferred to novel classes that are similar to those in the base classes. By using these generative features, we can reduce the estimation bias of the class distribution, which makes few-shot learning (FSL) less sensitive to the selection of support data. Our method is generic and flexible and can be used with any feature extractor and classifier. It can be easily integrated into existing FSL approaches. Experiments with different backbones and classifiers show that our proposed method consistently outperforms existing methods on various widely used benchmarks.

An Unsupervised Approach to Wind Turbine Blade Icing Detection Based on Beta Variational Graph Attention Autoencoder

Supervised deep-learning methods are data-driven and widely used for wind turbine blade icing detection (ID). Data driven methods generally require a complete dictionary of labeled sensor data. However, labeling sensor data increases engineering costs and can introduce costly errors such as incorrect data labels. Additionally, the reported data-driven approaches ignore the contribution of structural properties of multivariate sensor data to failure patterns identification. To address these shortcomings, the current study proposes a Beta variational graph attention autoencoder (β-VGATAE) for blade ID. The β-VGATAE model employs a Beta variational autoencoder (β-VAE) architecture to achieve unsupervised learning. A graph attention network is used as a spatial feature extractor within the β-VAE architecture since it considers the spatial structure of the sensor data. Actual sensor data from supervisory control and data acquisition systems were used to validate the proposed model. Specifically, we verified the rationality of designing each component in the β-VGATAE. Experimental results show that the highest levels of accuracy achieved were 90.9% and 93.4% for the respective scenarios involving two wind turbines; the β-VGATAE detection model has high accuracy and excellent generalization ability.

LTScoder: Long-Term Time Series Forecasting Based on a Linear Autoencoder Architecture

LTScoder: Long-Term Time Series Forecasting Based on a Linear Autoencoder Architecture

RFI detection with spiking neural networks

Abstract Detecting and mitigating radio frequency interference (RFI) is critical for enabling and maximising the scientific output of radio telescopes. The emergence of machine learning (ML) methods capable of handling large datasets has led to their application in radio astronomy, particularly in RFI detection. Spiking neural networks (SNNs), inspired by biological systems, are well suited for processing spatio-temporal data. This study introduces the first exploratory application of SNNs to an astronomical data processing task, specifically RFI detection. We adapt the nearest latent neighbours (NLNs) algorithm and auto-encoder architecture proposed by previous authors to SNN execution by direct ANN2SNN conversion, enabling simplified downstream RFI detection by sampling the naturally varying latent space from the internal spiking neurons. Our subsequent evaluation aims to determine whether SNNs are viable for future RFI detection schemes. We evaluate detection performance with the simulated HERA telescope and hand-labelled LOFAR observation dataset the original authors provided. We additionally evaluate detection performance with a new MeerKAT-inspired simulation dataset that provides a technical challenge for machine-learnt RFI detection methods. This dataset focuses on satellite-based RFI, an increasingly important class of RFI and is an additional contribution. Our SNN approach remains competitive with the original NLN algorithm and AOFlagger in AUROC, AUPRC, and F1-scores for the HERA dataset but exhibits difficulty in the LOFAR and Tabascal datasets. However, our method maintains this accuracy while completely removing the compute and memory-intense latent sampling step found in NLN. This work demonstrates the viability of SNNs as a promising avenue for ML-based RFI detection in radio telescopes by establishing a minimal performance baseline on traditional and nascent satellite-based RFI sources and is the first work to our knowledge to apply SNNs in astronomy.

Zooming Into Clarity: Image Denoising Through Innovative Autoencoder Architectures

Zooming Into Clarity: Image Denoising Through Innovative Autoencoder Architectures

Employing Sequence-to-Sequence Stacked LSTM Autoencoder Architecture to Forecast Indian Weather

One of the buzz words in current research depends for a weather forecast is the atmospheric temperature, and this value may change at any time. Predicting air temperature exactly is necessary since it is particularly important for various communities and events. In today’s world of massive amounts of data and unforeseen data fluctuations, long-short term memory (LSTM), a sort of neural network method, is becoming more and more popular. The present work employed a sequence to sequence LSTM autoencoder to forecast Indian weather patterns. The prediction model was trained using historical data obtained from the Khordha district administration. The proposed method primarily emphasizes two phases of the procedure. The present study involves an examination of a proposed model, wherein a comparison is made between four distinct variants of LSTM techniques: vanilla LSTM, stacked LSTM, bidirectional LSTM, and convLSTM. The objective of this analysis is to predict temperature patterns. Additionally, this paper utilizes an LSTM autoencoder model to forecast the temperature for the subsequent two-year period in the Khordha district of Odisha, India. The experimental investigation demonstrates that the utilization of Stacking encoder-decoders with bidirectional LSTM cells leads to a notable enhancement in the accuracy of the model, resulting in maximum efficacy. A notably lower Mean Absolute Error (MAE) result suggests that the 9-layer stacked autoencoder model has superior performance in predicting both the minimum and maximum temperature.

Graph Transformer GANs With Graph Masked Modeling for Architectural Layout Generation.

We present a novel graph Transformer generative adversarial network (GTGAN) to learn effective graph node relations in an end-to-end fashion for challenging graph-constrained architectural layout generation tasks. The proposed graph-Transformer-based generator includes a novel graph Transformer encoder that combines graph convolutions and self-attentions in a Transformer to model both local and global interactions across connected and non-connected graph nodes. Specifically, the proposed connected node attention (CNA) and non-connected node attention (NNA) aim to capture the global relations across connected nodes and non-connected nodes in the input graph, respectively. The proposed graph modeling block (GMB) aims to exploit local vertex interactions based on a house layout topology. Moreover, we propose a new node classification-based discriminator to preserve the high-level semantic and discriminative node features for different house components. To maintain the relative spatial relationships between ground truth and predicted graphs, we also propose a novel graph-based cycle-consistency loss. Finally, we propose a novel self-guided pre-training method for graph representation learning. This approach involves simultaneous masking of nodes and edges at an elevated mask ratio (i.e., 40%) and their subsequent reconstruction using an asymmetric graph-centric autoencoder architecture. This method markedly improves the model's learning proficiency and expediency. Experiments on three challenging graph-constrained architectural layout generation tasks (i.e., house layout generation, house roof generation, and building layout generation) with three public datasets demonstrate the effectiveness of the proposed method in terms of objective quantitative scores and subjective visual realism. New state-of-the-art results are established by large margins on these three tasks.

Batch correction of single-cell sequencing data via an autoencoder architecture.

Technical differences between gene expression sequencing experiments can cause variations in the data in the form of batch effect biases. These do not represent true biological variations between samples and can lead to false conclusions or hinder the ability to integrate multiple datasets. Since there is a growing need for the joint analysis of single-cell sequencing datasets from different sources, there is also a need to correct the resulting batch effects while maintaining the true biological variations in the data. We developed a semi-supervised deep learning architecture called Autoencoder-based Batch Correction (ABC) for integrating single-cell sequencing datasets. Our method removes batch effects through a guided process of data compression using supervised cell type classifier branches for biological signal retention. It aligns the different batches using an adversarial training approach. We comprehensively evaluate the performance of our method using four single-cell sequencing datasets and multiple measures for batch effect removal and biological variation conservation. ABC outperforms 10 state-of-the-art methods for this task including Seurat, scGen, ComBat, scanorama, scVI, scANVI, AutoClass, Harmony, scDREAMER, and CLEAR, correcting various types of batch effects while preserving intricate biological variations.

Privacy-Preserving Offloading in Edge Intelligence Systems With Inductive Learning and Local Differential Privacy

We address privacy and latency issues in edge-cloud computing environments where the neural network training is centralized. This paper considers the scenario where the edge devices are the only data sources for the deep learning model to be trained on the central server. Improper access to the massive amounts of data generated by edge devices could lead to privacy concerns. As a result, existing solutions for preserving privacy and reducing network latency in the edge environment rely on auxiliary datasets with no privacy risks or pre-trained models to build the client side feature extractor. However, finding auxiliary datasets or pre-trained models is not always guaranteed and may be challenging. To bridge this gap and eliminate the reliance on auxiliary datasets or pre-trained models of existing solutions, this paper presents DeepGuess, a privacy-preserving and latency-aware deep-learning framework. DeepGuess introduces a new learning mechanism enabled by the AutoEncoder architecture: inductive learning. With inductive learning, sensitive data stays on devices and is not explicitly sent to the central server to engage in back-propagations. To further enhance privacy, we propose a new local differential privacy algorithm that allows edge devices to apply random noise to features extracted from their sensitive data before being transferred to the non-trusted central server. The experimental evaluation of DeepGuess with various datasets and in a real-world scenario shows that our solution achieves comparable or even higher accuracy than existing solutions while reducing data transfer over the network by more than 50%.

Pattern recognition and signal parameters extraction using machine learning methods

Machine learning methods can be used for signal processing in different cases of physics research. A convolutional neural network was developed for the task of pulse counting in particle detectors for high energy physics. For the extraction of signal parameters was used a network with convolutional autoencoder architecture and a subsequent result reconstruction algorithm was developed and applied. A convolutional neural network was also developed for seismic studies with the task of identifying different events in seismograms. All of the algorithms with their architecture, input and output are presented and discussed.