Deep Convolutional Autoencoder Research Articles

Characterization of the Autoencoder Radiation Anomaly Detection (ARAD) model

In this work we demonstrate an in-depth analysis and characterization of the Autoencoder Radiation Anomaly Detection (ARAD) algorithm. ARAD is a deep convolutional autoencoder designed to detect anomalous radioactive signatures in gamma-ray spectra collected by NaI(Tl) detectors. This model works by learning a dimensionally constrained representation of background gamma-ray spectra called the latent space. The latent space cannot fully describe anomalous components in new spectra, resulting in a decrease in spectral reconstruction accuracy that triggers an alarm. This paper demonstrates the model’s performance on a set of data collected outside of the High Flux Isotope Reactor and Radiochemical Engineering Development Center facilities at Oak Ridge National Laboratory. We also perform an evaluation of the model’s detection performance using a set of publicly available synthetic data representing a radiation detector moving throughout an urban city street. We demonstrate the algorithm’s ability to detect sources in locations with highly dynamic background count rates resulting from variations in naturally occurring radioactive materials and precipitation-induced radon washout, a challenge for many traditional radiation detection algorithms. We compare these results against those from another unsupervised radiation anomaly detection algorithm based on principal component analysis. ARAD achieved excellent performance on both datasets and proves the viability and efficacy of autoencoders for radiation anomaly detection.

Deep convolutional autoencoder augmented CFD thermal analysis of bearings with inter pad groove mixing

The treatment of thermal mixing in inter pad grooves of a fluid film bearing is essential due to its influence on the heat transfer with the rotating shaft and stationary bearing. Lower fidelity models that either neglect or over approximate thermal groove mixing may lead to premature bearing or machinery failure, most commonly from babbitt thermally induced fatigue. Conventional models rely on bulk flow and thermal analyses yielding a single temperature at the groove outlet into the pad inlet. The high uncertainty of this approach carries over into downstream predictions for bearing life, stiffness and damping, and machinery vibration predictions. Contrary to a uniform temperature, CFD-Conjugate heat transfer studies reveal large gradient temperature distributions varying in both the radial and axial directions at the groove outlet, especially with jet lubrication implemented with multiple nozzles. These distributions vary continuously with time as the spinning shaft and bearing pads vibrate. A direct CFD simulation thus becomes computationally prohibitive.The present work introduces a novel approach which yields highly detailed lubricant temperature distributions at the pad inlets in a computationally economical manner. This is implemented with a surrogate groove model via a deep convolutional autoencoder neural network based on CFD (Computational Fluid Dynamics) data. The trained Convolutional Neural Network (CNN) shows excellent prediction capability for 2D temperature distribution at a circumferential groove outlet. The trained CNN is combined with a rotor-bearing model, and the combined model is verified by full CFD results and experimental data. In addition, this approach is expanded to include various oil injection types, illustrating their detailed heat transfer to the rotating shaft and bearing.

Knowledge extraction and retention based continual learning by using convolutional autoencoder-based learning classifier system

An ideal artificial intelligence-based autonomous system, interacting with a dynamic environment, is required to learn continuously as human do. Human beings retain the learned knowledge, accumulate, and utilize it to solve related problems. Currently, most artificial intelligence-based systems lack this capability and work in an isolated learning paradigm. In this paper, we present a novel continual learning model to solve the challenging problem of real-world images classification. The proposed model is capable of learning continuously by utilizing the previously learned knowledge. It can handle both multi-task and single incremental task scenarios as opposed to various existing models that cover only the multi-task scenarios. In the proposed model, a deep convolutional autoencoder is presented to extract features from images. In addition, a learning classifier system with an effective knowledge encoding scheme is proposed for mapping real-world images to code fragment-based compact knowledge representation. Experiments are conducted on three benchmark image datasets to validate the model: (i) CORe50, (ii) iCubWorld28, and (iii) STL-10. Experiments results demonstrate that the proposed model outperforms the baseline method as well as various state-of-the-art methods for both continual learning scenarios.

Automated Transient Electromagnetic Data Processing for Ground-Based and Airborne Systems by a Deep Learning Expert System

Modern transient electromagnetic (TEM) surveys, either ground-based or airborne, may yield thousands of line kilometers of data. Parts of these data, especially in areas with dense infrastructure, are often disturbed by electromagnetic couplings due to infrastructure, e.g., power cables and fences. In most cases and in particular when working in a hydro-geological context, such coupled data must be culled before inversion. The process of identifying and culling coupled data is a manual task, requiring specialists to examine and process the data in detail. Manual data processing is subjective, difficult to reproduce, and time-consuming. To automate the complex data processing workflows, we propose an expert system based on a deep convolutional auto-encoder to identify couplings in the data. We configure the auto-encoder to learn an encoded representation of TEM data in a latent space. A reconstruction part that decodes the encoded representation is also trained, aiming to reconstruct input data. If the data unaffected by electromagnetic couplings are observed by the auto-encoder, the reconstructed output will have low error to the input. However, when having couplings in the data, the reconstruction error is elevated, indicating a non-geologic anomaly. The size of the anomaly is based on the relative error between the input data and the reconstructed output normalized by the data standard deviation. We show that the proposed approach displays high quality data processing within a fraction of a second for a ground-based and an airborne system, which is either ready for inversion or requires minimal further quality inspection.

LOUD: Local Orthogonalization-Constrained Unsupervised Deep-Learning Denoiser

Random noise attenuation of seismic data is a fundamental problem in seismic data processing. It is not only an important problem itself but also is a crucial step for the subsequent tasks, e.g., migration and inversion. We propose a local orthogonalization constrained unsupervised deep learning denoiser (LOUD) to suppress seismic random noise based on a new loss function that specifically adapts to seismic data. Through unsupervised learning, we eliminate the common need in supervised learning approaches of collecting or generating sizable clean and noisy image pairs, which is challenging and expensive, especially for seismic data. We utilize a deep convolutional autoencoder to reconstruct the clean seismic image and leverage the local signal-and-noise orthogonalization as a constraint to guarantee that the removed noise component is orthogonal to the recovered signal. Experimental results on both synthetic and field datasets exhibit the effectiveness of our proposed method over traditional denoising methods.

Unsupervised Erratic Seismic Noise Attenuation With Robust Deep Convolutional Autoencoders

Erratic seismic noise, following a (known or unknown) non-Gaussian distribution, poses a formidable challenge to conventional methods of random noise attenuation. Many erratic noise cancellation methods, for instance, robust reduced-rank and sparsity-promoting filtering, have been proven to achieve promising results in overcoming this challenge. Among them, deep learning (DL) methods require no assumptions about the underlying clear seismic image and are also more robust against erratic and random noise. However, the success of existing DL-based denoising methods strongly depends on supervised learning from a large number of ground-truth seismic images affected by erratic noise and their clean counterparts, which are typically unavailable in a real-world setting. As an alternative, this article presents an unsupervised DL method for erratic-plus-Gaussian noise removal based on a robust deep convolutional autoencoder (RDCAE). In the RDCAE, the mean squared error (mse) loss in a classic DCAE is replaced by the smooth Welsch function to exploit the concept of robust image denoising. In this way, the erratic noise is downweighted by means of a curbed weight defined in terms of the Welsch function. In contrast, the random noise is diluted by combining the mean square in the Welsch function and the total variation (TV). Subsequently, the training procedures required for solving the RDCAE are derived on the basis of the backpropagation (BP) algorithm for a neural network. Experiments conducted on both synthetic and real field datasets are reported to illustrate the efficacy of the proposed method.

Spectral-Spatial Deep Support Vector Data Description for Hyperspectral Anomaly Detection

Hyperspectral anomaly detection (HAD) aims to distinguish anomalies from background-by-background modeling. Deep learning has been applied to HAD and achieves promising detection results. However, there exist several issues that need to be addressed: 1) unrealistic Gaussian assumption on the latent representations may limit its application; 2) deep features are not well-suited to anomaly detection due to the separation between feature learning and anomaly detection; 3) lack of adequate exploitation of spectral-spatial features; 4) negative effect caused by spectral band redundancy. In this article, we propose an end-to-end trainable deep one-class classification network for HAD. Specifically, a minimal enclosing hypersphere is trained to involve the deep features of background samples. These background samples are selected by a density clustering-based method. In this way, feature learning and anomaly detection are incorporated into a unified framework. Meanwhile, there is no explicit Gaussian assumption on the background features. Moreover, due to the complementarity of spectral and spatial features, a novel feature fusion strategy is proposed to fuse spectral and spatial features extracted by a two-stream deep convolutional autoencoder network. Finally, a band attention module is used to automatically learn small weights for redundant bands and thus reduce the negative effect caused by redundant bands. Experimental results on five public datasets demonstrate the superiority of the proposed method compared to several state-of-the-art HAD methods in the detection performance.

Model‐Reduced Adjoint‐Based Inversion Using Deep‐Learning: Example of Geological Carbon Sequestration Modeling

AbstractAs an extension of our previous research on deep‐learning‐based adjoint‐state approach (Xiao, Deng, & Wang, 2021, https://doi.org/10.1029/2020wr027400), we present a two‐level hybrid neural network architecture to efficiently derive a model‐reduced adjoint‐based optimization workflow with a large‐scale inverse modeling example in geological carbon‐storage applications. The first level employs a nonlinear dimensionality reduction technique, that is, deep convolutional autoencoder, owing to its high scalability and the availability of high‐performance deep‐learning libraries for flexible implementation. Analogous to the reduced‐order linear model, the second level emulates a neural‐network‐powered linear transition unit based on the reduced subspace. This hybrid framework provides a readily available model‐reduced adjoint derivation with negligible computational cost and computer storage. Once the gradient is obtained in reduced space, it is correspondingly projected back to full space for the inverse modeling. The performance of the method is verified by estimating the space‐dependent stochastic permeability field using two geological carbon storage (GCS) process‐based multiphase flow models with an increasing complexity. The experimental results indicate that the surrogate model can accurately characterize the spatial‐temporal evolution of the CO2 saturation fields using a relatively small number of training data. Although the proposed inversion workflow achieves comparable results similar to the conventional finite‐difference method, it is faster than high‐fidelity model simulations in several orders of magnitude; hence, it will be a beneficial simulation and modeling tool for engineers to manage the uncertainty of GCS processing.

RUL Prediction of Rolling Bearings Based on a DCAE and CNN

Predicting the remaining useful life (RUL) of mechanical equipment can improve production efficiency while effectively reducing the life cycle cost and failure rate. This paper proposes a method for predicting the remaining service life of equipment through a combination of a deep convolutional autoencoder (DCAE) and a convolutional neural network (CNN). For rolling bearings, a health indicator (HI) could be built by combining DCAE and self-organizing map (SOM) networks, performing more advanced characterization against the original vibration data and modeling the degradation state of the rolling bearings. The HI serves as the label of the original vibration data, and the original data with such label is input into the prediction model of the RUL based on a one-dimensional convolutional neural network (1D-CNN). The model was trained for predicting the RUL of a rolling bearing. The bearing degradation dataset was evaluated to verify the method’s effectiveness. The results demonstrate that the constructed HI can characterize the bearing degradation state effectively and that the method of predicting the RUL can accurately predict the bearing degradation trend.

Cover: International Journal of Intelligent Systems, Volume 36 Issue 12 December 2021

Cover Caption: The cover image is based on the Research Article MR-DCAE: Manifold regularization-based deep convolutional autoencoder for unauthorized broadcasting identification by Qinghe Zheng et al., https://doi.org/10.1002/int.22586.

Improving geometric P-norm-based glioma segmentation through deep convolutional autoencoder encapsulation

Gliomas are the most common types of brain tumors. Their shapes are irregular and their boundaries are ambiguous, producing a tumor that is hard to detect. To improve the segmentation performance, we propose a P-Norm-Based Glioma Segmentation by encapsulating it within a Deep Convolutional Autoencoder (DCA). In this paper, we first develop a new robust Deep P-norm Generative Model (DPGM) that can extract and classify the feature by the encoder section and segment this latter by the decoder section. After that, we utilise a Deep Discriminative Model (DDM) which applies the conditional random fields to refine the segmentation result. Effectiveness is gained by gradient descent training on the set of pre-P-norm filtering and thresolding convolutional operations using a novel geometric P-norm-based loss function. This latter will be optimized by the RANSAC layer, which is geared toward optimizing the strength of the correct model relative to the most convincing false model. We evaluate the accuracy and robustness of our method within MRI sequences (T1, T2, T2c & FLAIR) from the Multimodal Brain Tumor Image Segmentation Challenge (BRATS 2013 Leaderboard, BRATS 2013 Challenge, BRATS 2015 and BRATS 2017). The experiments demonstrate a high-quality segmentation results of our DPGM-DDM architecture and the average Dice is as high as 97.2%, 97.6%, and 98.4% respectively for complete, core, and enhancing tumor regions. Overall, our deep proposed architecture is effective in segmenting gliomas in multimodal images or fair images, and has the potential in routine examinations of gliomas in daily clinical practice.

Non-Invasive Hemodynamics Monitoring System Based on Electrocardiography via Deep Convolutional Autoencoder.

This study evaluates cardiovascular and cerebral hemodynamics systems by only using non-invasive electrocardiography (ECG) signals. The Massachusetts General Hospital/Marquette Foundation (MGH/MF) and Cerebral Hemodynamic Autoregulatory Information System Database (CHARIS DB) from the PhysioNet database are used for cardiovascular and cerebral hemodynamics, respectively. For cardiovascular hemodynamics, the ECG is used for generating the arterial blood pressure (ABP), central venous pressure (CVP), and pulmonary arterial pressure (PAP). Meanwhile, for cerebral hemodynamics, the ECG is utilized for the intracranial pressure (ICP) generator. A deep convolutional autoencoder system is applied for this study. The cross-validation method with Pearson’s linear correlation (R), root mean squared error (RMSE), and mean absolute error (MAE) are measured for the evaluations. Initially, the ECG is used to generate the cardiovascular waveform. For the ABP system—the systolic blood pressure (SBP) and diastolic blood pressures (DBP)—the R evaluations are 0.894 ± 0.004 and 0.881 ± 0.005, respectively. The MAE evaluations for SBP and DBP are, respectively, 6.645 ± 0.353 mmHg and 3.210 ± 0.104 mmHg. Furthermore, for the PAP system—the systolic and diastolic pressures—the R evaluations are 0.864 ± 0.003 mmHg and 0.817 ± 0.006 mmHg, respectively. The MAE evaluations for systolic and diastolic pressures are, respectively, 3.847 ± 0.136 mmHg and 2.964 ± 0.181 mmHg. Meanwhile, the mean CVP evaluations are 0.916 ± 0.001, 2.220 ± 0.039 mmHg, and 1.329 ± 0.036 mmHg, respectively, for R, RMSE, and MAE. For the mean ICP evaluation in cerebral hemodynamics, the R and MAE evaluations are 0.914 ± 0.003 and 2.404 ± 0.043 mmHg, respectively. This study, as a proof of concept, concludes that the non-invasive cardiovascular and cerebral hemodynamics systems can be potentially investigated by only using the ECG signal.

MR‐DCAE: Manifold regularization‐based deep convolutional autoencoder for unauthorized broadcasting identification

Nowadays, radio broadcasting plays an important role in people's daily life. However, unauthorized broadcasting stations may seriously interfere with normal broadcastings and further disrupt the management of civilian spectrum resources. Since they are easily hidden in the spectrum and are essentially the same as normal signals, it still remains challenging to automatically and effectively identify unauthorized broadcastings in complicated electromagnetic environments. In this paper, we introduce the manifold regularization-based deep convolutional autoencoder (MR-DCAE) model for unauthorized broadcasting identification. The specifically designed autoencoder (AE) is optimized by entropy-stochastic gradient descent, then the reconstruction errors in the testing phase can be adopted to determine whether the received signals are authorized. To make this indicator more discriminative, we design a similarity estimator for manifolds spanning various dimensions as the penalty term to ensure their invariance during the back-propagation of gradients. In theory, the consistency degree between discrete approximations in the manifold regularization (MR) and the continuous objects that motivate them can be guaranteed under an upper bound. To the best of our knowledge, this is the first time that MR has been successfully applied in AE to promote cross-layer manifold invariance. Finally, MR-DCAE is evaluated on the benchmark data set AUBI2020, and comparative experiments show that it achieves state-of-the-art performance. To help understand the principle behind MR-DCAE, convolution kernels and activation maps of test signals are both visualized. It can be observed that the expert knowledge hidden in normal signals can be extracted and emphasized, rather than simple overfitting.

Impartially Validated Multiple Deep-Chain Models to Detect COVID-19 in Chest X-ray Using Latent Space Radiomics.

The COVID-19 pandemic continues to spread globally at a rapid pace, and its rapid detection remains a challenge due to its rapid infectivity and limited testing availability. One of the simply available imaging modalities in clinical routine involves chest X-ray (CXR), which is often used for diagnostic purposes. Here, we proposed a computer-aided detection of COVID-19 in CXR imaging using deep and conventional radiomic features. First, we used a 2D U-Net model to segment the lung lobes. Then, we extracted deep latent space radiomics by applying deep convolutional autoencoder (ConvAE) with internal dense layers to extract low-dimensional deep radiomics. We used Johnson–Lindenstrauss (JL) lemma, Laplacian scoring (LS), and principal component analysis (PCA) to reduce dimensionality in conventional radiomics. The generated low-dimensional deep and conventional radiomics were integrated to classify COVID-19 from pneumonia and healthy patients. We used 704 CXR images for training the entire model (i.e., U-Net, ConvAE, and feature selection in conventional radiomics). Afterward, we independently validated the whole system using a study cohort of 1597 cases. We trained and tested a random forest model for detecting COVID-19 cases through multivariate binary-class and multiclass classification. The maximal (full multivariate) model using a combination of the two radiomic groups yields performance in classification cross-validated accuracy of 72.6% (69.4–74.4%) for multiclass and 89.6% (88.4–90.7%) for binary-class classification.

OutlierNets: Highly Compact Deep Autoencoder Network Architectures for On-Device Acoustic Anomaly Detection.

Human operators often diagnose industrial machinery via anomalous sounds. Given the new advances in the field of machine learning, automated acoustic anomaly detection can lead to reliable maintenance of machinery. However, deep learning-driven anomaly detection methods often require an extensive amount of computational resources prohibiting their deployment in factories. Here we explore a machine-driven design exploration strategy to create OutlierNets, a family of highly compact deep convolutional autoencoder network architectures featuring as few as 686 parameters, model sizes as small as 2.7 KB, and as low as 2.8 million FLOPs, with a detection accuracy matching or exceeding published architectures with as many as 4 million parameters. The architectures are deployed on an Intel Core i5 as well as a ARM Cortex A72 to assess performance on hardware that is likely to be used in industry. Experimental results on the model’s latency show that the OutlierNet architectures can achieve as much as 30× lower latency than published networks.

Active Manifold and Model-Order Reduction to Accelerate Multidisciplinary Analysis and Optimization

A computational framework is proposed for efficiently solving multidisciplinary analysis and optimization (MDAO) problems in a relatively high-dimensional design parameter space. It relies on projection-based reduced-order models (PROMs) and a new concept of an active manifold (AM) to mitigate the curse of dimensionality during the training of the PROMs. The AM is discovered using a deep convolutional autoencoder for dimensionality reduction: it is proposed as a superior alternative to the concept of active subspace whose capabilities are limited by the associated affine approximation. The computational framework also blends the concept of a global PROM as a surrogate model of a nonlinear partial differential equation (PDE)-based behavior function with that of a database of local, linear PROMs for approximating a linear PDE-based behavior function. It is demonstrated for the solution of a flexible instance of NASA’s Common Research Model for transport aircraft, where the objective function pertains to aerodynamics, the constraint relates to flutter, and the design space contains 58 structural and shape parameters. The obtained results illustrate the potential of the AM concept for mitigating the aforementioned curse of dimensionality. They also demonstrate the feasibility of the proposed computational framework for realistic MDAO problems and its ability to reduce solution time.

Graph Regularized Residual Subspace Clustering Network for hyperspectral image clustering

Hyperspectral image (HSI) clustering is a challenging task due to the high complexity of HSI data. Subspace clustering methods have achieved promising clustering performance in HSI clustering. However, the existing subspace clustering methods calculate linear affinity rather than nonlinear relationships between data points and often ignore high-level feature extraction, resulting in low clustering accuracy. To address these issues, this paper proposes a Graph Regularized Residual Subspace Clustering Network (GR-RSCNet) that jointly learns deep spectral-spatial representation and robust nonlinear affinity via a deep neural network. Technically, we recast a graph regularized subspace clustering model as a special self-expressive (SE) layer that is integrated into a deep residual convolutional autoencoder. Benefitting from the residual structure, GR-RSCNet can be easily trained from scratch by optimizing a joint loss function consisting of reconstruction error, global manifold regularization term, and self-representation loss. We extensively evaluate the proposed GR-RSCNet for four popular HSI datasets, demonstrating that the GR-RSCNet can achieve state-of-the-art clustering results that significantly outperform many prior arts.

Using Channel State Information for Physical Tamper Attack Detection in OFDM Systems: A Deep Learning Approach

This letter proposes a deep learning approach to detect a change in the antenna orientation of transmitter or receiver as a physical tamper attack in OFDM systems using channel state information. We treat the physical tamper attack problem as a semi-supervised anomaly detection problem and utilize a deep convolutional autoencoder (DCAE) to tackle it. The past observations of the estimated channel state information (CSI) are used to train the DCAE. Then, a post-processing is deployed on the trained DCAE output to perform the physical tamper detection. Our experimental results show that the proposed approach, deployed in an office and a hall environment, is able to detect on average 99.6% of tamper events (TPR = 99.6%) while creating zero false alarms (FPR = 0%).

Deep Character-Level Anomaly Detection Based on a Convolutional Autoencoder for Zero-Day Phishing URL Detection

Considering the fatality of phishing attacks, the data-driven approach using massive URL observations has been verified, especially in the field of cyber security. On the other hand, the supervised learning approach relying on known attacks has limitations in terms of robustness against zero-day phishing attacks. Moreover, it is known that it is critical for the phishing detection task to fully exploit the sequential features from the URL characters. Taken together, to ensure both sustainability and intelligibility, we propose the combination of a convolution operation to model the character-level URL features and a deep convolutional autoencoder (CAE) to consider the nature of zero-day attacks. Extensive experiments on three real-world datasets consisting of 222,541 URLs showed the highest performance among the latest deep-learning methods. We demonstrated the superiority of the proposed method by receiver-operating characteristic (ROC) curve analysis in addition to 10-fold cross-validation and confirmed that the sensitivity improved by 3.98% compared to the latest deep model.

Deep learning early stopping for non-degenerate ghost imaging

Quantum ghost imaging offers many advantages over classical imaging, including the ability to probe an object with one wavelength and record the image with another (non-degenerate ghost imaging), but suffers from slow image reconstruction due to sparsity and probabilistic arrival positions of photons. Here, we propose a two-step deep learning approach to establish an optimal early stopping point based on object recognition, even for sparsely filled images. In step one we enhance the reconstructed image after every measurement by a deep convolutional auto-encoder, followed by step two in which a classifier is used to recognise the image. We test this approach on a non-degenerate ghost imaging setup while varying physical parameters such as the mask type and resolution. We achieved a fivefold decrease in image acquisition time at a recognition confidence of 75%. The significant reduction in experimental running time is an important step towards real-time ghost imaging, as well as object recognition with few photons, e.g., in the detection of light sensitive structures.