Deep Generative Neural Networks Research Articles

Explainable semi-supervised deep learning shows that dementia is associated with small, avocado-shaped clocks with irregularly placed hands

The clock drawing test is a simple and inexpensive method to screen for cognitive frailties, including dementia. In this study, we used the relevance factor variational autoencoder (RF-VAE), a deep generative neural network, to represent digitized clock drawings from multiple institutions using an optimal number of disentangled latent factors. The model identified unique constructional features of clock drawings in a completely unsupervised manner. These factors were examined by domain experts to be novel and not extensively examined in prior research. The features were informative, as they distinguished dementia from non-dementia patients with an area under receiver operating characteristic (AUC) of 0.86 singly, and 0.96 when combined with participants’ demographics. The correlation network of the features depicted the “typical dementia clock” as having a small size, a non-circular or “avocado-like” shape, and incorrectly placed hands. In summary, we report a RF-VAE network whose latent space encoded novel constructional features of clocks that classify dementia from non-dementia patients with high performance.

Extreme and severe convective weather disasters: A dual-polarization radar nowcasting method based on physical constraints and a deep neural network model

It is difficult for traditional methods, such as the PySTEPS method, which is based on optical flows, and the UNet method, which is based on convolutional neural networks, to predict extreme weather events (e.g., hail, heavy rain, lightning and mesoscale cyclones) 0–60 min in advance because these events have short life cycles, occur at small scales, rapidly develop and are highly nonlinear. Based on the structural characteristics of severe weather, we propose a physically constrained generative adversarial deep neural network nowcasting model to improve the prediction accuracy and blurry data problem associated with nowcasting extreme convective weather within the 0–60 min interval. The improved TITAN algorithm is used to quickly identify and match strong storm cells in three dimensions, and the convection-diffusion constraint equation is established. Additionally, reflectivity (Zh), differential reflectivity (Zdr), the range derivative-specific differential phase (Kdp), correlation coefficient (cc), precipitation particle phase, and velocity divergence are considered when training the model with a large sample set, and constraint equations are used for optimization. The critical success index, false alarm ratio and miss probability scores of the nowcasting results obtained with this method are better than the PySTEPS and UNet results. Additionally, the root mean square error of a hailstorm nowcasted with a 30-min lead time is significantly better than that of the other methods, and the blurry data problem that appears over time is mitigated. We show that our model can provide nowcasts with high precision to support operations at various resolutions and lead times in cases for which alternative methods struggle.

Helium focused ion beam induced subsurface damage on Si and SiC substrates: experiments and generative deep neural network modeling via position-dependent input

Deep neural networks (DNNs) are reshaping many fields due to their end-to-end ability to learn directly from data, leading to outstanding performance, especially in image processing. However, their training requires large image datasets while image preparation can be expensive/time-consuming. If the underlying behavior can be formulated as spatial-dependent, we propose that every pixel from every image can be considered as a different data source, thus enabling a truly large pixel-based dataset with millions of elements (‘position-dependent input’), even when obtained from a few images. The approach is applied to helium focused ion beam nanofabrication, where the cross-section of the helium-damaged region essentially resembles that of a lightbulb, while close inspection reveals the presence of (i) an outer defective region in direct contact with the bulk substrate and (ii) an inner amorphous region filled with helium bubbles of gradually increasing size. Interestingly, the amorphous phase may swell upwards to form a protruding mesa, depending on the beam energy and dose, a feature that has resisted modeling so far. Through dedicated experiments on both Si and SiC substrates, and careful image analysis (segmentation), we describe the transformations that take place in the defective and amorphous regions with increasing energy and dose, and use the outlined position-dependent input together with a simple DNN of 4 hidden layers and 16 neurons per layer to describe all damage features realistically, inherently demonstrating generative behavior. This is the first time that any model predicts swelling satisfactorily. Generalization is surprisingly smooth and accurate.

Disruption prediction based on fusion feature extractor on J-TEXT

Predicting disruptions across different tokamaks is necessary for next generation device. Future large-scale tokamaks can hardly tolerate disruptions at high performance discharge, which makes it difficult for current data-driven methods to obtain an acceptable result. A machine learning method capable of transferring a disruption prediction model trained on one tokamak to another is required to solve the problem. The key is a feature extractor which is able to extract common disruption precursor traces in tokamak diagnostic data, and can be easily transferred to other tokamaks. Based on the concerns above, this paper presents a deep feature extractor, namely, the fusion feature extractor (FFE), which is designed specifically for extracting disruption precursor features from common diagnostics on tokamaks. Furthermore, an FFE-based disruption predictor on J-TEXT is demonstrated. The feature extractor is aimed to extracting disruption-related precursors and is designed according to the precursors of disruption and their representations in common tokamak diagnostics. Strong inductive bias on tokamak diagnostics data is introduced. The paper presents the evolution of the neural network feature extractor and its comparison against general deep neural networks, as well as a physics-based feature extraction with a traditional machine learning method. Results demonstrate that the FFE may reach a similar effect with physics-guided manual feature extraction, and obtain a better result compared with other deep learning methods.

Deep Crowd Anomaly Detection by Fusing Reconstruction and Prediction Networks

Abnormal event detection is one of the most challenging tasks in computer vision. Many existing deep anomaly detection models are based on reconstruction errors, where the training phase is performed using only videos of normal events and the model is then capable to estimate frame-level scores for an unknown input. It is assumed that the reconstruction error gap between frames of normal and abnormal scores is high for abnormal events during the testing phase. Yet, this assumption may not always hold due to superior capacity and generalization of deep neural networks. In this paper, we design a generalized framework (rpNet) for proposing a series of deep models by fusing several options of a reconstruction network (rNet) and a prediction network (pNet) to detect anomaly in videos efficiently. In the rNet, either a convolutional autoencoder (ConvAE) or a skip connected ConvAE (AEc) can be used, whereas in the pNet, either a traditional U-Net, a non-local block U-Net, or an attention block U-Net (aUnet) can be applied. The fusion of both rNet and pNet increases the error gap. Our deep models have distinct degree of feature extraction capabilities. One of our models (AEcaUnet) consists of an AEc with our proposed aUnet has capability to confirm better error gap and to extract high quality of features needed for video anomaly detection. Experimental results on UCSD-Ped1, UCSD-Ped2, CUHK-Avenue, ShanghaiTech-Campus, and UMN datasets with rigorous statistical analysis show the effectiveness of our models.

A Context-Aware Image Generation Method for Assisted Design of Movie Posters Using Generative Adversarial Network

Considering the continuous development of the film industry and the improvement of the living standard among people, movies have gradually come to the civilians. A good movie poster can effectively reflect the content of the movie, attract the audience, stimulate the demand and achieve a good publicity effect. The current movie poster design work is mainly carried out by professional designers, which requires a lot of time and labor cost. In this paper, we propose a context-aware image generation method for assisted design of movie posters using generative adversarial network (named as MPAD-CIP for short). First, the basic information and visual contents of the movie are perceived, and the representative images are extracted, with the use of convolution operations. Then, a backbone network of deep convolutional generative neural network is formulated to generate images for summary of movies. The backbone network is composed of two components: a generator and a discriminator. Their combination realizes the computer-assisted movie poster design by sensing visual context. In the experimental part, the proposed MPAD-CIP method is compared with several benchmark models to demonstrate that the posters generated by this paper are more realistic and versatile, and some of the generated posters are exhibited.

Prediction of the Soil Permeability Coefficient of Reservoirs Using a Deep Neural Network Based on a Dendrite Concept

Changes in the pore water pressure of soil are essential factors that affect the movement of structures during and after construction in terms of stability and safety. Soil permeability represents the quantity of water transferred using pore water pressure. However, these changes cannot be easily identified and require considerable time and money. This study predicted and evaluated the soil permeability coefficient using a multiple regression (MR) model, adaptive network-based fuzzy inference system (ANFIS), general deep neural network (DNN) model, and DNN using the dendrite concept (DNN−T, which was proposed in this study). The void ratio, unit weight, and particle size were obtained from 164 undisturbed samples collected from the embankments of reservoirs in South Korea as input variables for the aforementioned models. The data used in this study included seven input variables, and the ratios of the training data to the validation data were randomly extracted, such as 6:4, 7:3, and 8:2, and were used. The analysis results for each model showed a median correlation of r = 0.6 or less and a low model efficiency of Nash–Sutcliffe efficiency (NSE) = 0.35 or less as a result of predicting MR and ANFIS. The DNN and DNN−T both have good performance, with a strong correlation of r = 0.75 or higher. Evidently, the DNN−T performance in terms of r, NSE, and root mean square error (RMSE) improved more than that of the DNN. However, the difference between the mean absolute percent error (MAPE) of DNN−T and the DNN was that the error of the DNN was small (11%). Regarding the ratio of the training data to the verification data, 7:3 and 8:2 showed better results compared to 6:4 for indicators, such as r, NSE, RMSE, and MAPE. We assumed that this phenomenon was caused by the DNN−T thinking layer. This study shows that DNN−T, which changes the structure of the DNN, is an alternative for estimating the soil permeability coefficient in the safety inspection of construction sites and is an excellent methodology that can save time and budget.

Cross-Domain Echocardiography Segmentation with Multi-Space Joint Adaptation.

The segmentation of the left ventricle endocardium (LVendo) and the left ventricle epicardium (LVepi) in echocardiography plays an important role in clinical diagnosis. Recently, deep neural networks have been the most commonly used approach for echocardiography segmentation. However, the performance of a well-trained segmentation network may degrade in unseen domain datasets due to the distribution shift of the data. Adaptation algorithms can improve the generalization of deep neural networks to different domains. In this paper, we present a multi-space adaptation-segmentation-joint framework, named MACS, for cross-domain echocardiography segmentation. It adopts a generative adversarial architecture; the generator fulfills the segmentation task and the multi-space discriminators align the two domains on both the feature space and output space. We evaluated the MACS method on two echocardiography datasets from different medical centers and vendors, the publicly available CAMUS dataset and our self-acquired dataset. The experimental results indicated that the MACS could handle unseen domain datasets well, without requirements for manual annotations, and improve the generalization performance by 2.2% in the Dice metric.

Improving the Generalization of Deep Neural Networks in Seismic Resolution Enhancement

Seismic resolution enhancement is a key step for subsurface structure characterization. Although many have proposed the use of deep learning (DL) for resolution enhancement, these are typically hindered by the limitations in the application of synthetically trained networks onto real datasets. Domain adaptation (DA) offers an approach to reduce this disparity between training and inference data, aiming through the application of data transformations to bring the distributions of both data closer to each other. We propose a simple DA procedure, termed MLReal-Lite (the light version of the earlier introduced MLReal), that mainly relies on linear operations, namely convolution and correlation; these transformations introduce aspects of the field data into the synthetic data prior to training, and vice-versa with regard to the inference stage. Taking 1-D and 2-D resolution enhancement tasks as examples, we show how the inclusion of MLReal-Lite improves the performance of neural networks. Not only do the results demonstrate notable improvements in seismic resolution, they also exhibit a higher signal-to-noise ratio (SNR) and better continuity of events, in comparison to the tests without MLReal-Lite. Finally, while illustrated on a resolution enhancement task, our proposed methodology is applicable for any seismic data of dimensions N-D, offering a DA applicable from well ties through to 3-D seismic volumes, and beyond.

Tumor-Attentive Segmentation-Guided GAN for Synthesizing Breast Contrast-Enhanced MRI Without Contrast Agents.

Breast dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a sensitive imaging technique critical for breast cancer diagnosis. However, the administration of contrast agents poses a potential risk. This can be avoided if contrast-enhanced MRI can be obtained without using contrast agents. Thus, we aimed to generate T1-weighted contrast-enhanced MRI (ceT1) images from pre-contrast T1 weighted MRI (preT1) images in the breast. We proposed a generative adversarial network to synthesize ceT1 from preT1 breast images that adopted a local discriminator and segmentation task network to focus specifically on the tumor region in addition to the whole breast. The segmentation network performed a related task of segmentation of the tumor region, which allowed important tumor-related information to be enhanced. In addition, edge maps were included to provide explicit shape and structural information. Our approach was evaluated and compared with other methods in the local (n = 306) and external validation (n = 140) cohorts. Four evaluation metrics of normalized mean squared error (NRMSE), Pearson cross-correlation coefficients (CC), peak signal-to-noise ratio (PSNR), and structural similarity index map (SSIM) for the whole breast and tumor region were measured. An ablation study was performed to evaluate the incremental benefits of various components in our approach. Our approach performed the best with an NRMSE 25.65, PSNR 54.80 dB, SSIM 0.91, and CC 0.88 on average, in the local test set. Performance gains were replicated in the validation cohort. We hope that our method will help patients avoid potentially harmful contrast agents. Clinical and Translational Impact Statement-Contrast agents are necessary to obtain DCE-MRI which is essential in breast cancer diagnosis. However, administration of contrast agents may cause side effects such as nephrogenic systemic fibrosis and risk of toxic residue deposits. Our approach can generate DCE-MRI without contrast agents using a generative deep neural network. Thus, our approach could help patients avoid potentially harmful contrast agents resulting in an improved diagnosis and treatment workflow for breast cancer.

A Novel Jointly Optimized Cooperative DAE-DNN Approach Based on a New Multi-Target Step-Wise Learning for Speech Enhancement

In this paper, we present a new supervised speech enhancement approach based on a cooperative structure of deep autoencoders (DAEs) as generative models and deep neural networks (DNN). The DAE is used as a nonlinear alternative to nonnegative matrix factorization (NMF) for the extraction of harmonic structures and encoded features of the noise, clean and noisy signals, and a DNN is deployed as a nonlinear mapper. We introduce a deep network imitating NMF in a non-linear manner to overcome the problems of a simple linear model, such as performance degradation in non-stationary environments. Compared to combinatorial NMF and DNN methods, we do all of the decomposition, enhancement, and reconstruction processes in a nonlinear framework via a suitable cooperative structure of encoder, DNN, and decoders, and jointly optimize them. We also propose a supervised hierarchical multi-target training approach, performed in two steps, such that the DNN not only predicts the low-level encoded features as primary targets but it also predicts the high-level actual spectral signals as secondary targets. The first step acts as a pretraining for the second step which improves the learning strategy. Moreover, to exploit a more discriminative model for noise reduction, a DNN-based noise classification and fusion strategy (NCF) is also proposed. The experiments on TIMIT dataset reveal that the proposed methods outperform the previous approaches and achieve an average perceptual evaluation of speech quality (PESQ) improvement of up to about 0.3 for speech enhancement.

Hybrid Optimization-Enabled Deep Learning for Indoor Object Detection and Distance Estimation to Assist Visually Impaired Persons

Indoor object recognition mainly deals with recognizing indoor objects and in recent years, it becomes a crucial research topic among investigators. Visually Impaired Persons (VIPs) is a group of people who have a disability to visualize objects and indoor assistance to such people for safe navigation seems to be a challenging task. Several innovative devices and methods have been developed to assist VIP's towards their destination but, most of them failed to address the problem of multi-class object recognition. Moreover, such methods are incapable to provide an accurate distance between an object and a person. To bridge this gap, an efficacious model is proposed to assist VIP's in their daily lives for indoor object recognition using a newly designed Honey Adam African Vultures Optimization (HAAVO) algorithm. Here, object detection and object recognition is carried out using Generative Adversarial Network (GAN) and Deep Convolutional Neural Network (DCNN), respectively. The DCNN classifier and Deep Residual Network (DRN) utilized to estimate the distance is optimally trained using the proposed HAAVO. In addition, the designed HAAVO is obtained by integration of the Honey Badger Algorithm (HBA) with Adam Optimizer and African Vultures Optimization (AVO). The devised model provided high accuracy with respect to object detection and delivered superior performance values in accordance with testing accuracy, precision, and recall with the measures of 0.940, 0.946, and 0.953.

Deep generative modeling and clustering of single cell Hi-C data.

Deciphering 3D genome conformation is important for understanding gene regulation and cellular function at a spatial level. The recent advances of single cell Hi-C technologies have enabled the profiling of the 3D architecture of DNA within individual cell, which allows us to study the cell-to-cell variability of 3D chromatin organization. Computational approaches are in urgent need to comprehensively analyze the sparse and heterogeneous single cell Hi-C data. Here, we proposed scDEC-Hi-C, a new framework for single cell Hi-C analysis with deep generative neural networks. scDEC-Hi-C outperforms existing methods in terms of single cell Hi-C data clustering and imputation. Moreover, the generative power of scDEC-Hi-C could help unveil the differences of chromatin architecture across cell types. We expect that scDEC-Hi-C could shed light on deepening our understanding of the complex mechanism underlying the formation of chromatin contacts.

Inverse Design of Digital Materials Using Corrected Generative Deep Neural Network and Generative Deep Convolutional Neural Network

Generative networks are effective tools for digital materials (DM) inverse design. However, the optimization performance of generative networks is restricted by the increasing discrepancy between the optimized input and the prescribed input domain as the design loop increases. Herein, a correction technique is incorporated into generative deep neural network (GDNN) and generative deep convolutional neural network (GDCNN). The correction is performed by pulling the machine learning (ML)‐optimized inputs back to the prescribed domain at certain interval during the optimization process instead of only postprocessing at the end. A DM system with two phases, i.e., the matrix phase and the pore phase, is used for the structural design. The datasets are produced using a numerical model and describe the relationship between the material structure and the elastic modulus and tensile strength. The results show that the optimization effectiveness of corrected GDNN/GDCNN significantly improves given the fact that more structures converge to the best structures and fewer nonrepetitive structures are left after optimization, which helps to search for the best structures and decreases the computational burden when verifying the ML‐recommended structures. The corrected GDNN and GDCNN also manage to find structures with higher tensile strength in the new inverse design.

Real-Time Phaseless Microwave Frequency-Diverse Imaging with Deep Prior Generative Neural Network

The millimeter-wave frequency-diverse imaging regime has recently received considerable attention in both the security screening and synthetic aperture radar imaging literature. Considering that the minor systematic errors and alignment errors could still produce heavily corrupted images, these complex-based imaging reconstructions rely heavily on the precise measurement of both phase and amplitude of radiation field patterns and echo signals. In the literature, it is shown that by leveraging phase-retrieval techniques, salient reconstruction images can still be acquired, even in the presence of significant phase errors, which could ease the phase error calibration pressure to a large extent in practical imaging applications. In this paper, in the regime of phaseless frequency-diverse imaging, with the powerful feature inference and generation power of unsupervised generative models, an end-to-end deep prior generative neural network is designed to achieve near real-time imaging. The harsh imaging reconstruction with both the high radiation mode correlations and extremely low scene compression sampling ratio, which are extremely troublesome to tackle for generally applied matched-filter and compressed sensing approach in the current frequency-diverse imaging literature, can still be preferably handled with our reconstruction network. The well-trained reconstruction network is constituted by prior inference and deep generative modules with excellent generative capabilities and significant prior inference abilities. Using simulation experiments with radiation field data, we verify that the integration of phase-free frequency-change imaging with deep learning networks can effectively improve reconstruction capabilities and improve robustness to systematic phase errors. Compared with existing imaging methods, our imaging method has high imaging performance and can even reconstruct targets under low compression ratio conditions, which is somewhat competitive with current state-of-the-art algorithms. Moreover, we find that the proposed method has good anti-noise and stability.

Abnormal amygdala functional connectivity and deep learning classification in multifrequency bands in autism spectrum disorder: A multisite functional magnetic resonance imaging study.

Previous studies have explored resting-state functional connectivity (rs-FC) of the amygdala in patients with autism spectrum disorder (ASD). However, it remains unclear whether there are frequency-specific FC alterations of the amygdala in ASD and whether FC in specific frequency bands can be used to distinguish patients with ASD from typical controls (TCs). Data from 306 patients with ASD and 314 age-matched and sex-matched TCs were collected from 28 sites in the Autism Brain Imaging Data Exchange database. The bilateral amygdala, defined as the seed regions, was used to perform seed-based FC analyses in the conventional, slow-5, and slow-4 frequency bands at each site. Image-based meta-analyses were used to obtain consistent brain regions across 28 sites in the three frequency bands. By combining generative adversarial networks and deep neural networks, a deep learning approach was applied to distinguish patients with ASD from TCs. The meta-analysis results showed frequency band specificity of FC in ASD, which was reflected in the slow-5 frequency band instead of the conventional and slow-4 frequency bands. The deep learning results showed that, compared with the conventional and slow-4 frequency bands, the slow-5 frequency band exhibited a higher accuracy of 74.73%, precision of 74.58%, recall of 75.05%, and area under the curve of 0.811 to distinguish patients with ASD from TCs. These findings may help us to understand the pathological mechanisms of ASD and provide preliminary guidance for the clinical diagnosis of ASD.

Emotion Understanding from Facial Expressions using Stacked Generative Adversarial Network (GAN) and Deep Convolution Neural Network (DCNN)

Emotion Understanding from Facial Expressions using Stacked Generative Adversarial Network (GAN) and Deep Convolution Neural Network (DCNN)

Coupling a Neural Network with a Spatial Downscaling Procedure to Improve Probabilistic Nowcast for Urban Rain Radars

Rainfall forecasting plays a key role in mitigating environmental risks in urban areas, which are subject to increasing hydrogeological risk due to transformations in the urban landscape. We present a new technique for probabilistic precipitation nowcasting by generating an ensemble of equiprobable forecasts, which is especially useful for weather radars with limited spatial range, and that can be used operationally on devices with low computational capacity. The ensemble members are obtained by a novel stochastic noise generation process, consistent with the spatial scales not resolved by the prediction model, which allows continuous downscaling of the output of a deep generative neural network. Through an in-depth analysis of the results, for precipitation accumulated in the first hour, measured by all the most robust skill indicators, extended to an entire year of data at 5-min time resolution, we demonstrate that the proposed procedure is able to provide calibrated, reliable, and sharp ensemble rainfall forecasts with a quality comparable or superior to the state-of-the-art classical alternative optical flow technique. The ensemble generation procedure we propose is sufficiently general to be applied in principle to other deterministic architectures as well, thus enabling their generalization in probabilistic terms.

Generative and reinforcement learning approaches for the automated de novo design of bioactive compounds

Deep generative neural networks have been used increasingly in computational chemistry for de novo design of molecules with desired properties. Many deep learning approaches employ reinforcement learning for optimizing the target properties of the generated molecules. However, the success of this approach is often hampered by the problem of sparse rewards as the majority of the generated molecules are expectedly predicted as inactives. We propose several technical innovations to address this problem and improve the balance between exploration and exploitation modes in reinforcement learning. In a proof-of-concept study, we demonstrate the application of the deep generative recurrent neural network architecture enhanced by several proposed technical tricks to design inhibitors of the epidermal growth factor (EGFR) and further experimentally validate their potency. The proposed technical solutions are expected to substantially improve the success rate of finding novel bioactive compounds for specific biological targets using generative and reinforcement learning approaches.

Synthesis of a six-bar mechanism for generating knee and ankle motion trajectories using deep generative neural network

Robotic exoskeletons have demonstrated their effectiveness in post-stroke gait rehabilitation therapy. Nevertheless, further research is being conducted to improve existing rehabilitation exoskeletons in terms of ease-of-use and innovative design. Previously, the adaptation of linkage-based mechanisms for rehabilitation exoskeletons has been considered an option. However, finding linkage parameters that will produce the required gait trajectories using a linkage-based exoskeleton, is quite challenging. It is furthermore challenging to obtain parameters of a linkage-based mechanism designed for a gait rehabilitation task that has to produce two trajectories (for knee and ankle joints) simultaneously. In this work, we propose Deep Generative Neural Networks (DGNN) to obtain a set of optimal dimensions and parameters for the Stephenson III six-bar linkage-based gait exoskeleton. The proposed methodology demonstrates high efficacy in determining the linkage parameters for various target trajectories. The proposed framework, once trained, can accurately predict mechanism parameters to achieve two joint trajectories simultaneously. Subsequent to developing the model, walking trajectories from healthy human subjects are given to the model to determine the optimal linkage dimensions of the gait rehabilitation exoskeleton. The proposed model can be used to assist designers in quickly determining the optimized linkage dimensions of linkage-based mechanisms that can provide various target trajectories.