Deep Generative Networks Research Articles

Conditional Deep 3D-Convolutional Generative Adversarial Nets for RGB-D Generation

Generation of synthetic data is a challenging task. There are only a few significant works on RGB video generation and no pertinent works on RGB-D data generation. In the present work, we focus our attention on synthesizing RGB-D data which can further be used as dataset for various applications like object tracking, gesture recognition, and action recognition. This paper has put forward a proposal for a novel architecture that uses conditional deep 3D-convolutional generative adversarial networks to synthesize RGB-D data by exploiting 3D spatio-temporal convolutional framework. The proposed architecture can be used to generate virtually unlimited data. In this work, we have presented the architecture to generate RGB-D data conditioned on class labels. In the architecture, two parallel paths were used, one to generate RGB data and the second to synthesize depth map. The output from the two parallel paths is combined to generate RGB-D data. The proposed model is used for video generation at 30 fps (frames per second). The frame referred here is an RGB-D with the spatial resolution of 512 × 512.

On the capacity of deep generative networks for approximating distributions

We study the efficacy and efficiency of deep generative networks for approximating probability distributions. We prove that neural networks can transform a low-dimensional source distribution to a distribution that is arbitrarily close to a high-dimensional target distribution, when the closeness is measured by Wasserstein distances and maximum mean discrepancy. Upper bounds of the approximation error are obtained in terms of the width and depth of neural network. Furthermore, it is shown that the approximation error in Wasserstein distance grows at most linearly on the ambient dimension and that the approximation order only depends on the intrinsic dimension of the target distribution. On the contrary, when f-divergences are used as metrics of distributions, the approximation property is different. We show that in order to approximate the target distribution in f-divergences, the dimension of the source distribution cannot be smaller than the intrinsic dimension of the target distribution.

Using deep generative neural networks to account for model errors in Markov chain Monte Carlo inversion

SUMMARY Most geophysical inverse problems are non-linear and rely upon numerical forward solvers involving discretization and simplified representations of the underlying physics. As a result, forward modelling errors are inevitable. In practice, such model errors tend to be either completely ignored, which leads to biased and over-confident inversion results, or only partly taken into account using restrictive Gaussian assumptions. Here, we rely on deep generative neural networks to learn problem-specific low-dimensional probabilistic representations of the discrepancy between high-fidelity and low-fidelity forward solvers. These representations are then used to probabilistically invert for the model error jointly with the target geophysical property field, using the computationally cheap, low-fidelity forward solver. To this end, we combine a Markov chain Monte Carlo (MCMC) inversion algorithm with a trained convolutional neural network of the spatial generative adversarial network (SGAN) type, whereby at each MCMC step, the simulated low-fidelity forward response is corrected using a proposed model-error realization. Considering the crosshole ground-penetrating radar traveltime tomography inverse problem, we train SGAN networks on traveltime discrepancy images between: (1) curved-ray (high fidelity) and straight-ray (low fidelity) forward solvers; and (2) finite-difference-time-domain (high fidelity) and straight-ray (low fidelity) forward solvers. We demonstrate that the SGAN is able to learn the spatial statistics of the model error and that suitable representations of both the subsurface model and model error can be recovered by MCMC. In comparison with inversion results obtained when model errors are either ignored or approximated by a Gaussian distribution, we find that our method has lower posterior parameter bias and better explains the observed traveltime data. Our method is most advantageous when high-fidelity forward solvers involve heavy computational costs and the Gaussian assumption of model errors is inappropriate. Unstable MCMC convergence due to non-linearities introduced by our method remain a challenge to be addressed in future work.

Simulated annealing for optimization of graphs and sequences

Optimization of discrete structures aims at generating a new structure with the better property given an existing one, which is a fundamental problem in machine learning. Different from the continuous optimization, the realistic applications of discrete optimization (e.g., text generation) are very challenging due to the complex and long-range constraints, including both syntax and semantics, in discrete structures. In this work, we present SAGS, a novel Simulated Annealing framework for Graph and Sequence optimization. The key idea is to integrate powerful neural networks into metaheuristics (e.g., simulated annealing, SA) to restrict the search space in discrete optimization. We start by defining a sophisticated objective function, involving the property of interest and pre-defined constraints (e.g., grammar validity). SAGS searches from the discrete space towards this objective by performing a sequence of local edits, where deep generative neural networks propose the editing content and thus can control the quality of editing. We evaluate SAGS on paraphrase generation and molecule generation for sequence optimization and graph optimization, respectively. Extensive results show that our approach achieves state-of-the-art performance compared with existing paraphrase generation methods in terms of both automatic and human evaluations. Further, SAGS also significantly outperforms all the previous methods in molecule generation.

StyleGANs and Transfer Learning for Generating Synthetic Images in Industrial Applications

Deep learning applications on computer vision involve the use of large-volume and representative data to obtain state-of-the-art results due to the massive number of parameters to optimise in deep models. However, data are limited with asymmetric distributions in industrial applications due to rare cases, legal restrictions, and high image-acquisition costs. Data augmentation based on deep learning generative adversarial networks, such as StyleGAN, has arisen as a way to create training data with symmetric distributions that may improve the generalisation capability of built models. StyleGAN generates highly realistic images in a variety of domains as a data augmentation strategy but requires a large amount of data to build image generators. Thus, transfer learning in conjunction with generative models are used to build models with small datasets. However, there are no reports on the impact of pre-trained generative models, using transfer learning. In this paper, we evaluate a StyleGAN generative model with transfer learning on different application domains—training with paintings, portraits, Pokémon, bedrooms, and cats—to generate target images with different levels of content variability: bean seeds (low variability), faces of subjects between 5 and 19 years old (medium variability), and charcoal (high variability). We used the first version of StyleGAN due to the large number of publicly available pre-trained models. The Fréchet Inception Distance was used for evaluating the quality of synthetic images. We found that StyleGAN with transfer learning produced good quality images, being an alternative for generating realistic synthetic images in the evaluated domains.

Stochastic characterization and reconstruction of material microstructures for establishment of process-structure-property linkage using the deep generative model.

In material design, microstructure characterization and reconstruction are indispensable for understanding the role of a structure in a process-structure-property relation. The significant contribution of this paper is to introduce a methodology for the characterization and generation of material microstructures using deep generative networks as the first step in the establishment of a process-structure-property linkage for forward or inverse material design. Our approach can be divided into two parts: (i) characterization of material microstructures by a vector quantized variational auto-encoder, and (ii) determination of the correlation between the extracted microstructure characterizations and the given conditions, such as processing parameters and/or material properties, by a pixel convolutional neural network. As an example, we tested our framework in the generation of low-carbon-steel microstructures from the given material processing. The results were in satisfactory agreement with the experimental observation qualitatively and quantitatively, demonstrating the potential of applying the proposed method to forward or inverse material design. One of the advantages of the proposed methodology lies in the capability to capture the stochastic nature behind the microstructure generation. As a result, this methodology enables us to build a process-structure-property linkage while quantifying uncertainties, which not only makes a prediction more robust but also shows a way toward enhancing our understanding of the stochastic competitive phenomena behind the generation of material microstructures.

LRGAN: Visual anomaly detection using GAN with locality-preferred recoding

Deep neural networks, including deep auto-encoder (DAE) and generative adversarial networks (GAN), have been extensively applied for visual anomaly detection. These models generally assume that reconstruction errors should be lower for normal samples but higher for anomalies. However, it has been found that DAE based models can sometimes reconstruct anomalies very well and thus result in false alarms or misdetections. To address this problem, we propose a model using GAN with locality-preferred recoding, named LRGAN. LRGAN is inspired by the observation that both normal and abnormal samples are not completely scattered throughout the latent space but clustered separately at some local regions. Therefore, a locality-preferred recoding (LR) module is designed to compulsively represent the latent vectors of anomalies by normal ones. As a result, reconstructions of anomalies will approximate to normal samples and corresponding residuals can thus be enlarged. To partly avoid latent vectors of normal samples being recoded, we further present an improved model using GAN with an adaptive LR (ALR) module, named LRGAN+. ALR applies the clustering algorithm to generate a more compact codebook; more importantly, it helps LRGAN + automatically skip the LR module for possible normal samples with a threshold strategy. Our proposed method is evaluated on two public datasets (i.e., MNIST and CIFAR-10) and one real-world industrial dataset (i.e., Fasteners), considering both one-class and multi-class anomaly detection protocols. Experimental results demonstrate that LRGAN is comparable with state-of-the-art methods and LRGAN + outperforms these methods on all datasets.

An improved DualGAN for near-infrared image colorization

This paper focuses on the colorization problem of near-infrared (NIR) images. Traditional colorization methods of grayscale images usually depend on users’ intervention and cannot be extended to NIR image colorization due to inherent complexity, such as the same near-infrared lights emitted by objects with different colors. Furthermore, a large number of paired and labeled images, which cannot be guaranteed for the addressed problem, need to be provided during the training phase, whether for some traditional reference-based coloring methods or for CNN-based automatic coloring ones. Benefiting from the advantages of deep learning and generative adversarial networks (GANs) in the image-to-image translation, an improved DualGAN architecture is constructed to deal with the investigated problem. The developed architecture contains four blocks and any two adjacent blocks exist a direct connection channel, where convolution layers in each block enclose batch normalization and leaky ReLU nonlinearities. The adoption of dual deep learning networks is to establish the conversion translation relationship between NIR images and RGB images without paired and labeled requirements. Besides, a mixed loss function by integrating generator loss for discriminators’ training is designed to decrease the occurrence of incorrect images generated by generators. Finally, an intensive comparison analysis based on common data sets is conducted to verify superiority over leading-edge methods in qualitative and quantitative visual assessments.

Realistic generation of diffusion-weighted magnetic resonance brain images with deep generative models

We study two state of the art deep generative networks, the Introspective Variational Autoencoder and the Style-Based Generative Adversarial Network, for the generation of new diffusion-weighted magnetic resonance images. We show that high quality, diverse and realistic-looking images, as evaluated by external neuroradiologists blinded to the whole study, can be synthesized using these deep generative models. We evaluate diverse metrics with respect to quality and diversity of the generated synthetic brain images. These findings show that generative models could qualify as a method for data augmentation in the medical field, where access to large image database is in many aspects restricted.

Deep Probabilistic Canonical Correlation Analysis

We propose a deep generative framework for multi-view learning based on a probabilistic interpretation of canonical correlation analysis (CCA). The model combines a linear multi-view layer in the latent space with deep generative networks as observation models, to decompose the variability in multiple views into a shared latent representation that describes the common underlying sources of variation and a set of viewspecific components. To approximate the posterior distribution of the latent multi-view layer, an efficient variational inference procedure is developed based on the solution of probabilistic CCA. The model is then generalized to an arbitrary number of views. An empirical analysis confirms that the proposed deep multi-view model can discover subtle relationships between multiple views and recover rich representations.

Simultaneous deep generative modeling and clustering of single cell genomic data.

Recent advances in single-cell technologies, including single-cell ATAC-seq (scATAC-seq), have enabled large-scale profiling of the chromatin accessibility landscape at the single cell level. However, the characteristics of scATAC-seq data, including high sparsity and high dimensionality, have greatly complicated the computational analysis. Here, we proposed scDEC, a computational tool for single cell ATAC-seq analysis with deep generative neural networks. scDEC is built on a pair of generative adversarial networks (GANs), and is capable of learning the latent representation and inferring the cell labels, simultaneously. In a series of experiments, scDEC demonstrates superior performance over other tools in scATAC-seq analysis across multiple datasets and experimental settings. In downstream applications, we demonstrated that the generative power of scDEC helps to infer the trajectory and intermediate state of cells during differentiation and the latent features learned by scDEC can potentially reveal both biological cell types and within-cell-type variations. We also showed that it is possible to extend scDEC for the integrative analysis of multi-modal single cell data.

Adaptive sequential Monte Carlo for posterior inference and model selection among complex geological priors

SUMMARY Bayesian model selection enables comparison and ranking of conceptual subsurface models described by spatial prior models, according to the support provided by available geophysical data. Deep generative neural networks can efficiently encode such complex spatial priors, thereby, allowing for a strong model dimensionality reduction that comes at the price of enhanced non-linearity. In this setting, we explore a recent adaptive sequential Monte Carlo (ASMC) approach that builds on annealed importance sampling (AIS); a method that provides both the posterior probability density function (PDF) and the evidence (a central quantity for Bayesian model selection) through a particle approximation. Both techniques are well suited to parallel computation and rely on importance sampling over a sequence of intermediate distributions, linking the prior and the posterior PDF. Each subsequent distribution is approximated by updating the particle weights and states, compared with the previous approximation, using a small pre-defined number of Markov chain Monte Carlo (MCMC) proposal steps. Compared with AIS, the ASMC method adaptively tunes the tempering between neighboring distributions and performs resampling of particles when the variance of the particle weights becomes too large. We evaluate ASMC using two different conceptual models and associated synthetic cross-hole ground penetrating radar tomography data. For the most challenging test case, we find that the ASMC method is faster and more reliable in locating the posterior PDF than state-of-the-art adaptive MCMC. The evidence estimates are found to be robust with respect to the choice of ASMC algorithmic variables and much less sensitive to the model proposal type than MCMC. The variance of the evidence estimates are best estimated by replication of ASMC runs, while approximations based on single runs provide comparable estimates when using a sufficient number of proposal steps in approximating each intermediate distribution.

HI-GAN: A hierarchical generative adversarial network for blind denoising of real photographs

Although deep convolutional neural networks (DCNNs) and generative adversarial networks (GANs) have achieved remarkable success in image denoising, they have been facing a severe problem of the trade-off between removing noise and artifacts on the one hand, and preserving details on the other. In comparison with conventional DCNNs, GANs might be better in balancing between erasing different types of noise and recovering texture details. However, they often generate fake details and unexpected artifacts in the image owing to the instability of their discriminator during training. In this study, we propose a hierarchical generative adversarial network (HI-GAN) that adopts useful solutions for handling these serious problems of image denoising. Unlike the conventional GAN, the proposed HI-GAN comprises three main generators. The first generator tackles the problem of losing high-frequency features such as edges and texture. This generator was trained together with the discriminator to improve its ability to preserve essential details. The second generator focuses on eliminating the effect of instabilities caused by the discriminator and restoring low-frequency features in the noisy image . Both generators use different criteria to evaluate the denoising performance, and none of them outperformed the other. Then, a third generator is employed to help them cooperate more effectively and boost reconstruction performance. Moreover, to improve the effectiveness of the generators, we also propose a novel boosted residual dense UNet, which is designed to maximize information flow to pass through all convolutional layers in the network. In addition, we propose the AdaRaGAN loss function that effectively prevents the instability of the discriminator of the HI-GAN and improves the denoising performance. The experimental results of the experiments involving challenging datasets of real-world noisy images show that our proposed method is superior to other state-of-the-art denoisers in terms of quantitative metrics and visual quality. Our source codes and datasets for HI-GAN are available at https://github.com/ZeroZero19/HI-GAN.git.

Training-Free Deep Generative Networks for Compressed Sensing of Neural Action Potentials.

Energy consumption is an important issue for resource-constrained wireless neural recording applications with limited data bandwidth. Compressed sensing (CS) is a promising framework for addressing this challenge because it can compress data in an energy-efficient way. Recent work has shown that deep neural networks (DNNs) can serve as valuable models for CS of neural action potentials (APs). However, these models typically require impractically large datasets and computational resources for training, and they do not easily generalize to novel circumstances. Here, we propose a new CS framework, termed APGen, for the reconstruction of APs in a training-free manner. It consists of a deep generative network and an analysis sparse regularizer. We validate our method on two in vivo datasets. Even without any training, APGen outperformed model-based and data-driven methods in terms of reconstruction accuracy, computational efficiency, and robustness to AP overlap and misalignment. The computational efficiency of APGen and its ability to perform without training make it an ideal candidate for long-term, resource-constrained, and large-scale wireless neural recording. It may also promote the development of real-time, naturalistic brain-computer interfaces.

Density estimation using deep generative neural networks

Density estimation is one of the fundamental problems in both statistics and machine learning. In this study, we propose Roundtrip, a computational framework for general-purpose density estimation based on deep generative neural networks. Roundtrip retains the generative power of deep generative models, such as generative adversarial networks (GANs) while it also provides estimates of density values, thus supporting both data generation and density estimation. Unlike previous neural density estimators that put stringent conditions on the transformation from the latent space to the data space, Roundtrip enables the use of much more general mappings where target density is modeled by learning a manifold induced from a base density (e.g., Gaussian distribution). Roundtrip provides a statistical framework for GAN models where an explicit evaluation of density values is feasible. In numerical experiments, Roundtrip exceeds state-of-the-art performance in a diverse range of density estimation tasks.

RL-VAEGAN: Adversarial defense for reinforcement learning agents via style transfer

Reinforcement learning (RL) agents parameterized by deep neural networks have achieved great success in many domains. However, deep RL policies have been shown to be vulnerable to adversarial attacks, i.e., inputs with slight perturbations should result in a substantial agent failure. Inspired by recent advances in deep generative networks that have greatly facilitated the development of adversarial attacks, in this paper, we investigate the adversarial robustness of RL agents and propose a novel defense framework for RL based on the idea of style transfer. More precisely, our defense framework containing variational autoencoders (VAEs) and generative adversarial networks (GANs), called RL-VAEGAN, learns the distribution of the styles of the original and adversarial states, respectively, and naturally eliminates the threat of adversarial attacks for RL agents by transferring adversarial states to unperturbed legitimate one under the shared-content latent space assumption. We empirically show that our methods are effective against the state-of-the-art methods in white-box and black-box scenarios with diverse magnitudes of perturbations.

Fighting Fake News: Two Stream Network for Deepfake Detection via Learnable SRM

Benefitting from the development of deep generative networks, modern fake news generation methods called Deepfake rapidly go viral over the Internet, calling for efficient detection methods. Existing Deepfake detection methods basically use binary classification networks trained on frame-level inputs and lack leveraging temporal information in videos. Besides, the accuracy of these methods will rapidly decrease when processing low-quality data. In this work, we propose a two-stream network to detect Deepfake in video level with the capability of handling low-quality data. The proposed architecture firstly divides the input video into segments and then feeds selected frames of each segment into two streams: The first stream takes RGB information as input and tries to learn the semantic inconsistency. The second stream parallelly leverages noise features extracted by spatial rich model (SRM) filters. Additionally, our experiments found that traditional SRM filters with fixed weights contribute insignificant improvement, we thus design novel learnable SRM filters, which can better fit the noise inconsistency in tampered regions. Segmental fusion and stream fusion are conducted at last to combine the information from segments and streams. We evaluate our algorithm on the existing largest Deepfake dataset FaceForensics++ and the experimental results show that we obtain state-of-the-art performance.

Deep learning formulation of electrocardiographic imaging integrating image and signal information with data-driven regularization.

Electrocardiographic imaging (ECGI) is a promising tool to map the electrical activity of the heart non-invasively using body surface potentials (BSP). However, it is still challenging due to the mathematically ill-posed nature of the inverse problem to solve. Novel approaches leveraging progress in artificial intelligence could alleviate these difficulties. We propose a deep learning (DL) formulation of ECGI in order to learn the statistical relation between BSP and cardiac activation. The presented method is based on Conditional Variational AutoEncoders using deep generative neural networks. To quantify the accuracy of this method, we simulated activation maps and BSP data on six cardiac anatomies.We evaluated our model by training it on five different cardiac anatomies (5000 activation maps) and by testing it on a new patient anatomy over 200 activation maps. Due to the probabilistic property of our method, we predicted 10 distinct activation maps for each BSP data. The proposed method is able to generate volumetric activation maps with a good accuracy on the simulated data: the mean absolute error is 9.40 ms with 2.16 ms standard deviation on this testing set. The proposed formulation of ECGI enables to naturally include imaging information in the estimation of cardiac electrical activity from BSP. It naturally takes into account all the spatio-temporal correlations present in the data. We believe these features can help improve ECGI results.

Self-guided deep deterministic policy gradient with multi-actor

Reinforcement learning algorithms have made huge progress in recent years by leveraging the power of deep neural networks. Despite the success, deep reinforcement learning algorithms’ performance is largely dependent on the approach of exploration. Some of them engage in exploratory behavior by injecting external noise into the action space or adopting a gaussian policy. This paper presents a deep reinforcement learning algorithm without external noise called self-guided deep deterministic policy gradient with multi-actor (SDDPGM), which is the combination of deep deterministic policy gradient and generative adversarial networks (GANs). It employs the generator of GANs which trained from excellent experiences to guide the learning of the agent and makes discriminator constitute a subjective reward. Moreover, to make the learning more stable, SDDPGM applies a multi-actor mechanism that stands as a serially distinct actor based on the temporal phase of an episode. Finally, experiments show that SDDPGM is a promising deep reinforcement learning method.

Satellite image inpainting with deep generative adversarial neural networks

This work addresses the problem of recovering lost or damaged satellite image pixels (gaps) caused by sensor processing errors or by natural phenomena like cloud presence. Such errors decrease our ability to monitor regions of interest and significantly increase the average revisit time for all satellites. This paper presents a novel neural system based on conditional deep generative adversarial networks (cGAN) optimized to fill satellite imagery gaps using surrounding pixel values and static high-resolution visual priors. Experimental results show that the proposed system outperforms traditional and neural network baselines. It achieves a normalized least absolute deviations error of ( & decrease in error compared with the two baselines) and a mean squared error loss of ( & decrease in error) over the test set. The model can be deployed within a remote sensing data pipeline to reconstruct missing pixel measurements for near-real-time monitoring and inference purposes, thus empowering policymakers and users to make environmentally informed decisions.