Conditional Generative Model Research Articles

The Applicability of Conditional Generative Model Generating Groundwater Level Fluctuation Corresponding to Precipitation Pattern

The Applicability of Conditional Generative Model Generating Groundwater Level Fluctuation Corresponding to Precipitation Pattern

Modeling the Conditional Distribution of Co-Speech Upper Body Gesture Jointly Using Conditional-GAN and Unrolled-GAN

Co-speech gestures are a crucial, non-verbal modality for humans to communicate. Social agents also need this capability to be more human-like and comprehensive. This study aims to model the distribution of gestures conditioned on human speech features. Unlike previous studies that try to find injective functions that map speech to gestures, we propose a novel, conditional GAN-based generative model to not only convert speech into gestures but also to approximate the distribution of gestures conditioned on speech through parameterization. An objective evaluation and user study show that the proposed model outperformed the existing deterministic model, indicating that generative models can approximate real patterns of co-speech gestures better than the existing deterministic model. Our results suggest that it is critical to consider the nature of randomness when modeling co-speech gestures.

Generating Sea Surface Object Image Using Image-to-Image Translation

Abstract Sea objects training, the conditional adversarial networks require a large number of images to solve image-to-image translation problems. In the case of insufficient samples, it leads to network overfitting and poor training results. This project proposes a conditional adversarial generative model that retains the original background features in the absence of paired samples. The goal of this project is to reduce the deviation of the corresponding output from the original input. Firstly, the object images of different categories are labeled with color masks. Second, sea objects are generated randomly in the original background using model of this project. Finally, the generated results of this approach are compared with other approaches. The experimental results show that, compared with results from other conditional adversarial generative models, the generated object images using model of this project have the characteristics of richer texture and clearer structure.

Establishing an evaluation metric to quantify climate change image realism

With success on controlled tasks, deep generative models are being increasingly applied to humanitarian applications . In this paper, we focus on the evaluation of a conditional generative model that illustrates the consequences of climate change-induced flooding to encourage public interest and awareness on the issue. Because metrics for comparing the realism of different modes in a conditional generative model do not exist, we propose several automated and human-based methods for evaluation. To do this, we adapt several existing metrics and assess the automated metrics against gold standard human evaluation. We find that using Fréchet Inception Distance with embeddings from an intermediary Inception-v3 layer that precedes the auxiliary classifier produces results most correlated with human realism. While insufficient alone to establish a human-correlated automatic evaluation metric, we believe this work begins to bridge the gap between human and automated generative evaluation procedures, and to generate more realistic images of the future consequences of climate change.

CoSinGAN: Learning COVID-19 Infection Segmentation from a Single Radiological Image.

Computed tomography (CT) images are currently being adopted as the visual evidence for COVID-19 diagnosis in clinical practice. Automated detection of COVID-19 infection from CT images based on deep models is important for faster examination. Unfortunately, collecting large-scale training data systematically in the early stage is difficult. To address this problem, we explore the feasibility of learning deep models for lung and COVID-19 infection segmentation from a single radiological image by resorting to synthesizing diverse radiological images. Specifically, we propose a novel conditional generative model, called CoSinGAN, which can be learned from a single radiological image with a given condition, i.e., the annotation mask of the lungs and infected regions. Our CoSinGAN is able to capture the conditional distribution of the single radiological image, and further synthesize high-resolution (512 × 512) and diverse radiological images that match the input conditions precisely. We evaluate the efficacy of CoSinGAN in learning lung and infection segmentation from very few radiological images by performing 5-fold cross validation on COVID-19-CT-Seg dataset (20 CT cases) and an independent testing on the MosMed dataset (50 CT cases). Both 2D U-Net and 3D U-Net, learned from four CT slices by using our CoSinGAN, have achieved notable infection segmentation performance, surpassing the COVID-19-CT-Seg-Benchmark, i.e., the counterparts trained on an average of 704 CT slices, by a large margin. Such results strongly confirm that our method has the potential to learn COVID-19 infection segmentation from few radiological images in the early stage of COVID-19 pandemic.

Application of conditional generative model for sonic log estimation considering measurement uncertainty

Well-log data is a cost-effective means to characterize the petrophysical properties of a geological formation. Among the data, compressional- and shear-slowness (DTC and DTS, respectively) are the most reliable and have been widely applied in the interpretations. However, the availability of DTS data tends to be limited because of its high acquisition cost. This study proposes a method to reproduce or reconstruct the DTS data using other well-log data, such as gamma ray, neutron porosity, bulk density, and DTC. The developed method is based on the conditional variational autoencoder (CVAE) and effectively considers uncertainty associated with the variability of the measured data. The performance of this developed method is validated by applying the well-log data acquired from Satyr-5 and Callihoe-1 wells in the Northern Carnarvon Basin, Western Australia, and the prediction accuracy of the developed method is compared to recently developed data-driven methods (i.e., long short-term memory (LSTM) and bidirectional LSTM (bi-LSTM)). The results reveal that the developed method produces a better DTS estimation than LSTM and bi-LSTM. Furthermore, the effectiveness of the proposed method remains unaltered regardless of whether the data contain a specific trend over the depth or amount of training data are insufficient. As a further application of the developed method, an uncertainty relative to DTS estimation is quantitatively obtained from Monte-Carlo estimation, which uses a trained probability model of the developed method. Sensitivity analysis reveals the high effectiveness of DTC in improving the performance of the CVAE method. From our results, we can conclude that the proposed CVAE-based method is an effective tool for improving the efficiency and accuracy of DTS estimation.

Discriminative and generative models for anatomical shape analysis on point clouds with deep neural networks.

We introduce deep neural networks for the analysis of anatomical shapes that learn a low-dimensional shape representation from the given task, instead of relying on hand-engineered representations. Our framework is modular and consists of several computing blocks that perform fundamental shape processing tasks. The networks operate on unordered point clouds and provide invariance to similarity transformations, avoiding the need to identify point correspondences between shapes. Based on the framework, we assemble a discriminative model for disease classification and age regression, as well as a generative model for the accruate reconstruction of shapes. In particular, we propose a conditional generative model, where the condition vector provides a mechanism to control the generative process. For instance, it enables to assess shape variations specific to a particular diagnosis, when passing it as side information. Next to working on single shapes, we introduce an extension for the joint analysis of multiple anatomical structures, where the simultaneous modeling of multiple structures can lead to a more compact encoding and a better understanding of disorders. We demonstrate the advantages of our framework in comprehensive experiments on real and synthetic data. The key insights are that (i) learning a shape representation specific to the given task yields higher performance than alternative shape descriptors, (ii) multi-structure analysis is both more efficient and more accurate than single-structure analysis, and (iii) point clouds generated by our model capture morphological differences associated to Alzheimer's disease, to the point that they can be used to train a discriminative model for disease classification. Our framework naturally scales to the analysis of large datasets, giving it the potential to learn characteristic variations in large populations.

Conditional generative adversarial network model for simulating intensity measures of aftershocks

Earthquake disaster records demonstrate that the influence of aftershocks (ASs) needs to be considered in structural seismic design and performance assessment, owing to the additional damage caused by them. To study the failure mechanism of a structure undergoing a sequence earthquake (i.e., mainshock–aftershock (MS–AS)), a clear understanding of the correlation between the intensity measures (IMs) of an MS–AS is important. However, the above correlation has not yet been systematically studied. Previously, some researchers have investigated the correlation between the individual IMs of an MS–AS separately by a Copula function based on an assumption that all the IMs of the MS (AS) are independent. However, the IMs are actually related, owing to their definition. Concurrently, deep learning (particularly the generation models) can be used to reveal the potential connections between data without any assumptions. Moreover, these models can present the conditional probability distribution of data by adding specific conditions. This study aims to build a conditional generative adversarial network (CGAN) model to simulate the IMs of an AS of an MS–AS, which can not only reflect the correlation between the corresponding IMs of the MS–AS but also that between the IMs used. The performance of this model is ascertained based on residual analysis, and the IM AS predictability is tested using real records. This study selects 972 MS–AS ground motions from the Next Generation Attenuation-West2 (NGA-West2) database, randomly dividing them into 80% and 20% and using as the training set and testing set, respectively. The results show that the CGAN model can predict the IMs of ASs with good accuracy. Moreover, the ground motion prediction equation (GMPE) by Abrahamson et al. (ASK14) is selected to compare with the CGAN model, and it is exhibited that the CGAN model matches the as-recorded IMs of ASs better that the former. All these results demonstrate that the proposed CGAN model is a promising and reliable approach for IM prediction of an AS.

Object Synthesis by Learning Part Geometry with Surface and Volumetric Representations

We propose a conditional generative model, named Part Geometry Network (PG-Net), which synthesizes realistic objects and can be used as a robust feature descriptor for object reconstruction and classification. Surface and volumetric representations of objects have complementary properties of three-dimensional objects. Combining these modalities is more informative than using one modality alone. Therefore, PG-Net utilizes complementary properties of surface and volumetric representations by estimating curvature, surface area, and occupancy in voxel grids of objects with a single decoder as a multi-task learning. Objects are combinations of multiple parts, and therefore part geometry (PG) is essential to synthesize each part of the objects. PG-Net employs a part identifier to learn the part geometry. Additionally, we augmented a dataset by interpolating individual functional parts such as wings of an airplane, which helps learning part geometry and finding local/global minima of PG-Net. To demonstrate the capability of learning object representations of PG-Net, we performed object reconstruction and classification tasks on two standard large-scale datasets. PG-Net outperformed the state-of-the-art methods in object synthesis, classification, and reconstruction in a large margin.

Noise-free quantitative phase imaging in Gabor holography with conditional generative adversarial network.

This paper shows that deep learning can eliminate the superimposed twin-image noise in phase images of Gabor holographic setup. This is achieved by the conditional generative adversarial model (C-GAN), trained by input-output pairs of noisy phase images obtained from synthetic Gabor holography and the corresponding quantitative noise-free contrast-phase image obtained by the off-axis digital holography. To train the model, Gabor holograms are generated from digital off-axis holograms with spatial shifting of the real image and twin image in the frequency domain and then adding them with the DC term in the spatial domain. Finally, the digital propagation of the Gabor hologram with Fresnel approximation generates a super-imposed phase image for the C-GAN model input. Two models were trained: a human red blood cell model and an elliptical cancer cell model. Following the training, several quantitative analyses were conducted on the bio-chemical properties and similarity between actual noise-free phase images and the model output. Surprisingly, it is discovered that our model can recover other elliptical cell lines that were not observed during the training. Additionally, some misalignments can also be compensated with the trained model. Particularly, if the reconstruction distance is somewhat incorrect, this model can still retrieve in-focus images.

Establishing an evaluation metric to quantify climate change image realism * *33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada.

With success on controlled tasks, deep generative models are being increasingly applied to humanitarian applications (Nie et al 2017 Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (Berlin: Springer) pp 417–25, Yanardag et al 2017 Deep Empathy). In this paper, we focus on the evaluation of a conditional generative model that illustrates the consequences of climate change-induced flooding to encourage public interest and awareness on the issue. Because metrics for comparing the realism of different modes in a conditional generative model do not exist, we propose several automated and human-based methods for evaluation. To do this, we adapt several existing metrics and assess the automated metrics against gold standard human evaluation. We find that using Fréchet Inception Distance with embeddings from an intermediary Inception-v3 layer that precedes the auxiliary classifier produces results most correlated with human realism. While insufficient alone to establish a human-correlated automatic evaluation metric, we believe this work begins to bridge the gap between human and automated generative evaluation procedures, and to generate more realistic images of the future consequences of climate change.

Fast Underwater Image Enhancement for Improved Visual Perception

In this letter, we present a conditional generative adversarial network-based model for real-time underwater image enhancement. To supervise the adversarial training, we formulate an objective function that evaluates the perceptual image quality based on its global content, color, local texture, and style information. We also present EUVP, a large-scale dataset of a paired and an unpaired collection of underwater images (of ‘poor’ and ‘good’ quality) that are captured using seven different cameras over various visibility conditions during oceanic explorations and human-robot collaborative experiments. In addition, we perform several qualitative and quantitative evaluations which suggest that the proposed model can learn to enhance underwater image quality from both paired and unpaired training. More importantly, the enhanced images provide improved performances of standard models for underwater object detection, human pose estimation, and saliency prediction. These results validate that it is suitable for real-time preprocessing in the autonomy pipeline by visually-guided underwater robots. The model and associated training pipelines are available at https://github.com/xahidbuffon/funie-gan .

Real Time Trajectory Prediction Using Deep Conditional Generative Models

Data driven methods for time series forecasting that quantify uncertainty open new important possibilities for robot tasks with hard real time constraints, allowing the robot system to make decisions that trade off between reaction time and accuracy in the predictions. Despite the recent advances in deep learning, it is still challenging to make long term accurate predictions with the low latency required by real time robotic systems. In this letter, we propose a deep conditional generative model for trajectory prediction that is learned from a data set of collected trajectories. Our method uses encoder and decoder deep networks that map complete or partial trajectories to a Gaussian distributed latent space and back, allowing for fast inference of the future values of a trajectory given previous observations. The encoder and decoder networks are trained using stochastic gradient variational Bayes. In the experiments, we show that our model provides more accurate long term predictions with a lower latency than popular models for trajectory forecasting like recurrent neural networks or physical models based on differential equations. Finally, we test our proposed approach in a robot table tennis scenario to evaluate the performance of the proposed method in a robotic task with hard real time constraints.

A novel wind turbine data imputation method with multiple optimizations based on GANs

In the rising research and applications of data-driven technologies in mechanical systems, data missing has always been a serious problem. The problem of data missing on a large scope has brought grave challenges to the operation and maintenance of the machineries, such as wind turbines (WTs). In this paper, a WT data imputation method with multiple optimizations based on generative adversarial networks (GANs) is proposed. First, to tackle the problem of data missing in large-scale WTs, a conditional GANs-based deep learning generative model is designed according to data features. Second, the permutation of the training data is optimized, so that the convolutional kernel can be better applied. The optimization problem is creatively transformed to a travelling salesman problem (TSP), and two optimization functions are proposed based on data features. Then, the relationship between the training data and the convolutional kernel is studied, and two restrictions are put forward to make the imputation model more effective. Finally, four data imputation experiments and two optimization experiments are carried out using real WT data. The experiment results verify the effectiveness of the proposed method.

VSA-CGAN: An Intelligent Generation Model for Deep Learning Sample Database Construction

In order to solve the problem of model accuracy reduction caused by the difficulty of obtaining specific training samples or the insufficient number of samples in the application of existing object detection and recognition model based on deep learning, this article proposes a conditional generative adversarial network model (VSA-CGAN), which integrates the self-attention mechanism of visual perception to optimize the inference of object attention feature maps, so as to learn the global information of the image and the detailed features of the object. It is designed to add conditional features in the generator and the discriminator, associate the specific dimensions of the data with the semantic features, and explicitly indicate the model to generate the corresponding object signature category information, so as to generate the feature representation of the image which is more suitable for the distribution of the original data. The model in this article has completed numerical experiments on several general standard data sets, and compared with several mainstream generative adversarial network models in image data augmentation performance. The experimental results show that the generation model in this article has excellent object simulation ability and strong application prospects.

Conditional Generative Denoising Autoencoder.

We present a generative denoising autoencoder model that has an embedded data classifier in its architecture in order to take advantage of class-based discriminating features and produce better data samples. The proposed model is a conditional generative model and is sampled with a Markov chain Monte Carlo (MCMC) process according to a label that denotes the desired (or undesired) class or classes. In this sense, any chosen predefined class or characteristic may have a positive or negative effect on the image generation process, meaning that it can be instructed to be present or absent from the generated sample. We argue that allowing discriminative information in the form of feature detectors to be present in the latent representation of the autoencoder can be generally beneficial. This technique is an alternative approach to variational autoencoders (VAEs) that enforce a prior on the latent distribution. We further claim that supervised learning may be generally able to serve unsupervised learning through an interaction between the two paradigms. However, the extreme majority of research done on the interaction of the two learning regimes has the goal of using unsupervised learning to improve supervised learning. In this article, we explore the two learning paradigms' interaction in the opposite direction.

Segmentation and quantification of infarction without contrast agents via spatiotemporal generative adversarial learning.

Accurate and simultaneous segmentation and full quantification (all indices are required in a clinical assessment) of the myocardial infarction (MI) area are crucial for early diagnosis and surgical planning. Current clinical methods remain subject to potential high-risk, nonreproducibility and time-consumption issues. In this study, a deep spatiotemporal adversarial network (DSTGAN) is proposed as a contrast-free, stable and automatic clinical tool to simultaneously segment and quantify MIs directly from the cine MR image. The DSTGAN is implemented using a conditional generative model, which conditions the distributions of the objective cine MR image to directly optimize the generalized error of the mapping between the input and the output. The method consists of the following: (1) A multi-level and multi-scale spatiotemporal variation encoder learns a coarse to fine hierarchical feature to effectively encode the MI-specific morphological and kinematic abnormality structures, which vary for different spatial locations and time periods. (2) The top-down and cross-task generators learn the shared representations between segmentation and quantification to use the commonalities and differences between the two related tasks and enhance the generator preference. (3) Three inter-/intra-tasks to label the relatedness discriminators are iteratively imposed on the encoder and generator to detect and correct the inconsistencies in the label relatedness between and within tasks via adversarial learning. Our proposed method yields a pixel classification accuracy of 96.98%, and the mean absolute error of the MI centroid is 0.96mm from 165 clinical subjects. These results indicate the potential of our proposed method in aiding standardized MI assessments.

DUAL-GLOW: Conditional Flow-Based Generative Model for Modality Transfer.

Positron emission tomography (PET) imaging is an imaging modality for diagnosing a number of neurological diseases. In contrast to Magnetic Resonance Imaging (MRI), PET is costly and involves injecting a radioactive substance into the patient. Motivated by developments in modality transfer in vision, we study the generation of certain types of PET images from MRI data. We derive new flow-based generative models which we show perform well in this small sample size regime (much smaller than dataset sizes available in standard vision tasks). Our formulation, DUAL-GLOW, is based on two invertible networks and a relation network that maps the latent spaces to each other. We discuss how given the prior distribution, learning the conditional distribution of PET given the MRI image reduces to obtaining the conditional distribution between the two latent codes w.r.t. the two image types. We also extend our framework to leverage "side" information (or attributes) when available. By controlling the PET generation through "conditioning" on age, our model is also able to capture brain FDG-PET (hypometabolism) changes, as a function of age. We present experiments on the Alzheimers Disease Neuroimaging Initiative (ADNI) dataset with 826 subjects, and obtain good performance in PET image synthesis, qualitatively and quantitatively better than recent works.

Conditional Recurrent Flow: Conditional Generation of Longitudinal Samples with Applications to Neuroimaging.

We develop a conditional generative model for longitudinal image datasets based on sequential invertible neural networks. Longitudinal image acquisitions are common in various scientific and biomedical studies where often each image sequence sample may also come together with various secondary (fixed or temporally dependent) measurements. The key goal is not only to estimate the parameters of a deep generative model for the given longitudinal data, but also to enable evaluation of how the temporal course of the generated longitudinal samples are influenced as a function of induced changes in the (secondary) temporal measurements (or events). Our proposed formulation incorporates recurrent subnetworks and temporal context gating, which provide a smooth transition in a temporal sequence of generated data that can be easily informed or modulated by secondary temporal conditioning variables. We show that the formulation works well despite the smaller sample sizes common in these applications. Our model is validated on two video datasets and a longitudinal Alzheimer's disease (AD) dataset for both quantitative and qualitative evaluations of the generated samples. Further, using our generated longitudinal image samples, we show that we can capture the pathological progressions in the brain that turn out to be consistent with the existing literature, and could facilitate various types of downstream statistical analysis.

Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data

Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the governing equations of the physical model in the loss/likelihood functions. The resulting physics-constrained, deep learning models are trained without any labeled data (e.g. employing only input data) and provide comparable predictive responses with data-driven models while obeying the constraints of the problem at hand. This work employs a convolutional encoder-decoder neural network approach as well as a conditional flow-based generative model for the solution of PDEs, surrogate model construction, and uncertainty quantification tasks. The methodology is posed as a minimization problem of the reverse Kullback-Leibler (KL) divergence between the model predictive density and the reference conditional density, where the later is defined as the Boltzmann-Gibbs distribution at a given inverse temperature with the underlying potential relating to the PDE system of interest. The generalization capability of these models to out-of-distribution input is considered. Quantification and interpretation of the predictive uncertainty is provided for a number of problems.