Deep Generative Models Research Articles

Adversarial Variational Autoencoders to Extend and Improve Generative Model

Generative artificial intelligence (GenAI) has been advancing with many notable achievements like ChatGPT and Bard. The deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to the strong role of deep neural networks (DNNs) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims at synaptic plasticity of the human neuron network, fosters the generation ability of DGM, which produces surprising results with the support of statistical flexibility. Two popular approaches in DGM are the variational autoencoder (VAE) and generative adversarial network (GAN). Both VAE and GAN have their own strong points although they share and imply the underlying theory of statistics as well as significant complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. This research unifies VAE and GAN into a consistent and consolidated model called the adversarial variational autoencoder (AVA) in which the VAE and GAN complement each other; for instance, the VAE is a good data generator by encoding data via the excellent ideology of Kullback–Leibler divergence and the GAN is a significantly important method to assess the reliability of data as to whether it is real or fake. In other words, the AVA aims to improve the accuracy of generative models; besides, the AVA extends the function of simple generative models. In methodology, this research focuses on the combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.

Implications of data topology for deep generative models

Many deep generative models, such as variational autoencoders (VAEs) and generative adversarial networks (GANs), learn an immersion mapping from a standard normal distribution in a low-dimensional latent space into a higher-dimensional data space. As such, these mappings are only capable of producing simple data topologies, i.e., those equivalent to an immersion of Euclidean space. In this work, we demonstrate the limitations of such latent space generative models when trained on data distributions with non-trivial topologies. We do this by training these models on synthetic image datasets with known topologies (spheres, torii, etc.). We then show how this results in failures of both data generation as well as data interpolation. Next, we compare this behavior to two classes of deep generative models that in principle allow for more complex data topologies. First, we look at chart autoencoders (CAEs), which construct a smooth data manifold from multiple latent space chart mappings. Second, we explore score-based models, e.g., denoising diffusion probabilistic models, which estimate gradients of the data distribution without resorting to an explicit mapping to a latent space. Our results show that these models do demonstrate improved ability over latent space models in modeling data distributions with complex topologies, however, challenges still remain.

Bayesian inverse inference of material properties from microstructure images

In this paper, we introduce a Bayesian framework designed for inverse inference, aiming to predict material properties/process parameters from microstructure images. The integration of Bayesian inference techniques with deep generative models establishes a robust tool for applications in materials science, particularly in material characterization and property control. This integration provides a novel approach to clarifying the reliability of predictions. The application of this framework to a sample problem involving the prediction of material properties from artificial dual-phase steel microstructures demonstrates its capability to estimate these properties while accounting for prediction uncertainties. Moreover, even in comparison to conventional regression methods in terms of point estimation, the proposed framework exhibits superior accuracy in prediction. These results clearly illustrate that the framework presented in this paper constitutes a powerful tool for achieving efficient material design.

LineageVAE: reconstructing historical cell states and transcriptomes toward unobserved progenitors.

Single-cell RNA sequencing (scRNA-seq) enables comprehensive characterization of the cell state. However, its destructive nature prohibits measuring gene expression changes during dynamic processes such as embryogenesis or cell state divergence due to injury or disease. Although recent studies integrating scRNA-seq with lineage tracing have provided clonal insights between progenitor and mature cells, challenges remain. Because of their experimental nature, observations are sparse, and cells observed in the early state are not the exact progenitors of cells observed at later time points. To overcome these limitations, we developed LineageVAE, a novel computational methodology that utilizes deep learning based on the property that cells sharing barcodes have identical progenitors. LineageVAE is a deep generative model that transforms scRNA-seq observations with identical lineage barcodes into sequential trajectories toward a common progenitor in a latent cell state space. This method enables the reconstruction of unobservable cell state transitions, historical transcriptomes, and regulatory dynamics at a single-cell resolution. Applied to hematopoiesis and reprogrammed fibroblast datasets, LineageVAE demonstrated its ability to restore backward cell state transitions and infer progenitor heterogeneity and transcription factor activity along differentiation trajectories. The LineageVAE model was implemented in Python using the PyTorch deep learning library. The code is available on GitHub at https://github.com/LzrRacer/LineageVAE/.

OpenECAD: An efficient visual language model for editable 3D-CAD design

Computer-aided design (CAD) tools are utilized in the manufacturing industry for modeling everything from cups to spacecraft. These programs are complex to use and typically require years of training and experience to master. Structured and well-constrained 2D sketches and 3D constructions are crucial components of CAD modeling. A well-executed CAD model can be seamlessly integrated into the manufacturing process, thereby enhancing production efficiency. Deep generative models of 3D shapes and 3D object reconstruction models have garnered significant research interest. However, most of these models produce discrete forms of 3D objects that are not editable. Moreover, the few models based on CAD operations often have substantial input restrictions. In this work, we fine-tuned pre-trained models to create OpenECAD models (0.55B, 0.89B, 2.4B and 3.1B), leveraging the visual, logical, coding, and general capabilities of visual language models. OpenECAD models can process images of 3D designs as input and generate highly structured 2D sketches and 3D construction commands, ensuring that the designs are editable. These outputs can be directly used with existing CAD tools’ APIs to generate project files. To train our network, we created a series of OpenECAD datasets. These datasets are derived from existing public CAD datasets, adjusted and augmented to meet the specific requirements of vision language model (VLM) training. Additionally, we have introduced an approach that utilizes dependency relationships to define and generate sketches, further enriching the content and functionality of the datasets.

Separating group- and individual-level brain signatures in the newborn functional connectome: A deep learning approach

Recent studies indicate that differences in cognition among individuals may be partially attributed to unique brain wiring patterns. While functional connectivity (FC)-based fingerprinting has demonstrated high accuracy in identifying adults, early studies on neonates suggest that individualized FC signatures are absent. We posit that individual uniqueness is present in neonatal FC data and that conventional linear models fail to capture the rapid developmental trajectories characteristic of newborn brains. To explore this hypothesis, we employed a deep generative model, known as a variational autoencoder (VAE), leveraging two extensive public datasets: one comprising resting-state functional MRI (rs-fMRI) scans from 100 adults and the other from 464 neonates. VAE models trained on rs-fMRI from both adults and newborns produced superior age prediction performance (with r between predicted- and actual age ∼ 0.7) and individual identification accuracy (∼45 %) compared to models trained solely on adult or neonatal data. The VAE model also showed significantly higher individual identification accuracy than linear models (=10∼30 %). Importantly, the VAE differentiated connections reflecting age-related changes from those indicative of individual uniqueness, a distinction not possible with linear models. Moreover, we derived 20 latent variables, each corresponding to distinct patterns of cortical functional network (CFNs). These CFNs varied in their representation of brain maturation and individual signatures; notably, certain CFNs that failed to capture neurodevelopmental traits, in fact, exhibited individual signatures. CFNs associated with neonatal neurodevelopment predominantly encompassed unimodal regions such as visual and sensorimotor areas, whereas those linked to individual uniqueness spanned multimodal and transmodal brain regions. The VAE's capacity to extract features from rs-fMRI data beyond the capabilities of linear models positions it as a valuable tool for delineating cognitive traits inherent in rs-fMRI and exploring individualized imaging phenotypes.

Experience: A Comparative Analysis of Multivariate Time-Series Generative Models: A Case Study on Human Activity Data

Human activity recognition (HAR) is an active research field that has seen great success in recent years due to advances in sensory data collection methods and activity recognition systems. Deep artificial intelligence (AI) models have contributed to the success of HAR systems lately although still suffering from limitations such as data scarcity, the high costs of labelling data instances and datasets’ imbalance and bias. The temporal nature of human activity data, represented as time series data, impose an additional challenge to using AI models in HAR because most state-of-the-art models do not account for the time component of the data instances. These limitations have inspired the time-series research community to design generative models for sequential data but very little work has been done to evaluate the quality of such models. In this work, we conduct a comparative quality analysis of three generative models for time-series data, using a case study in which we aim to generate sensory human activity data from a seed public dataset. Additionally, we adapt and clearly explain four evaluation methods of synthetic time-series data from the literature and apply them to assess the quality of the synthetic activity data we generate. We show experimentally that high quality human activity data can be generated using deep generative models, and the synthetic data can thus be used in HAR systems to augment real activity data. We also demonstrate that the chosen evaluation methods effectively ensure that the generated data meets the essential quality benchmarks of realism, diversity, coherence and utility. Our findings suggest that using deep generative models to produce synthetic human activity data can potentially address challenges related to data scarcity, biases, and expensive labeling. This holds promise for enhancing the efficiency and reliability of HAR systems.

Generating Stochastic Structural Planes Using Statistical Models and Generative Deep Learning Models: A Comparative Investigation

Structural planes are one of the key factors controlling the stability of rock masses. A comprehensive understanding of the spatial distribution characteristics of structural planes is essential for accurately identifying key blocks, analyzing rock mass stability, and addressing various rock engineering challenges. This study compares the effectiveness of four stochastic structural plane generation methods—the Monte Carlo method, the Copula-based method, generative adversarial networks (GAN), and denoised diffusion models (DDPM)—in generating stochastic structural planes and capturing potential correlations between structural plane parameters. The Monte Carlo method employs the mean and variance of three parameters of the measured factual structural planes to generate data that follow the same distributions. The other three methods take the entire set of measured factual structural planes as the overall input to generate structural planes that exhibit the same probability distributions. Five sets of structural planes on four rock slopes in Norway are examined as an example. The validation and analysis were performed using histogram comparison, data feature comparison, scatter plot comparison, and linear regression analysis. The results show that the Monte Carlo method fails to capture the potential correlation between the dip direction and dip angle despite the best fit to the measured factual structural planes. The Copula-based method performs better with smaller datasets, and GAN and DDPM are better at capturing the correlation of measured factual structural planes in the case of large datasets. Therefore, in the case of a limited number of measured structural planes, it is advisable to employ the Copula-based method. In scenarios where the dataset is extensive, the deep generative model is recommended due to its ability to capture complex data structures. The results of this study can be utilized as a valuable point of reference for the accurate generation of stochastic structural planes within rock masses.

A deep generative model for selecting representative periods in renewable energy-integrated power systems

The extensive integration of renewable energies into power systems has led to a challenging and computationally demanding scenario for the system planning. This is due to the increased number of time series involved and the greater complexity of time series data integrated into power systems. In this paper, a new deep learning-based time aggregation method is proposed for selecting representative periods for renewable energy-integrated power system datasets where multiple variable energy resources are present. Relying on the capabilities of deep generative models, especially Generative Adversarial Network (GAN), the proposed method selects a representative period, including one or more representative days, obtained from multiple time series with spatio-temporal correlations among them. The proposed approach contains the Long Short-Term Memory (LSTM) Network in both generator and discriminator parts. Also, it includes a loss term, specifically designed for clustering tasks, in the minimax objective of the vanilla GAN loss function to enhance the clustering performance in the latent space. Furthermore, the learning rate of the proposed model, as the most important parameter of the learning algorithm, is adaptively fine-tuned during the training process, based on the training error, to enhance its learning performance. Two real-world test cases with various datasets and time series are used for data-based and model-based evaluations. Results obtained on these two real-world test cases confirm the superiority of the proposed model with respect to the state-of-the-art conventional and deep learning-based models, obtaining the performance improvement in the range of [38.55 %, 46.14 %] in terms of Mean Absolute Percentage Error (MAPE), in the range of [45.75 %, 70.16 %] in terms of Root Mean Squared Error (RMSE), and in the range of [65.81 %, 74.58 %] in terms of Mean Absolute Error (MAE). The proposed model can significantly enhance the tractability and scalability of the planning problem of renewable energy-integrated power systems, facilitating renewable energy integration into power systems which is a key issue for net zero transitioning of communities.

Exploiting the capacity of deep networks only at the training stage for nonlinear black-box system identification

To benefit from the modeling capacity of deep models in system identification without worrying about inference time, this study presents a novel training strategy that uses deep models only during the training stage. For this purpose, two separate models with different structures and goals are employed. The first one is a deep generative model aiming at modeling the distribution of system output(s), called the teacher model, and the second one is a shallow basis function model, named the student model, fed by system input(s) to predict the system output(s). That means these isolated paths must reach the same ultimate target. As deep models show a great performance in modeling highly nonlinear systems, aligning the representation space learned by these two models makes the student model inherit the teacher model’s approximation power. The proposed objective function consists of the objective of each student and teacher model, adding up with a distance penalty between the learned latent representations. The simulation results on three nonlinear benchmarks show a comparative performance with examined deep architectures applied on the same benchmarks. Algorithmic transparency and structure efficiency are also achieved as byproducts.

Synthetic lidar point cloud generation using deep generative models for improved driving scene object recognition

The imbalanced distribution of different object categories poses a challenge for training accurate object recognition models in driving scenes. Supervised machine learning models trained on imbalanced data are biased and easily overfit the majority classes, such as vehicles and pedestrians, which appear more frequently in driving scenes. We propose a novel data augmentation approach for object recognition in lidar point cloud of driving scenes, which leverages probabilistic generative models to produce synthetic point clouds for the minority classes and complement the original imbalanced dataset. We evaluate five generative models based on different statistical principles, including Gaussian mixture model, variational autoencoder, generative adversarial network, adversarial autoencoder and the diffusion model. Experiments with a real-world autonomous driving dataset show that the synthetic point clouds generated for the minority classes by the Latent Generative Adversarial Network result in significant improvement of object recognition performance for both minority and majority classes. The codes are available at https://github.com/AAAALEX-XIANG/Synthetic-Lidar-Generation.

Application progress of deep generative models in de novo drug design.

The deep molecular generative model has recently become a research hotspot in pharmacy. This paper analyzes a large number of recent reports and reviews these models. In the central part of this paper, four compound databases and two molecular representation methods are compared. Five model architectures and applications for deep molecular generative models are emphatically introduced. Three evaluation metrics for model evaluation are listed. Finally, the limitations and challenges in this field are discussed to provide a reference and basis for developing and researching new models published in future.

Tissue characterization at an enhanced resolution across spatial omics platforms with deep generative model

Recent advances in spatial omics have expanded the spectrum of profiled molecular categories beyond transcriptomics. However, many of these technologies are constrained by limited spatial resolution, hindering our ability to deeply characterize intricate tissue architectures. Existing computational methods primarily focus on the resolution enhancement of transcriptomics data, lacking the adaptability to address the emerging spatial omics technologies that profile various omics types. Here, we introduce soScope, a unified generative framework designed to enhance data quality and spatial resolution for molecular profiles obtained from diverse spatial technologies. soScope aggregates multimodal tissue information from omics, spatial relations and images, and jointly infers omics profiles at enhanced resolutions with omics-specific modeling through distribution priors. With comprehensive evaluations on diverse spatial omics platforms, including Visium, Xenium, spatial-CUT&Tag, and slide-DNA/RNA-seq, soScope improves performances in identifying biologically meaningful intestine and kidney architectures, revealing embryonic heart structure that cannot be resolved at the original resolution and correcting sample and technical biases arising from sequencing and sample processing. Furthermore, soScope extends to spatial multiomics technology spatial-CITE-seq and spatial ATAC-RNA-seq, leveraging cross-omics reference for simultaneous multiomics enhancement. soScope provides a versatile tool to improve the utilization of continually expanding spatial omics technologies and resources.

Deep Generative Model and Its Applications in Efficient Wireless Network Management: A Tutorial and Case Study

Deep Generative Model and Its Applications in Efficient Wireless Network Management: A Tutorial and Case Study

Velocity field reconstruction of mixing flow in T-junctions based on particle image database using deep generative models

Velocity field reconstruction of mixing flow in T-junctions based on particle image database using deep generative models

Deep Generative Modeling Reshapes Compression and Transmission: From Efficiency to Resiliency

Deep Generative Modeling Reshapes Compression and Transmission: From Efficiency to Resiliency

CryoDRGN-ET: deep reconstructing generative networks for visualizing dynamic biomolecules inside cells.

Advances in cryo-electron tomography (cryo-ET) have produced new opportunities to visualize the structures of dynamic macromolecules in native cellular environments. While cryo-ET can reveal structures at molecular resolution, image processing algorithms remain a bottleneck in resolving the heterogeneity of biomolecular structures in situ. Here, we introduce cryoDRGN-ET for heterogeneous reconstruction of cryo-ET subtomograms. CryoDRGN-ET learns a deep generative model of three-dimensional density maps directly from subtomogram tilt-series images and can capture states diverse in both composition and conformation. We validate this approach by recovering the known translational states in Mycoplasma pneumoniae ribosomes in situ. We then perform cryo-ET on cryogenic focused ion beam-milled Saccharomyces cerevisiae cells. CryoDRGN-ET reveals the structural landscape of S. cerevisiae ribosomes during translation and captures continuous motions of fatty acid synthase complexes inside cells. This method is openly available in the cryoDRGN software.

Deep-learning-based design of synthetic orthologs of SH3 signaling domains

Evolution-based deep generative models represent an exciting direction in understanding and designing proteins. An open question is whether such models can learn specialized functional constraints that control fitness in specific biological contexts. Here, we examine the ability of generative models to produce synthetic versions of Src-homology 3 (SH3) domains that mediate signaling in the Sho1 osmotic stress response pathway of yeast. We show that a variational autoencoder (VAE) model produces artificial sequences that experimentally recapitulate the function of natural SH3 domains. More generally, the model organizes all fungal SH3 domains such that locality in the model latent space (but not simply locality in sequence space) enriches the design of synthetic orthologs and exposes non-obvious amino acid constraints distributed near and far from the SH3 ligand-binding site. The ability of generative models to design ortholog-like functions invivo opens new avenues for engineering protein function in specific cellular contexts and environments.

A Deep Generative Model for Multi-Ship Trajectory Forecasting With Interaction Modeling

Abstract Multi-agent modeling is a challenging issue in intelligent systems, which is further compounded by heavy and complex traffic in maritime contexts. Trajectory forecasting can enhance operation safety. Nonetheless, effectively modeling interactions among vessels poses a significant difficulty. Toward this end, we propose a conditional variational autoencoder approach to ship trajectory prediction in a dynamic and multi-modal encounter situation. Leveraging a shared recurrent neural network architecture and attention mechanism, our method aggregates vessel trajectory data, enabling the model to learn and encapsulate meaningful encounter information across active vessels. We utilize automatic identification system data from the Oslofjord region to validate our approach. Through comprehensive experiments conducted on a four-ship encounter dataset, our proposed model demonstrates promising performance, by outperforming the benchmark models. Furthermore, we analyze the prediction model in a wide array of dimensions, showcasing its proficiency in complex ship behaviors learning, modeling ship interaction, and approximating actual trajectories.

ScCross: a deep generative model for unifying single-cell multi-omics with seamless integration, cross-modal generation, and in silico exploration

Single-cell multi-omics data reveal complex cellular states, providing significant insights into cellular dynamics and disease. Yet, integration of multi-omics data presents challenges. Some modalities have not reached the robustness or clarity of established transcriptomics. Coupled with data scarcity for less established modalities and integration intricacies, these challenges limit our ability to maximize single-cell omics benefits. We introduce scCross, a tool leveraging variational autoencoders, generative adversarial networks, and the mutual nearest neighbors (MNN) technique for modality alignment. By enabling single-cell cross-modal data generation, multi-omics data simulation, and in silico cellular perturbations, scCross enhances the utility of single-cell multi-omics studies.