Variational Autoencoder Architecture Research Articles

BERTIVITS: The Posterior Encoder Fusion of Pre-Trained Models and Residual Skip Connections for End-to-End Speech Synthesis

Enhancing the naturalness and rhythmicity of generated audio in end-to-end speech synthesis is crucial. The current state-of-the-art (SOTA) model, VITS, utilizes a conditional variational autoencoder architecture. However, it faces challenges, such as limited robustness, due to training solely on text and spectrum data from the training set. Particularly, the posterior encoder struggles with mid- and high-frequency feature extraction, impacting waveform reconstruction. Existing efforts mainly focus on prior encoder enhancements or alignment algorithms, neglecting improvements to spectrum feature extraction. In response, we propose BERTIVITS, a novel model integrating BERT into VITS. Our model features a redesigned posterior encoder with residual connections and utilizes pre-trained models to enhance spectrum feature extraction. Compared to VITS, BERTIVITS shows significant subjective MOS score improvements (0.16 in English, 0.36 in Chinese) and objective Mel-Cepstral coefficient reductions (0.52 in English, 0.49 in Chinese). BERTIVITS is tailored for single-speaker scenarios, improving speech synthesis technology for applications like post-class tutoring or telephone customer service.

Variational autoencoder-based techniques for a streamlined cross-topology modeling and optimization workflow in electrical drives

The generation and optimization of simulation data for electrical machines remain challenging, largely due to the complexities of magneto-static finite element analysis. Traditional methodologies are not only resource-intensive, but also time-consuming. Deep learning models can be used to shortcut these calculations. However, challenges arise when considering the unique parameter sets specific to each machine topology. Building on two recent studies (Parekh et al. in IEEE Trans. Magn. 58(9):1–4, 2022; Parekh et al., Deep learning based meta-modeling for multi-objective technology optimization of electrical machines, 2023, arXiv:2306.09087), that utilized a variational autoencoder to cohesively map diverse topologies into a singular latent space for subsequent optimization, this paper proposes a refined architecture and optimization workflow. Our modifications aim to streamline and enhance the robustness of both the training and optimization processes, and compare the results with the variational autoencoder architecture proposed recently.

Dimensionality Reduction for Onboard Modeling of Uncertain Atmospheres

Onboard density models are a key aspect of closed-loop guidance systems for hypersonic flight. Traditional approaches model density as a deterministic function of altitude, but a recent drive toward stochastic guidance approaches motivates onboard uncertainty propagation. Existing solutions for efficient uncertainty propagation generally treat density as an exponential function of altitude, but this approach is limited in its ability to capture relevant dispersions. This work models density as a Gaussian random field that is approximated by a Karhunen–Loève expansion, enabling a relatively high-fidelity, finite-dimensional parametric representation. Alternative models are also developed using a variational autoencoder architecture, resulting in greater dimensionality reduction at the expense of analytical description. Normalization schemes are presented and compared by their efficiency in capturing density variability in a limited number of terms, and normalization by reference dynamic pressure is shown to be the most compact approach. The model alternatives are compared both by their approximations of density itself and by their predictions of peak heat flux for dispersed direct-entry and aerocapture trajectories. An extension of this approach for modeling density as a function of multiple independent variables is also presented and demonstrated. Finally, it is shown that the Karhunen–Loève density model can be sequentially updated according to noisy density observations by formulating the problem as a Kalman measurement function.

CopyVAE: a variational autoencoder-based approach for copy number variation inference using single-cell transcriptomics.

Copy number variations (CNVs) are common genetic alterations in tumour cells. The delineation of CNVs holds promise for enhancing our comprehension of cancer progression. Moreover, accurate inference of CNVs from single-cell sequencing data is essential for unravelling intratumoral heterogeneity. However, existing inference methods face limitations in resolution and sensitivity. To address these challenges, we present CopyVAE, a deep learning framework based on a variational autoencoder architecture. Through experiments, we demonstrated that CopyVAE can accurately and reliably detect CNVs from data obtained using single-cell RNA sequencing. CopyVAE surpasses existing methods in terms of sensitivity and specificity. We also discussed CopyVAE's potential to advance our understanding of genetic alterations and their impact on disease advancement. CopyVAE is implemented and freely available under MIT license at https://github.com/kurtsemih/copyVAE.

Multivariate time series anomaly detection with variational autoencoder and spatial–temporal graph network

Effective anomaly detection in multivariate time series (MTS) is very essential for modern complex physical equipment. A single anomaly in physical equipment can cause a series of failures due to fault propagation. Therefore, the equipment needs to be comprehensively monitored by an anomaly detection system to ensure its health. However, given the rise in the size and complexity of the physical equipment, the number of sensors required to infer its health status has increased dramatically, and the complex temporal and spatial dependencies in sensors are difficult to capture. Aiming at the problem, this paper proposes a MTS anomaly detection algorithm called VSAD, which is based on spatial–temporal graph networks and variational autoencoder (VAE). It employs spatial–temporal graph networks to learn complex temporal and spatial dependencies in MTS. Moreover, its VAE architecture enables it to learn data distribution patterns in an unsupervised manner and its threshold is determined by an automatic threshold selection strategy. Finally, experimental results conducted on six publicly available benchmark datasets show that VSAD outperforms state-of-the-art methods for anomaly detection, with an average F1 scores improvement of about 7%.

IVP-VAE: Modeling EHR Time Series with Initial Value Problem Solvers

Continuous-time models such as Neural ODEs and Neural Flows have shown promising results in analyzing irregularly sampled time series frequently encountered in electronic health records. Based on these models, time series are typically processed with a hybrid of an initial value problem (IVP) solver and a recurrent neural network within the variational autoencoder architecture. Sequentially solving IVPs makes such models computationally less efficient. In this paper, we propose to model time series purely with continuous processes whose state evolution can be approximated directly by IVPs. This eliminates the need for recurrent computation and enables multiple states to evolve in parallel. We further fuse the encoder and decoder with one IVP solver utilizing its invertibility, which leads to fewer parameters and faster convergence. Experiments on three real-world datasets show that the proposed method can systematically outperform its predecessors, achieve state-of-the-art results, and have significant advantages in terms of data efficiency.

Recognition of multivariate geochemical anomalies using a geologically-constrained variational autoencoder network with spectrum separable module – A case study in Shangluo District, China

This study has developed a novel variational autoencoder architecture by incorporating the spectrum separable module, termed SSM-VAE, so as to recognize the multi-mineral-species geochemical patterns over the Shangluo district, central China, and then facilitate a better understanding of the regional metallogenesis. The primary advantage of SSM-VAE is its ability to explore the interlayer correlations and then integrate them into the reconstructed features, which greatly improved the geological interpretability of the model products. In the training process, a hybrid loss function with a geologically-constrained term derived from the fractal analysis was designed to guide the model to focus on anomalies related to mineralization. Afterward, the factor analysis together with the EM-MML thresholding algorithm were integrated as a post-processing tool. Finally, the mineral-spot identification rate (δ) versus the size of the anomalous area (S) are used to validate the ore-bearing potential of different anomaly divisions. Comparing to other state-of-the-art models without regard to the interlayer correlations, SSM-VAE can produce well-zoned, mineral species-specific, and more metalliferous anomalous patches. To be specific, we separated out 10 zoned anomaly divisions, and on average, when S = 1% there are 8.5013 mineral spots falling within.

Deep Generation Model Guided by the Docking Score for Active Molecular Design.

A deep generation model, as a novel drug design and discovery tool, shows obvious advantages in generating compounds with novel backbones and has been applied successfully in the field of drug discovery. However, it is still a challenge to generate molecules with expected properties, especially high activity. Here, to obtain compounds both with novelty and high activity to a target, we proposed a conditional molecular generation model COMG by considering the docking score and 3D pharmacophore matching during molecular generation. The proposed model was based on the conditional variational autoencoder architecture constrained by the pharmacophore matching score. During Bayesian optimization, the docking score was applied to enhance the target relevance of generated compounds. Furthermore, to overcome the problem of high structural similarity caused by Bayesian optimization, the idea of the scaffold memory unit was also introduced. The evaluation results of COMG show that our model not only can improve the structural diversity of generated molecules but also can effectively improve the proportion of target-related drug-active molecules. The obtained results indicate that our proposed model COMG is a useful drug design tool.

Digital Twin Hybrid Modeling for Enhancing Guided Wave Ultrasound Inspection Signals in Welded Rails

Guided wave ultrasound (GWU) systems have been widely used for monitoring structures such as rails, pipelines, and plates. In railway tracks, the monitoring process involves the complicated propagation of waves over several hundred meters. The propagating waves are multi-modal and interact with discontinuities differently, increasing complexity and leading to different response signals. When the researcher wants to gain insight into the behavior of guided waves, predicting response signals for different combinations of modes becomes necessary. However, the task can become computationally costly when physics-based models are used. Digital twins can enable a practitioner to deal systematically with the complexities of guided wave monitoring in practical or user-specified settings. This paper investigates the use of a hybrid digital model of an operational rail track to predict response signals for varying user-specified settings, specifically, the prediction of response signals for various combinations of modes of propagation in the rail. The digital twin hybrid model employs a physics-based model and a data-driven model. The physics-based model simulates the wave propagation response using techniques developed from the traditional 3D finite element method and the 2D semi-analytical finite element method (FEM). The physics-based model is used to generate virtual experimental signals containing different combinations of modes of propagation. These response signals are used to train the data-driven model based on a variational auto-encoder (VAE). Given an input baseline signal containing only the most dominant mode excited by a transducer, the VAE is trained to predict an inspection signal with increased complexity according to the specified combination of modes. The results show that, once the VAE has been trained, it can be used to predict inspection signals for different combinations of propagating modes, thus replacing the physics-based model, which is computationally costly. In the future, the VAE architecture will be adapted to predict response signals for varying environmental and operational conditions.

Variational Autoencoder with CCA for Audio–Visual Cross-modal Retrieval

Cross-modal retrieval is to utilize one modality as a query to retrieve data from another modality, which has become a popular topic in information retrieval, machine learning, and databases. Finding a method to effectively measure the similarity between different modality data is the major challenge of cross-modal retrieval. Although several research works have calculated the correlation between different modality data via learning a common subspace representation, the encoder’s ability to extract features from multi-modal information is not satisfactory. In this article, we present a novel variational autoencoder architecture for audio–visual cross-modal retrieval by learning paired audio–visual correlation embedding and category correlation embedding as constraints to reinforce the mutuality of audio–visual information. On the one hand, audio encoder and visual encoder separately encode audio data and visual data into two different latent spaces. Further, two mutual latent spaces are respectively constructed by canonical correlation analysis. On the other hand, probabilistic modeling methods are used to deal with possible noise and missing information in the data. Additionally, in this way, the cross-modal discrepancies from intra-modal and inter-modal information are simultaneously eliminated in the joint embedding subspace. We conduct extensive experiments over two benchmark datasets. The experimental results confirm that the proposed architecture is effective in learning audio–visual correlation and is appreciably better than the existing cross-modal retrieval methods.

Variational autoencoder for the identification of piecewise models

The paper presents a variational autoencoder (VAE) tailored for the identification of hybrid piecewise models in input-output form. We show that using a specialized autoencoder structure, the latent space can provide an interpretable representation in terms of the modes of the underlying hybrid system. In particular, we use categorical encoding of the discrete latent variables whose distribution is approximated via the encoder neural network, characterizing a partition of the regressor space, while the decoder consists of a set of neural networks, each corresponding to a local submodel of the piecewise hybrid system. By employing variational Bayesian framework for inference, the constitutive terms of the evidence lower bound (ELBO) are derived analytically with the chosen VAE architecture. The ELBO loss consists of a reconstruction error term and a regularization term over the latent modes. This loss is optimized in order to train the encoder-decoder networks concurrently via back-propagation. The developed framework is not restricted to simple piecewise affine (PWA) models and it can be straightforwardly extended to general class of piecewise non-linear systems over non-polyhedral domains.

Initial Study of Batik Generation using Variational Autoencoder

One of the most promising architectures for generative models is the variational autoencoder (VAE). To reconstruct Batik patterns for this work, we used a deep convolutional VAE architecture. Reconstruction outcomes from various batik motifs are mapped and contrasted using some criteria. As another crucial component of the convolutional network, batch normalization's impact on the model's performance was also examined. The dataset is used to study some learned latent space features. Through these findings, we laid the framework for next research on Batik generation utilizing VAE.

Exploiting the sparse characteristics in probabilistic feature space for hyperspectral anomaly detection

Nowadays, the low-rank representation (LRR) and deep learning-based methods have received much attention in anomaly detection for hyperspectral images (HSIs). However, most of these methods mainly focus on the powerful reconstruction capability of the neural networks while ignoring the potential probability distribution of both anomalies and background pixels. To solve the problem, we propose a sparse component extraction-based probability distribution representation detector (SC-PDRD) framework, which integrates the characteristic of the sparse component obtained by the LRR model with the powerful probability representation ability of the variational autoencoder (VAE) network. The LRR model effectively separates the anomaly component from the background, which also serves as the prior anomaly distribution for each pixel. Moreover, the VAE architecture tries to recover the potential anomaly distribution using the sparse detection map in the feature space. In addition, we employ the Chebyshev neighborhood to leverage spatial information. The modified Wasserstein distance measures the distance between the test pixel and its neighborhood. The final detection map is attained by combining the prior anomalous degree of the anomalies with the output of the VAE network. Experimental results on three real HSIs demonstrate the effectiveness and superiority of SC-PDRD.

A deep learning generative model approach for image synthesis of plant leaves.

A well-known drawback to the implementation of Convolutional Neural Networks (CNNs) for image-recognition is the intensive annotation effort for large enough training dataset, that can become prohibitive in several applications. In this study we focus on applications in the agricultural domain and we implement Deep Learning (DL) techniques for the automatic generation of meaningful synthetic images of plant leaves, which can be used as a virtually unlimited dataset to train or validate specialized CNN models or other image-recognition algorithms. Following an approach based on DL generative models, we introduce a Leaf-to-Leaf Translation (L2L) algorithm, able to produce collections of novel synthetic images in two steps: first, a residual variational autoencoder architecture is used to generate novel synthetic leaf skeletons geometry, starting from binarized skeletons obtained from real leaf images. Second, a translation via Pix2pix framework based on conditional generator adversarial networks (cGANs) reproduces the color distribution of the leaf surface, by preserving the underneath venation pattern and leaf shape. The L2L algorithm generates synthetic images of leaves with meaningful and realistic appearance, indicating that it can significantly contribute to expand a small dataset of real images. The performance was assessed qualitatively and quantitatively, by employing a DL anomaly detection strategy which quantifies the anomaly degree of synthetic leaves with respect to real samples. Finally, as an illustrative example, the proposed L2L algorithm was used for generating a set of synthetic images of healthy end diseased cucumber leaves aimed at training a CNN model for automatic detection of disease symptoms. Generative DL approaches have the potential to be a new paradigm to provide low-cost meaningful synthetic samples. Our focus was to dispose of synthetic leaves images for smart agriculture applications but, more in general, they can serve for all computer-aided applications which require the representation of vegetation. The present L2L approach represents a step towards this goal, being able to generate synthetic samples with a relevant qualitative and quantitative resemblance to real leaves.

Traffic Request Generation through a Variational Auto Encoder Approach

Traffic and transportation forecasting is a key issue in urban planning aimed to provide a greener and more sustainable environment to residents. Their privacy is a second key issue that requires synthetic travel data. A possible solution is offered by generative models. Here, a variational autoencoder architecture has been trained on a floating car dataset in order to grasp the statistical features of the traffic demand in the city of Rome. The architecture is based on multilayer dense neural networks for encoding and decoding parts. A brief analysis of parameter influence is conducted. The generated trajectories are compared with those in the dataset. The resulting reconstructed synthetic data are employed to compute the traffic fluxes and geographic distribution of parked cars. Further work directions are provided.

GeoSDVA: A Semi-Supervised Dirichlet Variational Autoencoder Model for Transportation Mode Identification

Inferring the transportation modes of travelers is an essential part of intelligent transportation systems. With the development of mobile services, it is easy to effectively obtain massive location readings of travelers with GPS-enabled smart devices, such as smartphones. These readings make understanding human activities very convenient. Therefore, how to automatically infer transportation modes from these massive readings has come into the spotlight. The existing methods for transportation mode identification are usually based on supervised learning. However, the raw GPS readings do not contain any labels, and it is expensive and time-consuming to annotate sufficient samples for training supervised learning-based models. In addition, not enough attention is paid to the problem that GPS readings collected in urban areas are affected by surrounding geographic information (e.g., the level of road transportation or the distribution of stations). To solve this problem, a geographic information-fused semi-supervised method based on a Dirichlet variational autoencoder, named GeoSDVA, is proposed in this paper for transportation mode identification. GeoSDVA first fuses the motion features of the GPS trajectories with the nearby geographic information. Then, both labeled and unlabeled trajectories are used to train the semi-supervised model based on the Dirichlet variational autoencoder architecture for transportation mode identification. Experiments on three real GPS trajectory datasets demonstrate that GeoSDVA can train an excellent transportation mode identification model with only a few labeled trajectories.

Molecular substructure tree generative model for de novo drug design.

Deep learning shortens the cycle of the drug discovery for its success in extracting features of molecules and proteins. Generating new molecules with deep learning methods could enlarge the molecule space and obtain molecules with specific properties. However, it is also a challenging task considering that the connections between atoms are constrained by chemical rules. Aiming at generating and optimizing new valid molecules, this article proposed Molecular Substructure Tree Generative Model, in which the molecule is generated by adding substructure gradually. The proposed model is based on the Variational Auto-Encoder architecture, which uses the encoder to map molecules to the latent vector space, and then builds an autoregressive generative model as a decoder to generate new molecules from Gaussian distribution. At the same time, for the molecular optimization task, a molecular optimization model based on CycleGAN was constructed. Experiments showed that the model could generate valid and novel molecules, and the optimized model effectively improves the molecular properties.

Unsupervised discovery of dynamic cell phenotypic states from transmitted light movies.

Identification of cell phenotypic states within heterogeneous populations, along with elucidation of their switching dynamics, is a central challenge in modern biology. Conventional single-cell analysis methods typically provide only indirect, static phenotypic readouts. Transmitted light images, on the other hand, provide direct morphological readouts and can be acquired over time to provide a rich data source for dynamic cell phenotypic state identification. Here, we describe an end-to-end deep learning platform, UPSIDE (Unsupervised Phenotypic State IDEntification), for discovering cell states and their dynamics from transmitted light movies. UPSIDE uses the variational auto-encoder architecture to learn latent cell representations, which are then clustered for state identification, decoded for feature interpretation, and linked across movie frames for transition rate inference. Using UPSIDE, we identified distinct blood cell types in a heterogeneous dataset. We then analyzed movies of patient-derived acute myeloid leukemia cells, from which we identified stem-cell associated morphological states as well as the transition rates to and from these states. UPSIDE opens up the use of transmitted light movies for systematic exploration of cell state heterogeneity and dynamics in biology and medicine.

Abstract 10605: Artificial Intelligence Effectively Measures Left Atrial Volumes and Function on Multiphasic Cardiac Cta in Patients with Atrial Fibrillation

Introduction: This study was performed to evaluate a novel AI algorithm for fully automated measurement of LA volumes and function using cardiac CTA in patients with atrial fibrillation (AF). Methods: A prototype AI algorithm (AI-LAV), with a conditional variational auto-encoder architecture, was developed and trained using 555 manually annotated ECG-gated, contrast-enhanced cardiac CTs. A total of 79 patients (mean age 63 ± 12 years; 35 with AF and 44 controls) evaluated between 2017 and 2020 were included in this retrospective study to validate the AI model. Images were analyzed by the trained AI algorithm and experienced radiologists. Left atrial volumes were obtained at cardiac end-systole (LAVmax), end-diastole (LAVmin), and pre-atrial contraction (LAVpc), which were then used to obtain LA function indices (total, passive, and active emptying fractions). Statistical analysis was performed. Results: AI was significantly faster than manual measurement of LA volumes and function (4 s vs 10.8 min, respectively). Agreement between the manual and automated methods was good to excellent overall, and there was stronger agreement in AF patients (all ICCs ≥ 0.877; p<0.001) than controls (all ICCs ≥ 0.799; p<0.001). The AI comparably estimated LA volumes and function in AF patients (all within 1.3 mL and 1.7% of the manual measurement), but overestimated volumes and function by clinically negligible amounts in controls (all by ≤ 4.2 ml and ≤ 3.5%). The AI’s ability to distinguish AF patients from controls using the LA volume index was similar to the experts’ (AUC 0.81 vs 0.82, respectively; p = 0.62). Conclusion: In a validation cohort, the novel AI algorithm efficiently performed fully automated multiphasic CTA-based quantification of left atrial volume and function with similar accuracy to expert manual quantification. Figure. Scatter and Bland-Altman plots for LAVmax and LAVmin. Segmentation comparison between expert (manual) and AI on cardiac CTA.

Abstract 10633: Deep Learning Model Accurately Quantifies LA Volumes on Routine Non-Contrast Chest CTs and Predicts Adverse Outcomes

Introduction/Hypothesis: To develop and validate a deep learning model to automatically segment and measure left atrial (LA) volumes from routine noncontrast chest CTs and evaluate for outcomes prognosis. Methods: A prototype AI algorithm, with a conditional variational auto-encoder architecture, was initially developed and trained using training data from 1313 patients (paired CTA and non-contrasted exams; “training group”). Left atrium contours from contrasted cases were used to train the model on corresponding non-contrasted dataset. We then retrospectively evaluated 274 patients (mean age 67.9 years, 54.6% male, 57.2% caucasian) who underwent a routine non ECG-gated, noncontrast chest CT for coronary artery calcium or lung cancer screening for validation of the AI model (“validation group”). Outcomes included heart failure (HF) hospitalization and new-onset atrial fibrillation within 5 years. Simple logistic regressions with appropriate univariate statistics were used for modelling outcomes. Results: In the validation group, there was excellent correlation between AI and expert results with a LA volume index (LAVi) intraclass correlation coefficient of 0.950 (0.936-0.960). Bland-Altman plot demonstrated that the AI underestimated LAVi by a mean 5.86 mL/m2. AI-LAVi independently predicted new-onset atrial fibrillation (AUC = 0.855, OR = 1.12, 95% CI 1.08 - 1.18; p<0.0001). AI-LAVi was independently associated with HF hospitalization (AUC = 0.903, OR = 1.08, 95% CI 1.04 - 1.13, p = 0.0009). Conclusion: In this large validation cohort, AI for automated measurement of LA volume on routine noncontrast chest CTs had excellent agreement with manual quantification. AI-LAVi is independently predictive for new-onset atrial fibrillation and HF hospitalization within 5 years.