Latent Distribution Research Articles

Context and cross-section data improve analyses of wine ratings

Abstract Much research shows that the ratings that critics, judges, and consumers assign to wines are heteroscedastic. A rating observed is one draw from a latent distribution that is wine- and judge-specific. Estimating the shape of a rating’s distribution by minimizing a sum of cross entropies has been proposed and tested. This article proposes a method of improving the accuracy of that estimate by using information about the context of a wine competition or cross-section ratings data. Tests using the distributions implied by 90 blind triplicate ratings show that the sum of squared errors for the solution using context or cross-section information is 50% more accurate than not using such information and over 99% more accurate than ignoring the uncertainty about a rating.

Deep learning model for smart wearables device to detect human health conduction

With the proliferation of smart wearables, motion wristbands provide a wealth of data essential for comprehending the dynamic nature of health. However, outlier detection is typically necessary due to the presence of unknown outliers in their multidimensional activity data. Conventional approaches frequently result in incorrect object identification due to the curse of dimensionality. Using the Gaussian Mixture Generative Model (GMGM), we provide a method to identify outliers and address this problem. Training on raw data is done using a VariationalAutoencoder (VAE). While avoiding rebuilding mistakes, we want to achieve as many brief features as possible. To predict the likelihood that examples contain many types of data, a DBN will utilise feature extractions and latent distributions in the future. The model's robustness is enhanced by enhancing the VAE, deep learning components, and the GMM overall. When densities surpass the training level, the Gaussian Mixture Model identifies outliers. To achieve this, it makes educated guesses about the densities of each data point. Compared to the deep learning Autoencoding Gaussian Mixture Model (DAGMM), GMGM achieves a 5.5 % higher area under the curve (AUC) on the ODDS standard dataset. Experiments conducted on real datasets further demonstrate the efficacy of this strategy.

Scalable Empirical Bayes Inference and Bayesian Sensitivity Analysis.

Consider a Bayesian setup in which we observe , whose distribution depends on a parameter , that is, . The parameter is unknown and treated as random, and a prior distribution chosen from some parametric family , is to be placed on it. For the subjective Bayesian there is a single prior in the family which represents his or her beliefs about , but determination of this prior is very often extremely difficult. In the empirical Bayes approach, the latent distribution on is estimated from the data. This is usually done by choosing the value of the hyperparameter that maximizes some criterion. Arguably the most common way of doing this is to let be the marginal likelihood of , that is, , and choose the value of that maximizes . Unfortunately, except for a handful of textbook examples, analytic evaluation of is not feasible. The purpose of this paper is two-fold. First, we review the literature on estimating it and find that the most commonly used procedures are either potentially highly inaccurate or don't scale well with the dimension of , the dimension of , or both. Second, we present a method for estimating , based on Markov chain Monte Carlo, that applies very generally and scales well with dimension. Let be a real-valued function of , and let be the posterior expectation of when the prior is . As a byproduct of our approach, we show how to obtain point estimates and globally-valid confidence bands for the family , . To illustrate the scope of our methodology we provide three detailed examples, having different characters.

Weight decay regularized adversarial training for attacking angle imbalance

In this paper, we explore the latent distribution of adversarial examples in terms of the angle between the adversarial perturbation and the image manifold to optimize adversarial training (AT). Firstly, we study the norm ratio of a random vector to its projection on the image manifold to demonstrate that adversarial perturbations favor dimensions outside the manifold. Second, we explore the effect of the angle between the adversarial perturbation and the image manifold. By studying this distribution and establishing its relation to the loss between adversarial and real samples, we find that the angle distribution is approximately normal. Consequently, this unbalanced training in terms of attacking angles causes the loss to fluctuate heavily for angles located in the tails of the distribution. To address this problem, we propose a novel weight decay regularized AT, termed WDR AT. Specifically, WDR AT assesses the attack fluctuation of each adversarial example and compares it with the scaled mean and down-weights anomalous adversarial examples. Finally, results obtained on four typical datasets, MNIST, CIFAR10, CIFAR100, and TinyImageNet demonstrate the superiority of WDR-AT over state-of-the-art AT methods.

Airfoil Shape Generation and Feature Extraction Using the Conditional VAE-WGAN-gp

A machine learning method was applied to solve an inverse airfoil design problem. A conditional VAE-WGAN-gp model, which couples the conditional variational autoencoder (VAE) and Wasserstein generative adversarial network with gradient penalty (WGAN-gp), is proposed for an airfoil generation method, and then, it is compared with the WGAN-gp and VAE models. The VAEGAN model couples the VAE and GAN models, which enables feature extraction in the GAN models. In airfoil generation tasks, to generate airfoil shapes that satisfy lift coefficient requirements, it is known that VAE outperforms WGAN-gp with respect to the accuracy of the reproduction of the lift coefficient, whereas GAN outperforms VAE with respect to the smoothness and variations of generated shapes. In this study, VAE-WGAN-gp demonstrated a good performance in all three aspects. Latent distribution was also studied to compare the feature extraction ability of the proposed method.

Diffusion model conditioning on Gaussian mixture model and negative Gaussian mixture gradient

Diffusion models (DMs) are a type of generative model that has had a significant impact on image synthesis and beyond. They can incorporate a wide variety of conditioning inputs — such as text or bounding boxes — to guide generation. In this work, we introduce a novel conditioning mechanism that applies Gaussian mixture models (GMMs) for feature conditioning, which helps steer the denoising process in DMs. Drawing on set theory, our comprehensive theoretical analysis reveals that the conditional latent distribution based on features differs markedly from that based on classes. Consequently, feature-based conditioning tends to generate fewer defects than class-based conditioning. Experiments are designed and carried out and the experimental results support our theoretical findings as well as effectiveness of proposed feature conditioning mechanism. Additionally, we propose a new gradient function named the Negative Gaussian Mixture Gradient (NGMG) and incorporate it into the training of diffusion models alongside an auxiliary classifier. We theoretically demonstrate that NGMG offers comparable advantages to the Wasserstein distance, serving as a more effective cost function when learning distributions supported by low-dimensional manifolds, especially in contrast to many likelihood-based cost functions, such as KL divergences.

Towards training noise-robust anomaly detection via collaborative adversarial flows

Unsupervised visual anomaly detection (UVAD) aims at identifying unobserved anomalies by training a classification model with anomaly-free images, which has been widespread applied to surface inspection of multiple industrial products. However, previous UVAD methods trained by ideal normal samples suffer from performance degradation in real world industry because of the inevitable intrusion of noisy samples into the training set. Addressing this issue, this article introduces a novel strategy, Collaborative Adversarial Flows (CAF), which employs multiple normalizing flows to efficiently filter out noisy samples and accurately construct distributions of normal samples. Unlike previous methods that focused solely on learning accurate representations of normal samples, CAF adds an additional learning objective of inferring noisy samples based on distributional metrics while optimizing representations. Furthermore, CAF establishes a flow-based collaborative adversarial paradigm that promotes high-likelihood samples and rejects low-likelihood noisy samples in optimizing the latent distribution, thereby enhancing the robustness of model under contaminated training set. Comprehensive experiments on the public benchmark MVTec AD under human-made noise intrusion demonstrate that CAF effectively improves the robustness to noisy samples and achieves impressive performance. In electronic manufacturing processes with real industrial noise, the applicability and robustness of CAF have been further proved by the inspection of surface mount devices.

Adaptive Learning of the Latent Space of Wasserstein Generative Adversarial Networks

Generative models based on latent variables, such as generative adversarial networks (GANs) and variational auto-encoders (VAEs), have gained lots of interests due to their impressive performance in many fields. However, many data such as natural images usually do not populate the ambient Euclidean space but instead reside in a lower-dimensional manifold. Thus an inappropriate choice of the latent dimension fails to uncover the structure of the data, possibly resulting in mismatch of latent representations and poor generative qualities. Towards addressing these problems, we propose a novel framework called the latent Wasserstein GAN (LWGAN) that fuses the Wasserstein auto-encoder and the Wasserstein GAN so that the intrinsic dimension of the data manifold can be adaptively learned by a modified informative latent distribution. We prove that there exist an encoder network and a generator network in such a way that the intrinsic dimension of the learned encoding distribution is equal to the dimension of the data manifold. We theoretically establish that our estimated intrinsic dimension is a consistent estimate of the true dimension of the data manifold. Meanwhile, we provide an upper bound on the generalization error of LWGAN, implying that we force the synthetic data distribution to be similar to the real data distribution from a population perspective. Comprehensive empirical experiments verify our framework and show that LWGAN is able to identify the correct intrinsic dimension under several scenarios, and simultaneously generate high-quality synthetic data by sampling from the learned latent distribution.

Normalizing flow sampling with Langevin dynamics in the latent space

AbstractNormalizing flows (NF) use a continuous generator to map a simple latent (e.g. Gaussian) distribution, towards an empirical target distribution associated with a training data set. Once trained by minimizing a variational objective, the learnt map provides an approximate generative model of the target distribution. Since standard NF implement differentiable maps, they may suffer from pathological behaviors when targeting complex distributions. For instance, such problems may appear for distributions on multi-component topologies or characterized by multiple modes with high probability regions separated by very unlikely areas. A typical symptom is the explosion of the Jacobian norm of the transformation in very low probability areas. This paper proposes to overcome this issue thanks to a new Markov chain Monte Carlo algorithm to sample from the target distribution in the latent domain before transporting it back to the target domain. The approach relies on a Metropolis adjusted Langevin algorithm whose dynamics explicitly exploits the Jacobian of the transformation. Contrary to alternative approaches, the proposed strategy preserves the tractability of the likelihood and it does not require a specific training. Notably, it can be straightforwardly used with any pre-trained NF network, regardless of the architecture. Experiments conducted on synthetic and high-dimensional real data sets illustrate the efficiency of the method.

Estimating Addiction-Related Brain Connectivity by Prior-Embedding Graph Generative Adversarial Networks.

The study of nicotine addiction mechanism is of great significance in both nicotine withdrawal and brain science. The detection of addiction-related brain connectivity using functional magnetic resonance imaging (fMRI) is a critical step in study of this mechanism. However, it is challenging to accurately estimate addiction-related brain connectivity due to the low-signal-to-noise ratio of fMRI and the issue of small sample size. In this work, a prior-embedding graph generative adversarial network (PG-GAN) is proposed to capture addiction-related brain connectivity accurately. By designing a dual-generator-based scheme, the addiction-related connectivity generator is employed to learn the feature map of addiction connection, while the reconstruction generator is used for sample reconstruction. Moreover, a bidirectional mapping mechanism is designed to maintain the consistency of sample distribution in the latent space so that addiction-related brain connectivity can be estimated more accurately. The proposed model utilizes prior knowledge embeddings to reduce the search space so that the model can better understand the latent distribution for the issue of small sample size. Experimental results demonstrate the effectiveness of the proposed PG-GAN.

Physical discovery in representation learning via conditioning on prior knowledge

Recent advances in electron, scanning probe, optical, and chemical imaging and spectroscopy yield bespoke data sets containing the information of structure and functionality of complex systems. In many cases, the resulting data sets are underpinned by low-dimensional simple representations encoding the factors of variability within the data. The representation learning methods seek to discover these factors of variability, ideally further connecting them with relevant physical mechanisms. However, generally, the task of identifying the latent variables corresponding to actual physical mechanisms is extremely complex. Here, we present an empirical study of an approach based on conditioning the data on the known (continuous) physical parameters and systematically compare it with the previously introduced approach based on the invariant variational autoencoders. The conditional variational autoencoder (cVAE) approach does not rely on the existence of the invariant transforms and hence allows for much greater flexibility and applicability. Interestingly, cVAE allows for limited extrapolation outside of the original domain of the conditional variable. However, this extrapolation is limited compared to the cases when true physical mechanisms are known, and the physical factor of variability can be disentangled in full. We further show that introducing the known conditioning results in the simplification of the latent distribution if the conditioning vector is correlated with the factor of variability in the data, thus allowing us to separate relevant physical factors. We initially demonstrate this approach using 1D and 2D examples on a synthetic data set and then extend it to the analysis of experimental data on ferroelectric domain dynamics visualized via piezoresponse force microscopy.

A conditional latent autoregressive recurrent model for generation and forecasting of beam dynamics in particle accelerators

Particle accelerators are complex systems that focus, guide, and accelerate intense charged particle beams to high energy. Beam diagnostics present a challenging problem due to limited non-destructive measurements, computationally demanding simulations, and inherent uncertainties in the system. We propose a two-step unsupervised deep learning framework named as Conditional Latent Autoregressive Recurrent Model (CLARM) for learning the spatiotemporal dynamics of charged particles in accelerators. CLARM consists of a Conditional Variational Autoencoder transforming six-dimensional phase space into a lower-dimensional latent distribution and a Long Short-Term Memory network capturing temporal dynamics in an autoregressive manner. The CLARM can generate projections at various accelerator modules by sampling and decoding the latent space representation. The model also forecasts future states (downstream locations) of charged particles from past states (upstream locations). The results demonstrate that the generative and forecasting ability of the proposed approach is promising when tested against a variety of evaluation metrics.

Polypharmacy side effect prediction based on semi-implicit graph variational auto-encoder.

Polypharmacy, the use of drug combinations, is an effective approach for treating complex diseases, but it increases the risk of adverse effects. To predict novel polypharmacy side effects based on known ones, many computational methods have been proposed. However, most of them generate deterministic low-dimensional embeddings when modeling the latent space of drugs, which cannot effectively capture potential side effect associations between drugs. In this study, we present SIPSE, a novel approach for predicting polypharmacy side effects. SIPSE integrates single-drug side effect information and drug-target protein data to construct novel drug feature vectors. Leveraging a semi-implicit graph variational auto-encoder, SIPSE models known polypharmacy side effects and generates flexible latent distributions for drug nodes. SIPSE infers the current node distribution by combining the distributions of neighboring nodes with embedding noise. By sampling node embeddings from these distributions, SIPSE effectively predicts polypharmacy side effects between drugs. One key innovation of SIPSE is its incorporation of uncertainty propagation through noise embedding and neighborhood sharing, enhancing its graph analysis capabilities. Extensive experiments on a benchmark dataset of polypharmacy side effects demonstrated that SIPSE significantly outperformed five state-of-the-art methods in predicting polypharmacy side effects.

Latent Prototype-Based Clustering: A Novel Exploratory Electroencephalography Analysis Approach.

Electroencephalography (EEG)-based applications in brain-computer interfaces (BCIs), neurological disease diagnosis, rehabilitation, etc., rely on supervised approaches such as classification that requires given labels. However, with the ever-increasing amount of EEG data, incomplete or incorrectly labeled or unlabeled EEG data are increasing. It likely degrades the performance of supervised approaches. In this work, we put forward a novel unsupervised exploratory EEG analysis solution by clustering based on low-dimensional prototypes in latent space that are associated with the respective clusters. Having the prototype as a baseline of each cluster, a compositive similarity is defined to act as the critic function in clustering, which incorporates similarities on three levels. The approach is implemented with a Generative Adversarial Network (GAN), termed W-SLOGAN, by extending the Stein Latent Optimization for GANs (SLOGAN). The Gaussian Mixture Model (GMM) is utilized as the latent distribution to adapt to the diversity of EEG signal patterns. The W-SLOGAN ensures that images generated from each Gaussian component belong to the associated cluster. The adaptively learned Gaussian mixing coefficients make the model remain effective in dealing with an imbalanced dataset. By applying the proposed approach to two public EEG or intracranial EEG (iEEG) epilepsy datasets, our experiments demonstrate that the clustering results are close to the classification of the data. Moreover, we present several findings that were discovered by intra-class clustering and cross-analysis of clustering and classification. They show that the approach is attractive in practice in the diagnosis of the epileptic subtype, multiple labelling of EEG data, etc.

Transparent Image Layer Diffusion using Latent Transparency

We present an approach enabling large-scale pretrained latent diffusion models to generate transparent images. The method allows generation of single transparent images or of multiple transparent layers. The method learns a "latent transparency" that encodes alpha channel transparency into the latent manifold of a pretrained latent diffusion model. It preserves the production-ready quality of the large diffusion model by regulating the added transparency as a latent offset with minimal changes to the original latent distribution of the pretrained model. In this way, any latent diffusion model can be converted into a transparent image generator by finetuning it with the adjusted latent space. We train the model with 1M transparent image layer pairs collected using a human-in-the-loop collection scheme. We show that latent transparency can be applied to different open source image generators, or be adapted to various conditional control systems to achieve applications like foreground/background-conditioned layer generation, joint layer generation, structural control of layer contents, etc. A user study finds that in most cases (97%) users prefer our natively generated transparent content over previous ad-hoc solutions such as generating and then matting. Users also report the quality of our generated transparent images is comparable to real commercial transparent assets like Adobe Stock.

Open-world structured sequence learning via dense target encoding

Structured sequences are popularly used to describe graph data with time-evolving node features and edges. A typical real-world scenario of structured sequences is that unknown class labels continuously arrive and thus the training and testing often across different class spaces. This scenario is also referred to as the open-world learning problem on structured sequences. In this paper, we present a new Dense Open-world Structured Sequence Learning model (DOSSL for short) to learn graph streams in the open-world learning setting. To capture both structural and temporal information, DOSSL uses a GNN-based stochastic recurrent neural network for learning node representation in graph streams, then a truncated Laplacian distribution to describe the latent distribution of graph nodes, and a sampling function is used to generate node representations. Further, DOSSL learns dense target embeddings for the known classes to improve the compactness of known class distribution and reserve enough space for open-world unknown classes. The ultimate open-world classifier is optimized to detect the samples from unknown classes under the constraints of DVAE loss, label loss, class uncertainty loss, and dense target loss. Through empirical analysis conducted on real-world datasets, it has been demonstrated that the advanced technique known as DOSSL exhibits the ability to acquire precise node classifiers by harnessing the power of graph streams.

Central Kurdish Text-to-Speech Synthesis with Novel End-to-End Transformer Training

Recent advancements in text-to-speech (TTS) models have aimed to streamline the two-stage process into a single-stage training approach. However, many single-stage models still lag behind in audio quality, particularly when handling Kurdish text and speech. There is a critical need to enhance text-to-speech conversion for the Kurdish language, particularly for the Sorani dialect, which has been relatively neglected and is underrepresented in recent text-to-speech advancements. This study introduces an end-to-end TTS model for efficiently generating high-quality Kurdish audio. The proposed method leverages a variational autoencoder (VAE) that is pre-trained for audio waveform reconstruction and is augmented by adversarial training. This involves aligning the prior distribution established by the pre-trained encoder with the posterior distribution of the text encoder within latent variables. Additionally, a stochastic duration predictor is incorporated to imbue synthesized Kurdish speech with diverse rhythms. By aligning latent distributions and integrating the stochastic duration predictor, the proposed method facilitates the real-time generation of natural Kurdish speech audio, offering flexibility in pitches and rhythms. Empirical evaluation via the mean opinion score (MOS) on a custom dataset confirms the superior performance of our approach (MOS of 3.94) compared with that of a one-stage system and other two-staged systems as assessed through a subjective human evaluation.

Versatile latent distribution-preserving tabular data synthesis-based endovascular treatment selection for intracranial aneurysm

Proper decision-making for endovascular treatment (EVT) is crucial in reducing complications of intracranial aneurysms (IAs) and improving the quality of patients’ lives. Electronic medical records (EMRs) possess comprehensive information about patients, which provides substantial data support for EVT decisions. Machine learning (ML) models demonstrate considerable performance in handling the relationship between various clinical indicators and treatment decisions based on EMRs. However, they still face challenges regarding the data scarcity and imbalance distribution of EMRs. To address these issues, we propose a latent distribution-preserving structure that can generate authentic EMRs in a category-aware manner. We first map the EMRs into the latent space to obtain the original prototypes and generate the minority prototypes by augmenting within the latent distributions of the minority categories. The original and synthesized minority prototypes are then reconstructed and synthesized to EMRs. The reconstruction process aligns the original and synthesized data space, promoting high authenticity in data-scarce sample generation. Subsequently, the synthesized and original EMRs are merged to obtain sufficient and balanced training data for ML models. Experiment results demonstrate the effectiveness of our proposed structure in enhancing generative performance across various state-of-the-art tabular GAN and boosts the treatment-decision performance by 2.8%–7.2% in AUC, 3.4%–17.2% in F1. The resources will be publicly available.

Enhancing time series data classification for structural damage detection through out-of-distribution representation learning

The analysis and classification of time series data, notably within Structural Health Monitoring (SHM) systems, face profound challenges due to the nonstationary nature of the data and changes in distribution over time. Additionally, the complexity of the SHM systems' time series datasets also relates to the presence of elusive latent distributions, making it challenging to pinpoint the dynamics of data distribution shifts. Therefore, this paper uses a Deep Learning (DL) model to address dynamic distribution changes and target elusive latent distributions in the data of SHM systems by adopting an out-of-distribution (OOD) representation learning method. The core idea is that by using adversarial training and domain-invariant feature learning, the used model enhances the analysis of time series data by dividing the data into distinct latent sub-domains, aiming to maximize the variance in distribution between each segment, facilitating a more unified analysis across different data segments. The effectiveness is demonstrated through testing on extensive datasets, including both real-world bridge data and laboratory data, showing a significant improvement over traditional DL models, especially in its ability to generalize to new, unseen distributions.

IMPROVING NORMATIVE MODELING FOR MULTI-MODAL NEUROIMAGING DATA USING MIXTURE-OF-PRODUCT-OF-EXPERTS VARIATIONAL AUTOENCODERS.

Normative models in neuroimaging learn the brain patterns of healthy population distribution and estimate how disease subjects like Alzheimer's Disease (AD) deviate from the norm. Existing variational autoencoder (VAE)-based normative models using multimodal neuroimaging data aggregate information from multiple modalities by estimating product or averaging of unimodal latent posteriors. This can often lead to uninformative joint latent distributions which affects the estimation of subject-level deviations. In this work, we addressed the prior limitations by adopting the Mixture-of-Product-of-Experts (MoPoE) technique which allows better modelling of the joint latent posterior. Our model labelled subjects as outliers by calculating deviations from the multimodal latent space. Further, we identified which latent dimensions and brain regions were associated with abnormal deviations due to AD pathology.