Distribution Of Latent Variables Research Articles

Patch-Level Feature Selection for Thoracic Disease Classification by Chest X-ray Images Using Information Bottleneck.

Chest X-ray (CXR) examination serves as a widely employed clinical test in medical diagnostics. Many studied have tried to apply artificial intelligence (AI) programs to analyze CXR images. Despite numerous positive outcomes, assessing the applicability of AI models for comprehensive diagnostic support remains a formidable challenge. We observed that, even when AI models exhibit high accuracy on one dataset, their performance may deteriorate when tested on another. To address this issue, we propose incorporating a variational information bottleneck (VIB) at the patch level to enhance the generalizability of diagnostic support models. The VIB introduces a probabilistic model aimed at approximating the posterior distribution of latent variables given input data, thereby enhancing the model's generalization capabilities on unseen data. Unlike the conventional VIB approaches that flatten features and use a re-parameterization trick to sample a new latent feature, our method applies the trick to 2D feature maps. This design allows only important pixels to respond, and the model will select important patches in an image. Moreover, the proposed patch-level VIB seamlessly integrates with various convolutional neural networks, offering a versatile solution to improve performance. Experimental results illustrate enhanced accuracy in standard experiment settings. In addition, the method shows robust improvement when training and testing on different datasets.

Instance transfer partial least squares for semi-supervised adaptive soft sensor

In order to deal with the unknown data drift problem caused by the change of working conditions in the process of industrial soft sensor modeling, this paper proposes a semi-supervised soft sensor modeling algorithm based on instance transfer learning, named instance transfer partial least squares. Under the framework of nonlinear partial least squares, the algorithm introduces the instance transfer inner model, and realizes the distribution adaptation of the modeling data and the data to be predicted through sample importance weighting. First, a nonlinear partial least squares model is constructed by using labeled samples in the source domain, which consists of a linear outer model and a nonlinear inner model. The linear outer model maps the original data to the hidden variable space, and the nonlinear inner model realizes the nonlinear modeling of hidden variables through kernel ridge regression. Second, a small number of labeled samples from the target domain are used to update the model weights of the nonlinear inner model, enabling adaptation of the latent variable distribution. The effectiveness of the algorithm is verified on a toy dataset, a simulation dataset and a real pulverized coal power plant dataset.

Distributional Drift Adaptation With Temporal Conditional Variational Autoencoder for Multivariate Time Series Forecasting.

Due to the nonstationary nature, the distribution of real-world multivariate time series (MTS) changes over time, which is known as distribution drift. Most existing MTS forecasting models greatly suffer from distribution drift and degrade the forecasting performance over time. Existing methods address distribution drift via adapting to the latest arrived data or self-correcting per the meta knowledge derived from future data. Despite their great success in MTS forecasting, these methods hardly capture the intrinsic distribution changes, especially from a distributional perspective. Accordingly, we propose a novel framework temporal conditional variational autoencoder (TCVAE) to model the dynamic distributional dependencies over time between historical observations and future data in MTSs and infer the dependencies as a temporal conditional distribution to leverage latent variables. Specifically, a novel temporal Hawkes attention (THA) mechanism represents temporal factors that subsequently fed into feedforward networks to estimate the prior Gaussian distribution of latent variables. The representation of temporal factors further dynamically adjusts the structures of Transformer-based encoder and decoder to distribution changes by leveraging a gated attention mechanism (GAM). Moreover, we introduce conditional continuous normalization flow (CCNF) to transform the prior Gaussian to a complex and form-free distribution to facilitate flexible inference of the temporal conditional distribution. Extensive experiments conducted on six real-world MTS datasets demonstrate the TCVAE's superior robustness and effectiveness over the state-of-the-art MTS forecasting baselines. We further illustrate the TCVAE applicability through multifaceted case studies and visualization in real-world scenarios.

Anomaly detection for multivariate times series through the multi-scale convolutional recurrent variational autoencoder

To realize the anomaly detection for industrial multi-sensor data, we develop a novel multi-scale convolutional recurrent variational autoencoder (MSCRVAE) model. It is a hybrid of convolutional autoencoder and convolutional long short-term memory with variational autoencoder (ConvLSTM-VAE). The ConvLSTM-VAE part helps the MSCRVAE model not only capture the spatial and temporal dependence of data but also learn robust high-level representations by estimating the distributions of latent variables. Moreover, a typical loss function that combines the characteristics of both autoencoder and variational autoencoder is designed for the MSCRVAE model. The experimental results on three public data illustrate the superiority of the MSCRVAE model in anomaly detection with the best average F1-score up to 0.90. The interquartile range of the boxplots on the public data also proves the robustness of the MSCRVAE model. We also illustrate the role of the two components in the loss function by exploring their changes during training. What’s more, on the private data, the MSCRVAE model also performs well and the residual matrices provide reasonable interpretations to the anomaly detection results.

Generalized Cramér’s coefficient via f-divergence for contingency tables

AbstractVarious measures in two-way contingency table analysis have been proposed to express the strength of association between row and column variables in contingency tables. Tomizawa et al. (2004) proposed more general measures, including Cramér’s coefficient, using the power-divergence. In this paper, we propose measures using the f-divergence that has a wider class than the power-divergence. Unlike statistical hypothesis tests, these measures provide quantification of the association structure in contingency tables. The contribution of our study is proving that a measure applying a function that satisfies the condition of the f-divergence has desirable properties for measuring the strength of association in contingency tables. With this contribution, we can easily construct a new measure using a divergence that has essential properties for the analyst. For example, we conducted numerical experiments with a measure applying the $$\theta$$ θ -divergence. Furthermore, we can give further interpretation of the association between the row and column variables in the contingency table, which could not be obtained with the conventional one. We also show a relationship between our proposed measures and the correlation coefficient in a bivariate normal distribution of latent variables in the contingency tables.

Predicting new drug indications based on double variational autoencoders

Experimental drug development is costly, complex, and time-consuming, and the number of drugs that have been put into application treatment is small. The identification of drug-disease correlations can provide important information for drug discovery and drug repurposing. Computational drug repurposing is an important and effective method that can be used to determine novel treatments for diseases. In recent years, an increasing number of large databases have been utilized for biological data research, particularly in the fields of drugs and diseases. Consequently, researchers have begun to explore the application of deep neural networks in biological data development. One particularly promising method for unsupervised learning is the deep generative model, with the variational autoencoder (VAE) being among the mainstream models. Here, we propose a drug indication prediction algorithm called DIDVAE (predicting new drug indications based on double variational autoencoders), which generates new data by learning the latent variable distribution of known data to achieve the goal of predicting drug-disease associations. In the experiment, we compared the DIDVAE algorithm with the BBNR, DrugNet, MBiRW and DRRS algorithms on a unified dataset. The comprehensive experimental results show that, compared with these prediction algorithms, the DIDVAE algorithm provides an overall improved prediction. In addition, further analysis and verification of the predicted unknown drug-disease association also proved the practicality of the method.

Bayesian Approach for Modelling Spatial–Temporal Crime Data

An important issue in many cities is related to crime events, and spatio–temporal Bayesian approach leads to identifying crime patterns and hotspots. In Bayesian analysis of spatio–temporal crime data, there is no closed form for posterior distribution because of its non-Gaussian distribution and existence of latent variables. In this case, we face different challenges such as high dimensional parameters, extensive simulation and time-consuming computation in applying MCMC methods. In this paper, we use INLA to analyze crime data in Colombia. The advantages of this method can be the estimation of criminal events at a specific time and location and exploring unusual patterns in places.

A Novel Fault Detection Method Based on One-Dimension Convolutional Adversarial Autoencoder (1DAAE)

Fault detection is an important and demanding problem in industry. Recently, many researchers have addressed the use of deep learning architectures for fault detection applications such as an autoencoder. Traditional methods based on an autoencoder usually complete fault detection by comparing reconstruction errors, and ignore a lot of useful information about the distribution of latent variables. To deal with this problem, this paper proposes a novel unsupervised fault detection method named one-dimension convolutional adversarial autoencoder (1DAAE), which introduces two new ideas: one-dimension convolution layers for the encoder to obtain better features and the adversarial thought to impose the latent variable z to cluster into a prior distribution. The proposed method not only has powerful feature representation ability than the traditional autoencoder, but has also enhanced the discrimination ability by imposing a prior distribution of the latent variables to cluster. Then, two anomaly scores for 1DAAE were proposed to detect fault samples, one based on reconstruction errors, and the other based on latent variable distribution. Finally, it was shown by the experiments that the proposed method outperformed the autoencoder-based, adversarial autoencoder-based, one-dimension convolutional autoencoder-based and generative adversarial network-based algorithms on the Tennessee Eastman process. Through the experiments, we found that the both one-dimension convolution layers and the latent vector distribution are helpful for fault detection.

Reconstruction of incomplete wildfire data using deep generative models

We present our submission to the Extreme Value Analysis 2021 Data Challenge in which teams were asked to accurately predict distributions of wildfire frequency and size within spatio-temporal regions of missing data. For this competition, we developed a variant of the powerful variational autoencoder models, which we call Conditional Missing data Importance-Weighted Autoencoder (CMIWAE). Our deep latent variable generative model requires little to no feature engineering and does not necessarily rely on the specifics of scoring in the Data Challenge. It is fully trained on incomplete data, with the single objective to maximize log-likelihood of the observed wildfire information. We mitigate the effects of the relatively low number of training samples by stochastic sampling from a variational latent variable distribution, as well as by ensembling a set of CMIWAE models trained and validated on different splits of the provided data.

State Causality and Adaptive Covariance Decomposition Based Time Series Forecasting

Time series forecasting is a very vital research topic. The scale of time series in numerous industries has risen considerably in recent years as a result of the advancement of information technology. However, the existing algorithms pay little attention to generating large-scale time series. This article designs a state causality and adaptive covariance decomposition-based time series forecasting method (SCACD). As an observation sequence, the majority of time series is generated under the influence of hidden states. First, SCACD builds neural networks to adaptively estimate the mean and covariance matrix of latent variables; Then, SCACD employs causal convolution to forecast the distribution of future latent variables; Lastly, to avoid loss of information, SCACD applies a sampling approach based on Cholesky decomposition to generate latent variables and observation sequences. Compared to existing outstanding time series prediction models on six real datasets, the model can achieve long-term forecasting while also being lighter, and the forecasting accuracy is improved in the great majority of the prediction tasks.

Counterfactual Sensitivity and Robustness

We propose a framework for analyzing the sensitivity of counterfactuals to parametric assumptions about the distribution of latent variables in structural models. In particular, we derive bounds on counterfactuals as the distribution of latent variables spans nonparametric neighborhoods of a given parametric specification while other “structural” features of the model are maintained. Our approach recasts the infinite‐dimensional problem of optimizing the counterfactual with respect to the distribution of latent variables (subject to model constraints) as a finite‐dimensional convex program. We also develop an MPEC version of our method to further simplify computation in models with endogenous parameters (e.g., value functions) defined by equilibrium constraints. We propose plug‐in estimators of the bounds and two methods for inference. We also show that our bounds converge to the sharp nonparametric bounds on counterfactuals as the neighborhood size becomes large. To illustrate the broad applicability of our procedure, we present empirical applications to matching models with transferable utility and dynamic discrete choice models.

Prediction of Water Diffusion in Wide Varieties of Polymers with All-Atom Molecular Dynamics Simulations and Deep Generative Models.

Permeation through polymer membranes is an important technology in the chemical industry, and in its design, the self-diffusion coefficient is one of the physical quantities that determine permeability. Since the self-diffusion coefficient sensitively reflects intra- and intermolecular interactions, analysis using an all-atom model is required. However, all-atom simulations are computationally expensive and require long simulation times for the diffusion of small molecules dissolved in polymers. MD-GAN, a machine learning model, is effective in accelerating simulations and reducing computational costs. The target systems for MD-GAN prediction were limited to polyethylene melts in previous studies; therefore, this study extended MD-GAN to systems containing copolymers with branches and successfully predicted water diffusion in various polymers. The correlation coefficient between the predicted self-diffusion coefficient and that of the long-time simulation was 1.00. Additionally, we found that incorporating statistical domain knowledge into MD-GAN improved accuracy, reducing the mean-square displacement prediction outliers from 14.6% to 5.3%. Lastly, the distribution of latent variables with embedded dynamics information within the model was found to be strongly related to accuracy. We believe that these findings can be useful for the practical applications of MD-GAN.

Segmentation in Structural Equation Modeling Using a Combination of Partial Least Squares and Modified Fuzzy Clustering

The application of a structural equation modeling (SEM) assumes that all data follow only one model. This assumption may be inaccurate in certain cases because individuals tend to differ in their responses, and failure to consider heterogeneity may threaten the validity of the SEM results. This study focuses on unobservable heterogeneity, where the difference between two or more data sets does not depend on observable characteristics. In this study, we propose a new method for estimating SEM parameters containing unobserved heterogeneity within the data and assume that the heterogeneity arises from the outer model and inner model. The method combines partial least squares (PLS) and modified fuzzy clustering. Initially, each observation was randomly assigned weights in each selected segment. These weights continued to be iteratively updated using a specific objective function. The sum of the weighted residual squares resulting from the outer and inner models of PLS-SEM is an objective function that must be minimized. We then conducted a simulation study to evaluate the performance of the method by considering various factors, including the number of segments, model specifications, residual variance of endogenous latent variables, residual variance of indicators, population size, and distribution of latent variables. From the simulation study and its application to the actual data, we conclude that the proposed method can classify observations into correct segments and precisely predict SEM parameters in each segment.

Generative population synthesis for joint household and individual characteristics

Household surveys provide immense value in the fields of transportation and urban planning. However, even the most well-funded surveying agencies rely on sampling methods to estimate the nature of the true population, and the collected microdata is frequently aggregated, or limited in volume and detail to protect the privacy of respondents. Population synthesis provides a means to scale this microdata to represent larger regions for use in microsimulation. Despite their accuracy and widespread adoption, traditional synthesis algorithms for reweighting microdata samples scale poorly with the number of variables and geographic regions being modeled, and can suffer from non-convergence with smaller sample sizes. Several generative models have been proposed to address these shortcomings, but lack features such as sub-region modeling, and the ability to simultaneously generate both individuals and households. This work proposes an extension to recent generative approaches capable of generating synthetic populations consisting of both individual and household-level variables, that uses a two-part Variational Autoencoder (VAE) and Conditional-VAE (CVAE) to learn a distribution of latent variables in the general population, and use them to generate new samples. This can help in synthesizing smaller, traditionally under-sampled groups. This approach is benchmarked against a state of the art open source population synthesizer. In addition, the VAE/CVAE model is tested under increasingly minimal training data. Findings indicate the VAE/CVAE model creates more accurate populations, using less time than the traditional synthesizer under small to medium dimensional datasets (4–16 variables). The VAE/CVAE also performs well with few (n = 100) training samples, with diminishing returns for additional training samples.

Grand Canonical Ensembles of Sparse Networks and Bayesian Inference.

Maximum entropy network ensembles have been very successful in modelling sparse network topologies and in solving challenging inference problems. However the sparse maximum entropy network models proposed so far have fixed number of nodes and are typically not exchangeable. Here we consider hierarchical models for exchangeable networks in the sparse limit, i.e., with the total number of links scaling linearly with the total number of nodes. The approach is grand canonical, i.e., the number of nodes of the network is not fixed a priori: it is finite but can be arbitrarily large. In this way the grand canonical network ensembles circumvent the difficulties in treating infinite sparse exchangeable networks which according to the Aldous-Hoover theorem must vanish. The approach can treat networks with given degree distribution or networks with given distribution of latent variables. When only a subgraph induced by a subset of nodes is known, this model allows a Bayesian estimation of the network size and the degree sequence (or the sequence of latent variables) of the entire network which can be used for network reconstruction.

Mix-VAEs: A novel multisensor information fusion model for intelligent fault diagnosis

Multisensor information are usually required to recognize the health condition of machinery by domain experts, since redundancy and complementarity of multisensor information can enhance robustness of fault diagnosis. However, the mainstream approaches of intelligent fault diagnosis based on deep neural networks only focus on applications of single sensor. In this paper, a novel intelligent fault diagnosis model using mixture of Gaussians (MoGs) and variational auto-encoders (VAEs), named Mix-VAEs, is proposed to leverage redundancy and complementarity of multisensor information. The proposed Mix-VAEs mainly contains two modules. First, feature extracting module is constructed by several parallel and independent VAEs which are used to learn data representation from multisensor data respectively, and their latent variable distributions extracted by VAEs are used for fusion. Second, in multisensor information fusion module, a novel fusion method is proposed where latent variable distributions are combined to form MoGs, and the fused features are obtained in a sampling manner from MoGs. The performance of the proposed model is verified through two datasets consisting of multisensor data. Experimental results demonstrate the robustness of the proposed model in two scenarios, sensor failure and local signal missing, while also showing the better performance than other fusion models.

Score Matching for Compositional Distributions

Compositional data are challenging to analyse due to the non-negativity and sum-to-one constraints on the sample space. With real data, it is often the case that many of the compositional components are highly right-skewed, with large numbers of zeros. Major limitations of currently available models for compositional data include one or more of the following: insufficient flexibility in terms of distributional shape; difficulty in accommodating zeros in the data in estimation; and lack of computational viability in moderate to high dimensions. In this article, we propose a new model, the polynomially tilted pairwise interaction (PPI) model, for analysing compositional data. Maximum likelihood estimation is difficult for the PPI model. Instead, we propose novel score matching estimators, which entails extending the score matching approach to Riemannian manifolds with boundary. These new estimators are available in closed form and simulation studies show that they perform well in practice. As our main application, we analyse real microbiome count data with fixed totals using a multinomial latent variable model with a PPI model for the latent variable distribution. We prove that, under certain conditions, the new score matching estimators are consistent for the parameters in the new multinomial latent variable model.

Estimating Distributions of Parameters in Nonlinear State Space Models with Replica Exchange Particle Marginal Metropolis-Hastings Method.

Extracting latent nonlinear dynamics from observed time-series data is important for understanding a dynamic system against the background of the observed data. A state space model is a probabilistic graphical model for time-series data, which describes the probabilistic dependence between latent variables at subsequent times and between latent variables and observations. Since, in many situations, the values of the parameters in the state space model are unknown, estimating the parameters from observations is an important task. The particle marginal Metropolis–Hastings (PMMH) method is a method for estimating the marginal posterior distribution of parameters obtained by marginalization over the distribution of latent variables in the state space model. Although, in principle, we can estimate the marginal posterior distribution of parameters by iterating this method infinitely, the estimated result depends on the initial values for a finite number of times in practice. In this paper, we propose a replica exchange particle marginal Metropolis–Hastings (REPMMH) method as a method to improve this problem by combining the PMMH method with the replica exchange method. By using the proposed method, we simultaneously realize a global search at a high temperature and a local fine search at a low temperature. We evaluate the proposed method using simulated data obtained from the Izhikevich neuron model and Lévy-driven stochastic volatility model, and we show that the proposed REPMMH method improves the problem of the initial value dependence in the PMMH method, and realizes efficient sampling of parameters in the state space models compared with existing methods.

Transfer learning for regression via latent variable represented conditional distribution alignment

Since labelled data are expensive to generate both computationally and experimentally, how to establish data-driven models with limited data has become an important challenge in various engineering and scientific applications. Transfer learning has been used to improve the performance of many tasks by leveraging rich labelled data from related domains. This paper focuses on the transfer learning problem for regression under the situation of conditional distribution shift, which is a common scenario in manufacturing industry and has not been fully studied. Hence, we first propose a Latent variable represented Conditional Distribution Alignment method (LCDA), which exploits the low-dimensional latent representation to describe the global conditional distribution discrepancy. Then the properties of the latent variables are optimised by matching the kernel embedding conditional distribution between domains, so that the global distribution discrepancy can be aligned with the residual function generated by the latent variables. By combining domain priors with machine learning, the proposed method provides a potential direction to reduce labelling consumption for the intelligent manufacturing. Series of experiments on two industrial applications are conducted to validate the effectiveness and analyse the characteristics of the proposed method.

Bayesian longitudinal item response modeling with multivariate asymmetric serial dependencies

It is usually impossible to impose experimental conditions in large-scale longitudinal (observational) studies in education. This increases the risk of bias due to for instance unobserved heterogeneity, missing background variables, and dropouts. A flexible statistical model is required for the nature of the observational assessment data and to account for the unexplained heterogeneity. A general class of longitudinal item response theory (IRT) models is proposed, where growth in performance can be monitored using a skewed multivariate normal distribution for the latent variables. Change in performance and unexplained heterogeneity is addressed through structured covariance patterns and skewed multivariate latent variable distributions. The Cholesky decomposition of the covariance matrix is considered to model the dependence structure. A novel multivariate skew-normal distribution is defined by the antedependence model with centered skew-normal distributed errors. A hybrid MCMC approach is developed for parameter estimation, model-fit assessment, and model comparison. Results of simulation studies show good parameter recovery. A longitudinal assessment study by the Brazilian federal government is considered to show the performance of the general LIRT model.