Variational Bayes Framework Research Articles

A Deep Probabilistic Transfer Learning Framework for Soft Sensor Modeling With Missing Data.

Soft sensors have been extensively developed and applied in the process industry. One of the main challenges of the data-driven soft sensors is the lack of labeled data and the need to absorb the knowledge from a related source operating condition to enhance the soft sensing performance on the target application. This article introduces deep transfer learning to soft sensor modeling and proposes a deep probabilistic transfer regression (DPTR) framework. In DPTR, a deep generative regression model is first developed to learn Gaussian latent feature representations and model the regression relationship under the stochastic gradient variational Bayes framework. Then, a probabilistic latent space transfer strategy is designed to reduce the discrepancy between the source and target latent features such that the knowledge from the source data can be explored and transferred to enhance the target soft sensor performance. Besides, considering the missing values in the process data in the target operating condition, the DPTR is further extended to handle the missing data problem utilizing the strong generation and reconstruction capability of the deep generative model. The effectiveness of the proposed method is validated through an industrial multiphase flow process.

Competing Mutual Information Constraints with Stochastic Competition-Based Activations for Learning Diversified Representations

This work aims to address the long-established problem of learning diversified representations. To this end, we combine information-theoretic arguments with stochastic competition-based activations, namely Stochastic Local Winner-Takes-All (LWTA) units. In this context, we ditch the conventional deep architectures commonly used in Representation Learning, that rely on non-linear activations; instead, we replace them with sets of locally and stochastically competing linear units. In this setting, each network layer yields sparse outputs, determined by the outcome of the competition between units that are organized into blocks of competitors. We adopt stochastic arguments for the competition mechanism, which perform posterior sampling to determine the winner of each block. We further endow the considered networks with the ability to infer the sub-part of the network that is essential for modeling the data at hand; we impose appropriate stick-breaking priors to this end. To further enrich the information of the emerging representations, we resort to information-theoretic principles, namely the Information Competing Process (ICP). Then, all the components are tied together under the stochastic Variational Bayes framework for inference. We perform a thorough experimental investigation for our approach using benchmark datasets on image classification. As we experimentally show, the resulting networks yield significant discriminative representation learning abilities. In addition, the introduced paradigm allows for a principled investigation mechanism of the emerging intermediate network representations.

Heterogeneous Supervised Topic Models

AbstractResearchers in the social sciences are often interested in the relationship between text and an outcome of interest, where the goal is to both uncover latent patterns in the text and predict outcomes for unseen texts. To this end, this paper develops the heterogeneous supervised topic model (HSTM), a probabilistic approach to text analysis and prediction. HSTMs posit a joint model of text and outcomes to find heterogeneous patterns that help with both text analysis and prediction. The main benefit of HSTMs is that they capture heterogeneity in the relationship between text and the outcome across latent topics. To fit HSTMs, we develop a variational inference algorithm based on the auto-encoding variational Bayes framework. We study the performance of HSTMs on eight datasets and find that they consistently outperform related methods, including fine-tuned black-box models. Finally, we apply HSTMs to analyze news articles labeled with pro- or anti-tone. We find evidence of differing language used to signal a pro- and anti-tone.

Joint Topology Learning and Graph Signal Recovery Using Variational Bayes in Non-Gaussian Noise

This brief proposes a joint graph signal recovery and topology learning algorithm using a Variational Bayes (VB) framework in the case of non-Gaussian measurement noise. It is assumed that the graph signal is Gaussian Markov Random Field (GMRF) and the graph weights are considered statistical with the Gaussian prior. Moreover, the non-Gaussian noise is modeled using two distributions: Mixture of Gaussian (MoG), and Laplace. All the unknowns of the problem which are graph signal, Laplacian matrix, and the (Hyper)parameters are estimated by a VB framework. All the posteriors are calculated in closed forms and the iterative VB algorithm is devised to solve the problem. The efficiency of the proposed algorithm in comparison to some state-of-the-art algorithms in the literature is shown in the simulation results.

Composite Filtering for UWB-Based Localization of Quadrotor UAV With Skewed Measurements and Uncertain Dynamics

Nowadays, the ultrawideband (UWB) has become a popular solution to the indoor localization problem of quadrotor unmanned aerial vehicle (UAV) due to its low power consumption and ease of implementation. Nonetheless, the accuracy of UAV localization is affected by the skew- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$t$ </tex-math></inline-formula> measurement noises of the UWB sensor induced mainly by the nonline-of-sight (NLOS) errors and the multipath effects, and the uncertainties in the UAV dynamics caused by external wind disturbances. In this article, a composite filtering approach is proposed for time difference of arrival (TDOA)-based UWB localization of the quadrotor UAV. The main advantage of the composite filtering approach lies in its capability of dealing with the skew- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$t$ </tex-math></inline-formula> measurement noises and the dynamic uncertainties simultaneously. Specifically, the external winds are estimated and rejected in real time via the disturbance observer (DO). Meanwhile, the posterior state distribution under skew- <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$t$ </tex-math></inline-formula> measurement noises is updated in an iterative fashion within the variational Bayes (VB) framework. The effectiveness of the proposed approach is validated through both numerical simulation and experimental tests on a quadrotor UAV.

Multipartition clustering of mixed data with Bayesian networks

Real-world applications often involve multifaceted data with several reasonable interpretations. To cluster this data, we need methods that are able to produce multiple clustering solutions. To this purpose, it is interesting to learn a finite mixture model with multiple latent variables, where each latent variable represents a unique way to partition the data. However, although there is an extensive literature on multipartition clustering methods for categorical data and for continuous data, there is a lack of work for mixed data. In this paper, we propose a multipartition clustering method that is able to efficiently deal with mixed data by exploiting the Bayesian network factorization and the variational Bayes framework. We show the flexibility and applicability of the proposed method by solving clustering, density estimation, and missing data imputation tasks in real-world data sets. For reproducibility, all code, data, and results can be found in the following public repository: https://github.com/ferjorosa/mpc-mixed.

Integrating sample similarities into latent class analysis: a tree-structured shrinkage approach.

This paper is concerned with using multivariate binary observations to estimate the probabilities of unobserved classes with scientific meanings. We focus on the setting where additional information about sample similarities is available and represented by a rooted weighted tree. Every leaf in the given tree contains multiple samples. Shorter distances over the tree between the leaves indicate a priori higher similarity in class probability vectors. We propose a novel data integrative extension to classical latent class models with tree-structured shrinkage. The proposed approach enables (1) borrowing of information across leaves, (2) estimating data-driven leaf groups with distinct vectors of class probabilities, and (3) individual-level probabilistic class assignment given the observed multivariate binary measurements. We derive and implement a scalable posterior inference algorithm in a variational Bayes framework. Extensive simulations show more accurate estimation of class probabilities than alternatives that suboptimally use the additional sample similarity information. A zoonotic infectious disease application is used to illustrate the proposed approach. The paper concludes by a brief discussion on model limitations andextensions.

Identification of Wiener–Hammerstein models based on variational bayesian approach in the presence of process noise

This paper addresses the identification of Wiener–Hammerstein (WH) models in the presence of process and measurement noises, which has not been well studied yet in the existing works. To achieve an unbiased estimation, the model parameters are obtained by maximizing the likelihood function, which is solved in the expectation-maximization framework. Due to the difficulty of computing the posterior distributions of the latent variables of WH models, variational Bayes (VB) is used here, and a method for approximating the posterior distributions based on Monte Carlo integral is proposed in VB framework. To the best of our knowledge, it is the first time to use VB approach for WH model identification. Two simulation examples demonstrate the effectiveness of the proposed method. Moreover, the proposed method is used for a WH benchmark problem, and the results show that it improves the identification performance.

Testing Homogeneity for Normal Mixture Models: Variational Bayes Approach

The test of homogeneity for normal mixtures has been conducted in diverse research areas, but constructing a theory of the test of homogeneity is challenging because the parameter set for the null hypothesis corresponds to singular points in the parameter space. In this paper, we examine this problem from a new perspective and offer a theory of hypothesis testing for homogeneity based on a variational Bayes framework. In the conventional theory, the constant order term of the free energy has remained unknown, however, we clarify its asymptotic behavior because it is necessary for constructing a hypothesis test. Numerical experiments shows the validity of our theoretical results.

A framework for integrating directed and undirected annotations to build explanatory models of cis-eQTL data.

A longstanding goal of regulatory genetics is to understand how variants in genome sequences lead to changes in gene expression. Here we present a method named Bayesian Annotation Guided eQTL Analysis (BAGEA), a variational Bayes framework to model cis-eQTLs using directed and undirected genomic annotations. We used BAGEA to integrate directed genomic annotations with eQTL summary statistics from tissues of various origins. This analysis revealed epigenetic marks that are relevant for gene expression in different tissues and cell types. We estimated the predictive power of the models that were fitted based on directed genomic annotations. This analysis showed that, depending on the underlying eQTL data used, the directed genomic annotations could predict up to 1.5% of the variance observed in the expression of genes with top nominal eQTL association p-values < 10-7. For genes with estimated effect sizes in the top 25% quantile, up to 5% of the expression variance could be predicted. Based on our results, we recommend the use of BAGEA for the analysis of cis-eQTL data to reveal annotations relevant to expression biology.

Interpretable factor models of single-cell RNA-seq via variational autoencoders.

MotivationSingle-cell RNA-seq makes possible the investigation of variability in gene expression among cells, and dependence of variation on cell type. Statistical inference methods for such analyses must be scalable, and ideally interpretable.ResultsWe present an approach based on a modification of a recently published highly scalable variational autoencoder framework that provides interpretability without sacrificing much accuracy. We demonstrate that our approach enables identification of gene programs in massive datasets. Our strategy, namely the learning of factor models with the auto-encoding variational Bayes framework, is not domain specific and may be useful for other applications.Availability and implementationThe factor model is available in the scVI package hosted at https://github.com/YosefLab/scVI/.Contact v@nxn.se Supplementary information Supplementary data are available at Bioinformatics online.

Adaptive group-regularized logistic elastic net regression.

SummaryIn high-dimensional data settings, additional information on the features is often available. Examples of such external information in omics research are: (i) n}{}p-values from a previous study and (ii) omics annotation. The inclusion of this information in the analysis may enhance classification performance and feature selection but is not straightforward. We propose a group-regularized (logistic) elastic net regression method, where each penalty parameter corresponds to a group of features based on the external information. The method, termed gren, makes use of the Bayesian formulation of logistic elastic net regression to estimate both the model and penalty parameters in an approximate empirical–variational Bayes framework. Simulations and applications to three cancer genomics studies and one Alzheimer metabolomics study show that, if the partitioning of the features is informative, classification performance, and feature selection are indeed enhanced.

A stochastic variational framework for Recurrent Gaussian Processes models

Gaussian Processes (GPs) models have been successfully applied to the problem of learning from sequential observations. In such context, the family of Recurrent Gaussian Processes (RGPs) have been recently introduced with a specifically designed structure to handle dynamical data. However, RGPs present a limitation shared by most GP approaches: they become computationally infeasible when facing very large datasets. In the present work, with the aim of improving scalability, we modify the original variational approach used with RGPs in order to enable inference via stochastic mini-batch optimization, giving rise to the Stochastic Recurrent Variational Bayes (S-REVARB) framework. We review recent related literature and comprehensively contextualize it with our approach. Moreover, we propose two learning procedures, the Local and Global S-REVARB algorithms, which prevent computational costs from scaling with the number of training samples. The global variant permits even greater scalability by also preventing the number of variational parameters from increasing with the training set, through the use of neural networks as sequential recognition models. The proposed framework is evaluated in the task of dynamical system identification for large scale datasets, a scenario not readily supported by the standard batch inference for RGPs. The promising results indicate that the S-REVARB framework opens up the possibility of applying powerful hierarchical recurrent GP-based models to massive sequential data.

Simultaneous clustering and feature selection via nonparametric Pitman–Yor process mixture models

Mixture models constitute one of the most important machine learning approaches. Indeed, they can be considered as the workhorse of generative machine learning. The majority of existing works consider mixtures of Gaussians. Unlike these works, this paper concentrates on nonparametric Bayesian models with Dirichlet-based mixtures. In particular, we consider the case when a Pitman–Yor process prior is adopted. Two central problems when considering such mixtures can be regarded as selecting ‘meaningful’ (or relevant) features and estimating the model’s parameters. We develop an efficient algorithm for model inference, based on the collapsed variational Bayes framework with 0th-order Taylor approximation. The merits and efficacy of the proposed nonparametric Bayesian model are demonstrated via challenging applications that concern real-world data clustering and 3D objects recognition.

Bayesian denoising hashing for robust image retrieval

Learning to hash is one of the most popular techniques in image retrieval, but few work investigates its robustness to noise corrupted images in which the unknown pattern of noise would heavily deteriorate the performance. To deal with this issue, we present in this paper a Bayesian denoising hashing algorithm whose output can be regarded a denoised version of the input hash code. We show that our method essentially seeks to reconstruct a new but more robust hash code by preserving the original input information while imposing extra constraints so as to correct the corrupted bits. We optimized this model in variational Bayes framework which has a closed-form update in each iteration that is more efficient than numerical optimization. Furthermore, our method can be added at the top of any original hashing layer, serving as a post-processing denoising layer with no change to previous training procedure. Experiments on three popular datasets demonstrate that the proposed method yields robust and meaningful hash code, which significantly improves the performance of state-of-the-art hash learning methods on challenging tasks such as large-scale natural image retrieval and retrieval with corrupted images.

Goodness of Fit of Logistic Regression Models for Random Graphs

ABSTRACTLogistic regression is a natural and simple tool to understand how covariates contribute to explain the topology of a binary network. Once the model is fitted, the practitioner is interested in the goodness of fit of the regression to check if the covariates are sufficient to explain the whole topology of the network and, if they are not, to analyze the residual structure. To address this problem, we introduce a generic model that combines logistic regression with a network-oriented residual term. This residual term takes the form of the graphon function of a W-graph. Using a variational Bayes framework, we infer the residual graphon by averaging over a series of blockwise constant functions. This approach allows us to define a generic goodness-of-fit criterion, which corresponds to the posterior probability for the residual graphon to be constant. Experiments on toy data are carried out to assess the accuracy of the procedure. Several networks from social sciences and ecology are studied to illustrate the proposed methodology. Supplementary material for this article is available online.

Sparse Bayesian registration of medical images for self-tuning of parameters and spatially adaptive parametrization of displacements.

We extend Bayesian models of non-rigid image registration to allow not only for the automatic determination of registration parameters (such as the trade-off between image similarity and regularization functionals), but also for a data-driven, multiscale, spatially adaptive parametrization of deformations. Adaptive parametrizations have been used with success to promote both the regularity and accuracy of registration schemes, but so far on non-probabilistic grounds - either as part of multiscale heuristics, or on the basis of sparse optimization. Under the proposed model, a sparsity-inducing prior on transformation parameters complements the classical smoothness-inducing prior, and favors parametrizations that use few degrees of freedom. As a result, finer bases get introduced only in the presence of coherent image information and motion, while coarser bases ensure better extrapolation of the motion to textureless, uninformative regions. The space of possible parametrizations consists of arbitrary combinations of basis functions chosen among any preset, widely overcomplete (and typically multiscale) dictionary. Inference is tackled in an efficient Variational Bayes framework. In addition we propose a flexible mixture-of-Gaussian model of data that proves to be more faithful for a variety of image modalities than the sum-of-squared differences. The performance of the proposed approach is demonstrated on time series of (cine and tagged) magnetic resonance and echocardiographic cardiac images. The proposed algorithm matches the state-of-the-art on benchmark datasets evaluating accuracy of motion and strain, and is highly automated.

Quantifying Registration Uncertainty With Sparse Bayesian Modelling.

We investigate uncertainty quantification under a sparse Bayesian model of medical image registration. Bayesian modelling has proven powerful to automate the tuning of registration hyperparameters, such as the trade-off between the data and regularization functionals. Sparsity-inducing priors have recently been used to render the parametrization itself adaptive and data-driven. The sparse prior on transformation parameters effectively favors the use of coarse basis functions to capture the global trends in the visible motion while finer, highly localized bases are introduced only in the presence of coherent image information and motion. In earlier work, approximate inference under the sparse Bayesian model was tackled in an efficient Variational Bayes (VB) framework. In this paper we are interested in the theoretical and empirical quality of uncertainty estimates derived under this approximate scheme vs. under the exact model. We implement an (asymptotically) exact inference scheme based on reversible jump Markov Chain Monte Carlo (MCMC) sampling to characterize the posterior distribution of the transformation and compare the predictions of the VB and MCMC based methods. The true posterior distribution under the sparse Bayesian model is found to be meaningful: orders of magnitude for the estimated uncertainty are quantitatively reasonable, the uncertainty is higher in textureless regions and lower in the direction of strong intensity gradients.

Variational Bayes Group Sparse Time-Adaptive Parameter Estimation With Either Known or Unknown Sparsity Pattern

In this paper, we study the problem of time-adaptive group sparse signal estimation from a Bayesian viewpoint. We propose two online variational Bayes schemes that are specifically designed to estimate and track group sparse signals in time. The proposed schemes address both the cases where the grouping information of the signal is either known or not. For the case of known group sparsity pattern, the proposed scheme builds on a novel hierarchical model for the Bayesian adaptive group lasso. Utilizing the variational Bayes framework, update equations for all model parameters are given, for both the batch and time adaptive estimation scenarios. To address the case where the group sparsity pattern is unknown, the hierarchical Bayesian model of the former scheme is extended by organizing the penalty parameters of the Bayesian lasso in a conditional autoregressive model. Intrinsic conditional autoregression is exploited to penalize the signal coefficients in a structured manner and thus obtain group sparse solutions automatically. Again, a robust and computationally efficient online variational Bayes estimator is developed, capitalizing on the conjugacy of the proposed hierarchical Bayesian formulation. Experimental results are reported that corroborate the superior estimation performance of the proposed online schemes, when compared with state-of-the-art methods.

Memory and Computation Trade-Offs for Efficient I-Vector Extraction

This work aims at reducing the memory demand of the data structures that are usually pre-computed and stored for fast computation of the i-vectors, a compact representation of spoken utterances that is used by most state-of-the-art speaker recognition systems. We propose two new approaches allowing accurate i-vector extraction but requiring less memory, showing their relations with the standard computation method introduced for eigenvoices, and with the recently proposed fast eigen-decomposition technique. The first approach computes an i-vector in a Variational Bayes (VB) framework by iterating the estimation of one sub-block of i-vector elements at a time, keeping fixed all the others, and can obtain i-vectors as accurate as the ones obtained by the standard technique but requiring only 25% of its memory. The second technique is based on the Conjugate Gradient solution of a linear system, which is accurate and uses even less memory, but is slower than the VB approach. We analyze and compare the time and memory resources required by all these solutions, which are suited to different applications, and we show that it is possible to get accurate results greatly reducing memory demand compared with the standard solution at almost the same speed.