Variational Expectation-maximization Algorithm Research Articles

Probabilistic domain-knowledge modeling of disorder pathogenesis for dynamics forecasting of acute onset

Disease pathogenesis, a type of domain knowledge about biological mechanisms leading to diseases, has not been adequately encoded in machine-learning-based medical diagnostic models because of the inter-patient variabilities and complex dependencies of the underlying pathogenetic mechanisms. We propose 1) a novel pathogenesis probabilistic graphical model (PPGM) to quantify the dynamics underpinning patient-specific data and pathogenetic domain knowledge, 2) a Bayesian-based inference paradigm to answer the medical queries and forecast acute onsets. The PPGM model consists of two components: a Bayesian network of patient attributes and a temporal model of pathogenetic mechanisms. The model structure was reconstructed from expert knowledge elicitation, and its parameters were estimated using Variational Expectation-Maximization algorithms. We benchmarked our model with two well-established hidden Markov models (HMMs) – Input-output HMM (IO-HMM) and Switching Auto-Regressive HMM (SAR-HMM) – to evaluate the computational costs, forecasting performance, and execution time. Two case studies on Obstructive Sleep Apnea (OSA) and Paroxysmal Atrial Fibrillation (PAF) were used to validate the model. While the performance of the parameter learning step was equivalent to those of IO-HMM and SAR-HMM models, our model forecasting ability was outperforming those two models. The merits of the PPGM model are its representation capability to capture the dynamics of pathogenesis and perform medical inferences and its interpretability for physicians. The model has been used to perform medical queries and forecast the acute onset of OSA and PAF. Additional applications of the model include prognostic healthcare and preventive personalized treatments.

A representation learning model based on variational inference and graph autoencoder for predicting lncRNA-disease associations

BackgroundNumerous studies have demonstrated that long non-coding RNAs are related to plenty of human diseases. Therefore, it is crucial to predict potential lncRNA-disease associations for disease prognosis, diagnosis and therapy. Dozens of machine learning and deep learning algorithms have been adopted to this problem, yet it is still challenging to learn efficient low-dimensional representations from high-dimensional features of lncRNAs and diseases to predict unknown lncRNA-disease associations accurately.ResultsWe proposed an end-to-end model, VGAELDA, which integrates variational inference and graph autoencoders for lncRNA-disease associations prediction. VGAELDA contains two kinds of graph autoencoders. Variational graph autoencoders (VGAE) infer representations from features of lncRNAs and diseases respectively, while graph autoencoders propagate labels via known lncRNA-disease associations. These two kinds of autoencoders are trained alternately by adopting variational expectation maximization algorithm. The integration of both the VGAE for graph representation learning, and the alternate training via variational inference, strengthens the capability of VGAELDA to capture efficient low-dimensional representations from high-dimensional features, and hence promotes the robustness and preciseness for predicting unknown lncRNA-disease associations. Further analysis illuminates that the designed co-training framework of lncRNA and disease for VGAELDA solves a geometric matrix completion problem for capturing efficient low-dimensional representations via a deep learning approach.ConclusionCross validations and numerical experiments illustrate that VGAELDA outperforms the current state-of-the-art methods in lncRNA-disease association prediction. Case studies indicate that VGAELDA is capable of detecting potential lncRNA-disease associations. The source code and data are available at https://github.com/zhanglabNKU/VGAELDA.

Regularized LTI system identification in the presence of outliers: A variational EM approach

Regularized system identification of linear time invariant systems in the presence of outliers is investigated. The finite impulse response (fir) model and the Gaussian scale mixture are chosen to be the system model and the noise model, respectively. Two special cases of the noise model are considered: the well-known Student’s t distribution and a proposed G-confluent distribution. Both the fir model parameter and the latent variables in the noise model are treated as parameters of our statistical model and moreover, the scale of the noise variance is treated as a hyper-parameter besides the hyper-parameters used to parameterize the priors of the impulse response and the latent variables. Then a variational expectation–maximization algorithm is proposed for inference of the parameters and hyper-parameters of the statistical model, and the algorithm is guaranteed to converge to a stationary point. Monte Carlo numerical simulations show that when the relative size of outliers is small, the proposed approach performs comparably to a state-of-the-art method and when the relative size of outliers and/or the occurrence probability of outliers is large, the proposed approach outperforms the state-of-the-art method.

Greedy clustering of count data through a mixture of multinomial PCA

Count data is becoming more and more ubiquitous in a wide range of applications, with datasets growing both in size and in dimension. In this context, an increasing amount of work is dedicated to the construction of statistical models directly accounting for the discrete nature of the data. Moreover, it has been shown that integrating dimension reduction to clustering can drastically improve performance and stability. In this paper, we rely on the mixture of multinomial PCA, a mixture model for the clustering of count data, also known as the probabilistic clustering-projection model in the literature. Related to the latent Dirichlet allocation model, it offers the flexibility of topic modeling while being able to assign each observation to a unique cluster. We introduce a greedy clustering algorithm, where inference and clustering are jointly done by mixing a classification variational expectation maximization algorithm, with a branch & bound like strategy on a variational lower bound. An integrated classification likelihood criterion is derived for model selection, and a thorough study with numerical experiments is proposed to assess both the performance and robustness of the method. Finally, we illustrate the qualitative interest of the latter in a real-world application, for the clustering of anatomopathological medical reports, in partnership with expert practitioners from the Institut Curie hospital.

A stochastic block model for interaction lengths

We propose a new stochastic block model that focuses on the analysis of interaction lengths in dynamic networks. The model does not rely on a discretization of the time dimension and may be used to analyze networks that evolve continuously over time. The framework relies on a clustering structure on the nodes, whereby two nodes belonging to the same latent group tend to create interactions and non-interactions of similar lengths. We introduce a variational expectation–maximization algorithm to perform inference, and adapt a widely used clustering criterion to perform model choice. Finally, we validate our methodology using simulated data experiments and showing two illustrative applications concerning face-to-face interaction data and a bike sharing network.

Renormalization Analysis of Topic Models.

In practice, to build a machine learning model of big data, one needs to tune model parameters. The process of parameter tuning involves extremely time-consuming and computationally expensive grid search. However, the theory of statistical physics provides techniques allowing us to optimize this process. The paper shows that a function of the output of topic modeling demonstrates self-similar behavior under variation of the number of clusters. Such behavior allows using a renormalization technique. A combination of renormalization procedure with the Renyi entropy approach allows for quick searching of the optimal number of topics. In this paper, the renormalization procedure is developed for the probabilistic Latent Semantic Analysis (pLSA), and the Latent Dirichlet Allocation model with variational Expectation–Maximization algorithm (VLDA) and the Latent Dirichlet Allocation model with granulated Gibbs sampling procedure (GLDA). The experiments were conducted on two test datasets with a known number of topics in two different languages and on one unlabeled test dataset with an unknown number of topics. The paper shows that the renormalization procedure allows for finding an approximation of the optimal number of topics at least 30 times faster than the grid search without significant loss of quality.

Estimation in a binomial stochastic blockmodel for a weighted graph by a variational expectation maximization algorithm

Stochastic blockmodels have been widely proposed as a probabilistic random graph model for the analysis of networks data as well as for detecting community structure in these networks. In a number of real-world networks, not all ties among nodes have the same weight. Ties among networks nodes are often associated with weights that differentiate them in terms of their strength, intensity, or capacity. In this paper, we are interested in the case of co-citation networks, where the nodes are words and each edge joining a pair of words is weighted by the number of co-citation of these two words together in the same document. In this type of networks, the weight associated to each edge is an integer value bounded by the the whole number of documents in the considered corpus. Hence, we propose an extension of the stochastic blockmodels to deal with the case of a binomial distribution for the edge’s weights. We provide an inference method through a variational expectation maximization algorithm to estimate the parameters in binomial stochastic blockmodels for weighted networks. To prove the validity of the method and to highlight its main features, we set some applications of the proposed approach by using some simulated data and then some real data sets. Stochastic blockmodels belong to latent classes models. Classes defines a node’s clustering. We compare the clustering found through binomial stochastic blockmodels with the ones found fitting a stochastic blockmodel with Poisson distributed edges’ weights. Inferred Poisson and binomial stochastic blockmodels mainly differs. Moreover, in our examples, the statistical error is lower for binomial stochastic blockmodels.

Probabilistic Linear Discriminant Analysis Based on L1-Norm and Its Bayesian Variational Inference.

Probabilistic linear discriminant analysis (PLDA) is a very effective feature extraction approach and has obtained extensive and successful applications in supervised learning tasks. It employs the squared L2 -norm to measure the model errors, which assumes a Gaussian noise distribution implicitly. However, the noise in real-life applications may not follow a Gaussian distribution. Particularly, the squared L2 -norm could extremely exaggerate data outliers. To address this issue, this article proposes a robust PLDA model under the assumption of a Laplacian noise distribution, called L1-PLDA. The learning process employs the approach by expressing the Laplacian density function as a superposition of an infinite number of Gaussian distributions via introducing a new latent variable and then adopts the variational expectation-maximization (EM) algorithm to learn parameters. The most significant advantage of the new model is that the introduced latent variable can be used to detect data outliers. The experiments on several public databases show the superiority of the proposed L1-PLDA model in terms of classification and outlier detection.

Infinite ShapeOdds: Nonparametric Bayesian Models for Shape Representations

Learning compact representations for shapes (binary images) is important for many applications. Although neural network models are very powerful, they usually involve many parameters, require substantial tuning efforts and easily overfit small datasets, which are common in shape-related applications. The state-of-the-art approach, ShapeOdds, as a latent Gaussian model, can effectively prevent overfitting and is more robust. Nonetheless, it relies on a linear projection assumption and is incapable of capturing intrinsic nonlinear shape variations, hence may leading to inferior representations and structure discovery. To address these issues, we propose Infinite ShapeOdds (InfShapeOdds), a Bayesian nonparametric shape model, which is flexible enough to capture complex shape variations and discover hidden cluster structures, while still avoiding overfitting. Specifically, we use matrix Gaussian priors, nonlinear feature mappings and the kernel trick to generalize ShapeOdds to a shape-variate Gaussian process model, which can grasp various nonlinear correlations among the pixels within and across (different) shapes. To further discover the hidden structures in data, we place a Dirichlet process mixture (DPM) prior over the representations to jointly infer the cluster number and memberships. Finally, we exploit the Kronecker-product structure in our model to develop an efficient, truncated variational expectation-maximization algorithm for model estimation. On synthetic and real-world data, we show the advantage of our method in both representation learning and latent structure discovery.

A latent topic model with Markov transition for process data.

We propose a latent topic model with a Markov transition for process data, which consists of time-stamped events recorded in a log file. Such data are becoming more widely available in computer-based educational assessment with complex problem-solving items. The proposed model can be viewed as an extension of the hierarchical Bayesian topic model with a hidden Markov structure to accommodate the underlying evolution of an examinee's latent state. Using topic transition probabilities along with response times enables us to capture examinees' learning trajectories, making clustering/classification more efficient. A forward-backward variational expectation-maximization (FB-VEM) algorithm is developed to tackle the challenging computational problem. Useful theoretical properties are established under certain asymptotic regimes. The proposed method is applied to a complex problem-solving item in the 2012 version of the Programme for International Student Assessment (PISA).

TOPIC EXTRACTION OF CRAWLED DOCUMENTS COLLECTION USING CORRELATED TOPIC MODEL IN MAPREDUCE FRAMEWORK

The tremendous increase in the amount of available research documents impels researchers to propose topic models to extract the latent semantic themes of a documents collection. However, how to extract the hidden topics of the documents collection has become a crucial task for many topic model applications. Moreover, conventional topic modeling approaches suffer from the scalability problem when the size of documents collection increases. In this paper, the Correlated Topic Model with variational Expectation-Maximization algorithm is implemented in MapReduce framework to solve the scalability problem. The proposed approach utilizes the dataset crawled from the public digital library. In addition, the full-texts of the crawled documents are analysed to enhance the accuracy of MapReduce CTM. The experiments are conducted to demonstrate the performance of the proposed algorithm. From the evaluation, the proposed approach has a comparable performance in terms of topic coherences with LDA implemented in MapReduce framework.

Topic Extraction of Crawled Documents Collection using Correlated Topic Model in Mapreduce Framework

The tremendous increase in the amount of available research documents impels researchers to propose topic models to extract the latent semantic themes of a documents collection. However, how to extract the hidden topics of the documents collection has become a crucial task for many topic model applications. Moreover, conventional topic modeling approaches suffer from the scalability problem when the size of documents collection increases. In this paper, the Correlated Topic Model with variational Expectation-Maximization algorithm is implemented in MapReduce framework to solve the scalability problem. The proposed approach utilizes the dataset crawled from the public digital library. In addition, the full-texts of the crawled documents are analysed to enhance the accuracy of MapReduce CTM. The experiments are conducted to demonstrate the performance of the proposed algorithm. From the evaluation, the proposed approach has a comparable performance in terms of topic coherences with LDA implemented in MapReduce framework.

Bayesian weighted Mendelian randomization for causal inference based on summary statistics.

The results from Genome-Wide Association Studies (GWAS) on thousands of phenotypes provide an unprecedented opportunity to infer the causal effect of one phenotype (exposure) on another (outcome). Mendelian randomization (MR), an instrumental variable (IV) method, has been introduced for causal inference using GWAS data. Due to the polygenic architecture of complex traits/diseases and the ubiquity of pleiotropy, however, MR has many unique challenges compared to conventional IV methods. We propose a Bayesian weighted Mendelian randomization (BWMR) for causal inference to address these challenges. In our BWMR model, the uncertainty of weak effects owing to polygenicity has been taken into account and the violation of IV assumption due to pleiotropy has been addressed through outlier detection by Bayesian weighting. To make the causal inference based on BWMR computationally stable and efficient, we developed a variational expectation-maximization (VEM) algorithm. Moreover, we have also derived an exact closed-form formula to correct the posterior covariance which is often underestimated in variational inference. Through comprehensive simulation studies, we evaluated the performance of BWMR, demonstrating the advantage of BWMR over its competitors. Then we applied BWMR to make causal inference between 130 metabolites and 93 complex human traits, uncovering novel causal relationship between exposure and outcome traits. The BWMR software is available at https://github.com/jiazhao97/BWMR. Supplementary data are available at Bioinformatics online.

Variational EM method for blur estimation using the spike-and-slab image prior

Most state-of-the-art blind image deconvolution methods rely on the Bayesian paradigm to model the deblurring problem and estimate both the blur kernel and latent image. It is customary to model the image in the filter space, where it is supposed to be sparse, and utilize convenient priors to account for this sparsity. In this paper, we propose the use of the spike-and-slab prior together with an efficient variational Expectation Maximization (EM) inference scheme to estimate the blur in the image. The spike-and-slab prior, which constitutes the gold standard in sparse machine learning, selectively shrinks irrelevant variables while mildly regularizing the relevant ones. The proposed variational Expectation Maximization algorithm is more efficient than usual Markov Chain Monte Carlo (MCMC) inference and, also, proves to be more accurate than the standard mean-field variational approximation. Additionally, all the prior model parameters are estimated by the proposed scheme. After blur estimation, a non-blind restoration method is used to obtain the actual estimation of the sharp image. We investigate the behavior of the prior in the experimental section together with a series of experiments with synthetically generated and real blurred images that validate the method's performance in comparison with state-of-the-art blind deconvolution techniques.

Multitarget Localization With Inaccurate Sensor Locations via Variational EM Algorithm

In wireless sensor networks, target localization has been the focus of considerable research effort, and the compressive sensing (CS) based localization method is of particular interest. However, most existing works usually assume that the positions of sensors are known exactly, while in practice, they may not be accurate. When the assumption is violated, the localization performance will deteriorate dramatically. In this article, we propose a novel CS-based multiple target localization method using the superimposed received signal strength, measured by sensors with inaccurate positions. In order to address such issues, we regard the known but inaccurate sensor locations as adjustable parameters. Accordingly, the sensor positions can be refined through the adjustment of parameters. As a result, the problem is reformulated as a joint sparse signal estimation and parameter optimization task. Then, the variational expectation-maximization (EM) algorithm and the subspace trust-region method are applied to iteratively estimate the unknown target locations and refine inaccurate sensor locations. Simulation results are included to confirm the superior localization performance of the proposed algorithm by comparing with the state-of-the-art methods.

Low-Rank Phase Retrieval via Variational Bayesian Learning

In this paper, we consider the problem of low-rank phase retrieval whose objective is to estimate a complex low-rank matrix from magnitude-only measurements. We propose a hierarchical prior model for low-rank phase retrieval, in which a Gaussian-Wishart hierarchical prior is placed on the underlying low-rank matrix to promote the low-rankness of the matrix. Based on the proposed hierarchical model, a variational expectation-maximization (EM) algorithm is developed. The proposed method is less sensitive to the choice of the initialization point and works well with random initialization. Simulation results are provided to illustrate the effectiveness of the proposed algorithm.

Spectral Inference for Large Stochastic Blockmodels With Nodal Covariates

In many applications of network analysis, it is important to distinguish between observed and unobserved factors affecting network structure. To this end, we develop spectral estimators for both unobserved blocks and the effect of covariates in stochastic blockmodels. On the theoretical side, we establish asymptotic normality of our estimators for the subsequent purpose of performing inference. On the applied side, we show that computing our estimator is much faster than standard variational expectation-maximization algorithms and scales well for large networks. The results in this paper provide a foundation to estimate the effect of observed covariates as well as unobserved latent community structure on the probability of link formation in networks.

Tracking Multiple Audio Sources With the von Mises Distribution and Variational EM

In this paper we address the problem of simultaneously tracking several moving audio sources, namely the problem of estimating source trajectories from a sequence of observed features. We propose to use the von Mises distribution to model audio-source directions of arrival with circular random variables. This leads to a Bayesian filtering formulation which is intractable because of the combinatorial explosion of associating observed variables with latent variables, over time. We propose a variational approximation of the filtering distribution. We infer a variational expectation-maximization algorithm that is both computationally tractable and time efficient. We propose an audio-source birth method that favors smooth source trajectories and which is used both to initialize the number of active sources and to detect new sources. We perform experiments with the recently released LOCATA dataset comprising two moving sources and a moving microphone array mounted onto a robot.

Bayesian estimation of generalized Gamma mixture model based on variational EM algorithm

In this paper, we propose a Bayesian inference method for the generalized Gamma mixture model (GΓMM) based on variational expectation-maximization algorithm. Specifically, the shape parameters, the inverse scale parameters, and the mixing coefficients in the GΓMM are treated as random variables, while the power parameters are left as parameters without assigning prior distributions. The help function is designed to approximate the lower bound of the variational objective function, which facilitates the assignment of the conjugate prior distributions and leads to the closed-form update equations. On this basis, the variational E-step and the variational M-step are alternatively implemented to infer the posteriors of the variables and estimate the parameters. The computational demand is reduced by the proposed method. More importantly, the effective number of components of the GΓMM can be determined automatically. The experimental results demonstrate the effectiveness of the proposed method especially in modeling the asymmetric and heavy-tailed data.

The dynamic stochastic topic block model for dynamic networks with textual edges

The present paper develops a probabilistic model to cluster the nodes of a dynamic graph, accounting for the content of textual edges as well as their frequency. Ver-tices are clustered in groups which are homogeneous both in terms of interaction frequency and discussed topics. The dynamic graph is considered stationary on a latent time interval if the proportions of topics discussed between each pair of node groups do not change in time during that interval. A classification variational expectation-maximization (C-VEM) algorithm is adopted to perform inference. A model selection criterion is also derived to select the number of node groups, time clusters and topics. Experiments on simulated data are carried out to assess the proposed methodology. We finally illustrate an application to the Enron dataset.