Variational Expectation-maximization Algorithm Research Articles

Maximum likelihood estimation for discrete latent variable models via evolutionary algorithms

We propose an evolutionary optimization method for maximum likelihood and approximate maximum likelihood estimation of discrete latent variable models. The proposal is based on modified versions of the expectation–maximization (EM) and variational EM (VEM) algorithms, which are based on the genetic approach and allow us to accurately explore the parameter space, reducing the chance to be trapped into one of the multiple local maxima of the log-likelihood function. Their performance is examined through an extensive Monte Carlo simulation study where they are employed to estimate latent class, hidden Markov, and stochastic block models and compared with the standard EM and VEM algorithms. We observe a significant increase in the chance to reach global maximum of the target function and a high accuracy of the estimated parameters for each model. Applications focused on the analysis of cross-sectional, longitudinal, and network data are proposed to illustrate and compare the algorithms.

A Note on Improving Variational Estimation for Multidimensional Item Response Theory.

Survey instruments and assessments are frequently used in many domains of social science. When the constructs that these assessments try to measure become multifaceted, multidimensional item response theory (MIRT) provides a unified framework and convenient statistical tool for item analysis, calibration, and scoring. However, the computational challenge of estimating MIRT models prohibits its wide use because many of the extant methods can hardly provide results in a realistic time frame when the number of dimensions, sample size, and test length are large. Instead, variational estimation methods, such as Gaussian variational expectation-maximization (GVEM) algorithm, have been recently proposed to solve the estimation challenge by providing a fast and accurate solution. However, results have shown that variational estimation methods may produce some bias on discrimination parameters during confirmatory model estimation, and this note proposes an importance-weighted version of GVEM (i.e., IW-GVEM) to correct for such bias under MIRT models. We also use the adaptive moment estimation method to update the learning rate for gradient descent automatically. Our simulations show that IW-GVEM can effectively correct bias with modest increase of computation time, compared with GVEM. The proposed method may also shed light on improving the variational estimation for other psychometrics models.

Modeling and monitoring multilayer attributed weighted directed networks via a generative model

As data with network structures are widely seen in diverse applications, the modeling and monitoring of network data have drawn considerable attention in recent years. When individuals in a network have multiple types of interactions, a multilayer network model should be considered to better characterize its behavior. Most existing network models have concentrated on characterizing the topological structure among individuals, and important attributes of individuals are largely disregarded in existing works. In this article, first, we propose a unified static Network Generative Model (static-NGM), which incorporates individual attributes in network topology modeling. The proposed model can be utilized for a general multilayer network with weighted and directed edges. A variational expectation maximization algorithm is developed to estimate model parameters. Second, to characterize the time-dependent property of a network sequence and perform network monitoring, we extend the static-NGM model to a sequential version, namely, the sequential-NGM model, with the Markov assumption. Last, a sequential-NGM chart is developed to detect shifts and identify root causes of shifts in a network sequence. Extensive simulation experiments show that considering attributes improves the parameter estimation accuracy and that the proposed monitoring method also outperforms the three competitive approaches, static-NGM chart, score test-based chart (ST chart) and Bayes factor-based chart (BF chart), in both shift detection and root cause diagnosis. We also perform a case study with Enron E-mail data; the results further validate the proposed method.

Feature Selection and Damage Identification for Urban Railway Track Using Bayesian Globally Sparse Principal Component Analysis

Urban railway track infrastructures often suffer from damage that affects their service performance due to a variety of factors. In this study, an unsupervised feature selection and damage identification method based on globally sparse probabilistic principal component analysis (PCA) is proposed for urban railway tracks using the monitoring data of train-induced dynamic responses. A Bayesian framework is proposed for generating principal components on a basis of vectors (original variables) with a global sparseness pattern instead of separate patterns in a traditional sparse PCA. In this framework, a variational expectation-maximization algorithm is employed to obtain the tractable calculation of the marginal likelihood function for learning all uncertain parameters of the Bayesian model. The obtained principal components are linear combinations of the very same set of important variables, making our method better interpretable than the traditional sparse PCA. We can clearly understand which original variables are most relevant for describing the data. The track damage is identified simply by discriminating the corresponding measured dynamic responses using the binary elements of the latent vector inferred from the Bayesian globally sparse PCA algorithm. The usefulness is demonstrated by successfully identifying the track bed plate crack damage through the actual train-induced dynamic responses collected from the structural health monitoring system of an urban railway track infrastructure, where the method is able to achieve F1 scores of 90% or higher for various scenarios.

Modeling comorbidity of chronic diseases using coupled hidden Markov model with bivariate discrete copula.

A range of chronic diseases have a significant influence on each other and share common risk factors. Comorbidity, which shows the existence of two or more diseases interacting or triggering each other, is an important measure for actuarial valuations. The main proposal of the study is to model parallel interacting processes describing two or more chronic diseases by a combination of hidden Markov theory and copula function. This study introduces a coupled hidden Markov model with the bivariate discrete copula function in the hidden process. To estimate the parameters of the model and deal with the numerical intractability of the log-likelihood, we use a variational expectation maximization algorithm. To perform the variational expectation maximization algorithm, a lower bound of the model's log-likelihood is defined, and estimators of the parameters are computed in the M-part. A possible numerical underflow occurring in the computation of forward-backward probabilities is solved. The simulation study is conducted for two different levels of association to assess the performance of the proposed model, resulting in satisfactory findings. The proposed model was applied to hospital appointment data from a private hospital. The model defines the dependency structure of unobserved disease data and its dynamics. The application results demonstrate that the model is useful for investigating disease comorbidity when only population dynamics over time and no clinical data are available.

Channel estimation for underwater acoustic OFDM communications via temporal sparse Bayesian learning

Channel estimation is a challenging task for underwater acoustic (UWA) communications. In this paper, we investigate the channel estimation for OFDM based UWA communications by utilizing the channel characteristics as auxiliary information. Firstly, the channel estimation problem is formulated as a sparse signal recovery problem using block-wise single measurement vector (SMV) model and multiple measurement vector (MMV) model. Then we develop a novel channel estimators under the temporal sparse Bayesian learning (TSBL) framework by jointly exploiting the temporal correlation and sparsity based on the block-wise SMV model. For the MMV model, a variational Bayesian inference based TSBL (VBI-TSBL) method is introduced that considers the inter-block correlations and exploits the row-sparsity by a linear temporal correlation model. The two Bayesian inference methods realize the posterior and parameter estimation of channel via expectation maximization (EM) and variational EM algorithm, respectively. Simulation and sea-trial experiments show the proposed channel estimators outperform the benchmarks in terms of bit error rate, mean square error of channel estimation and the robustness.

Anomaly detection for fabricated artifact by using unstructured 3D point cloud data

3D point cloud data has been widely used in surface quality inspection to measure fabricated artifacts, allowing the high density and precision of measurements and providing quantitative 3D geometric characteristics for anomalies. Unlike structured 3D point cloud data, unstructured 3D point cloud data can capture the surface geometry completely. However, anomaly detection by using unstructured 3D point cloud data is more challenging, due to the nonexistence of global coordinate ordering and the difficulty of mathematically modeling anomalies and discriminating outliers. To deal with these challenges, this article formulates the anomaly detection problem into a probabilistic framework. By categorizing points into three types, i.e., reference surface point, anomaly point, and outlier point, a novel Bayesian network is proposed to model the unstructured 3D point cloud data. The variational expectation-maximization algorithm is used to estimate parameters and make inference on the unknown types of points. Both simulation and real case studies demonstrate the accuracy and robustness of the proposed method.

Dirichlet process mixture of Gaussian process functional regressions and its variational EM algorithm

Gaussian Process Functional Regression (GPFR) is a powerful tool in functional data analysis. In practical applications, functional data may be generated from different signal sources, and a single GPFR is not flexible enough to accurately model the data. To tackle the heterogeneity problem, a finite mixture of Gaussian Process Functional Regressions (mix-GPFR) was suggested. However, the number of components in mix-GPFR needs to be specified a priori, which is difficult to determine in practice. In this paper, we propose a Dirichlet Process Mixture of Gaussian Process Functional Regressions (DPM-GPFR), in which there are potentially infinite many GPFR components dominated by a Dirichlet process. Thus, DPM-GPFR is far more flexible than a single GPFR, and sidestep the model selection problem in mix-GPFR. We further develop a fully Bayesian treatment for learning DPM-GPFR based on the Variational Expectation-Maximization (VEM) algorithm. Experimental results on both synthetic datasets and real-world datasets demonstrate the effectiveness of our proposed method.

Spectral Estimation of Large Stochastic Blockmodels with Discrete Nodal Covariates

In many applications of network analysis, it is important to distinguish between observed and unobserved factors affecting network structure. We show that a network model with discrete unobserved link heterogeneity and binary (or discrete) covariates corresponds to a stochastic blockmodel (SBM). We develop a spectral estimator for the effect of covariates on link probabilities, exploiting the correspondence of SBMs and generalized random dot product graphs (GRDPG). We show that computing our estimator is much faster than standard variational expectation–maximization algorithms and scales well for large networks. Monte Carlo experiments suggest that the estimator performs well under different data generating processes. Our application to Facebook data shows evidence of homophily in gender, role and campus-residence, while allowing us to discover unobserved communities. Finally, we establish asymptotic normality of our estimators.

Fusing topology contexts and logical rules in language models for knowledge graph completion

Knowledge graph completion (KGC) aims to infer missing facts based on the observed ones, which is significant for many downstream applications. Given the success of deep learning and pre-trained language models (LMs), some LM-based methods are proposed for the KGC task. However, most of them focus on modeling the text of fact triples and ignore the deeper semantic information (e.g., topology contexts and logical rules) that is significant for KG modeling. For such a reason, we propose a unified framework FTL-LM to Fuse Topology contexts and Logical rules in Language Models for KGC, which mainly contains a novel path-based method for topology contexts learning and a variational expectation–maximization (EM) algorithm for soft logical rule distilling. The former utilizes a heterogeneous random-walk to generate topology paths and further reasoning paths that can represent topology contexts implicitly and can be modeled by a LM explicitly. The strategies of mask language modeling and contrastive path learning are introduced to model these topology contexts. The latter implicitly fuses logical rules by a variational EM algorithm with two LMs. Specifically, in the E-step, the triple LM is updated under the supervision of observed triples and valid hidden triples verified by the fixed rule LM. And in the M-step, we fix the triple LM and fine-tune the rule LM to update logical rules. Experiments on three common KGC datasets demonstrate the superiority of the proposed FTL-LM, e.g., it achieves 2.1% and 3.1% Hits@10 improvement over the state-of-the-art LM-based model LP-BERT in the WN18RR and FB15k-237, respectively.

A variational Bayesian approach to multiantenna spectrum sensing under correlated noise

In this paper, we address the problem of spectrum sensing when the secondary user (SU) is equipped with a multiantenna receiver and noise across antenna elements are correlated. Covariance matrix of correlated noise is non-diagonal and its standard-condition-number (SCN) is much greater than one, which results in significantly degraded detection performance. Many of the existing techniques either whiten the covariance matrix or utilize geometrical properties of noise covariance matrix for the design of decision statistic. However, these techniques have low detection performance in uncorrelated noise or when the correlation is low. To address this problem and further improve the detection performance, a novel spectrum sensing algorithm based on variational Bayesian Gaussian mixture model (VBGMM) is proposed. Specifically, a new decision vector is proposed which is amenable to VBGMM and incorporates signal features suitable for spectrum sensing in both correlated and uncorrelated noise. Decision vectors extracted from the received signal are clustered using variational expectation-maximization (VEM) algorithm to detect the presence of primary user (PU). Compared to existing techniques, the proposed method is shown by simulation to achieve higher detection rates for the entire range of correlation.

Variational Inference and Learning of Piecewise Linear Dynamical Systems.

Modeling the temporal behavior of data is of primordial importance in many scientific and engineering fields. Baseline methods assume that both the dynamic and observation equations follow linear-Gaussian models. However, there are many real-world processes that cannot be characterized by a single linear behavior. Alternatively, it is possible to consider a piecewise linear model which, combined with a switching mechanism, is well suited when several modes of behavior are needed. Nevertheless, switching dynamical systems are intractable because their computational complexity increases exponentially with time. In this article, we propose a variational approximation of piecewise linear dynamical systems. We provide full details of the derivation of two variational expectation-maximization algorithms: a filter and a smoother. We show that the model parameters can be split into two sets: static and dynamic parameters, and that the former parameters can be estimated offline together with the number of linear modes, or the number of states of the switching variable. We apply the proposed method to the head-pose tracking, and we thoroughly compare our algorithms with several state of the art trackers.

Individualized passenger travel pattern multi-clustering based on graph regularized tensor latent dirichlet allocation

Individual passenger travel patterns have significant value in understanding passenger’s behavior, such as learning the hidden clusters of locations, time, and passengers. The learned clusters further enable commercially beneficial actions such as customized services, promotions, data-driven urban-use planning, peak hour discovery, and so on. However, the individualized passenger modeling is very challenging for the following reasons: 1) The individual passenger travel data are multi-dimensional spatiotemporal big data, including at least the origin, destination, and time dimensions; 2) Moreover, individualized passenger travel patterns usually depend on the external environment, such as the distances and functions of locations, which are ignored in most current works. This work proposes a multi-clustering model to learn the latent clusters along the multiple dimensions of Origin, Destination, Time, and eventually, Passenger (ODT-P). We develop a graph-regularized tensor Latent Dirichlet Allocation (LDA) model by first extending the traditional LDA model into a tensor version and then applies to individual travel data. Then, the external information of stations is formulated as semantic graphs and incorporated as the Laplacian regularizations; Furthermore, to improve the model scalability when dealing with massive data, an online stochastic learning method based on tensorized variational Expectation-Maximization algorithm is developed. Finally, a case study based on passengers in the Hong Kong metro system is conducted and demonstrates that a better clustering performance is achieved compared to state-of-the-arts with the improvement in point-wise mutual information index and algorithm convergence speed by a factor of two.

Variational Approximation-Based Model Selection for Microbial Network Inference.

Microbial associations are characterized by both direct and indirect interactions between the constituent taxa in a microbial community, and play an important role in determining the structure, organization, and function of the community. Microbial associations can be represented using a weighted graph (microbial network), whose nodes represent taxa and edges represent pairwise associations. A microbial network is typically inferred from a sample-taxa matrix that is obtained by sequencing multiple biological samples and identifying the taxa counts in each sample. However, it is known that microbial associations are impacted by environmental and/or host factors. Thus, a sample-taxa matrix generated in a microbiome study involving a wide range of values for the environmental and/or clinical metadata variables may in fact be associated with more than one microbial network. In this study, we consider the problem of inferring multiple microbial networks from a given sample-taxa count matrix. Each sample is a count vector assumed to be generated by a mixture model consisting of component distributions that are multivariate Poisson log-normal. We present a variational expectation maximization algorithm for the model selection problem to infer the correct number of components of this mixture model. Our approach involves reframing the mixture model as a latent variable model, treating only the mixing coefficients as parameters, and subsequently approximating the marginal likelihood using an evidence lower bound framework. Our algorithm is evaluated on a large simulated dataset generated using a collection of different graph structures (band, hub, cluster, random, and scale-free).

Inference of Multiplicative Factors Underlying Neural Variability in Calcium Imaging Data.

Understanding brain function requires disentangling the high-dimensional activity of populations of neurons. Calcium imaging is an increasingly popular technique for monitoring such neural activity, but computational tools for interpreting extracted calcium signals are lacking. While there has been a substantial development of factor analysis-type methods for neural spike train analysis, similar methods targeted at calcium imaging data are only beginning to emerge. Here we develop a flexible modeling framework that identifies low-dimensional latent factors in calcium imaging data with distinct additive and multiplicative modulatory effects. Our model includes spike-and-slab sparse priors that regularize additive factor activity and gaussian process priors that constrain multiplicative effects to vary only gradually, allowing for the identification of smooth and interpretable changes in multiplicative gain. These factors are estimated from the data using a variational expectation-maximization algorithm that requires a differentiable reparameterization of both continuous and discrete latent variables. After demonstrating our method on simulated data, we apply it to experimental data from the zebrafish optic tectum, uncovering low-dimensional fluctuations in multiplicative excitability that govern trial-to-trial variation in evoked responses.

Unsupervised Speech Enhancement Using Dynamical Variational Autoencoders

Dynamical variational autoencoders (DVAEs) are a class of deep generative models with latent variables, dedicated to model time series of high-dimensional data. DVAEs can be considered as extensions of the variational autoencoder (VAE) that include temporal dependencies between successive observed and/or latent vectors. Previous work has shown the interest of using DVAEs over the VAE for speech spectrograms modeling. Independently, the VAE has been successfully applied to speech enhancement in noise, in an unsupervised noise-agnostic set-up that requires neither noise samples nor noisy speech samples at training time, but only requires clean speech signals. In this paper, we extend these works to DVAE-based single-channel unsupervised speech enhancement, hence exploiting both speech signals unsupervised representation learning and dynamics modeling. We propose an unsupervised speech enhancement algorithm that combines a DVAE speech prior pre-trained on clean speech signals with a noise model based on nonnegative matrix factorization, and we derive a variational expectation-maximization (VEM) algorithm to perform speech enhancement. The algorithm is presented with the most general DVAE formulation and is then applied with three specific DVAE models to illustrate the versatility of the framework. Experimental results show that the proposed DVAE-based approach outperforms its VAE-based counterpart, as well as several supervised and unsupervised noise-dependent baselines, especially when the noise type is unseen during training.

Learning from missing data with the binary latent block model

Missing data can be informative. Ignoring this information can lead to misleading conclusions when the data model does not allow information to be extracted from the missing data. We propose a co-clustering model, based on the binary Latent Block Model, that aims to take advantage of this nonignorable nonresponses, also known as Missing Not At Random data. A variational expectation–maximization algorithm is derived to perform inference and a model selection criterion is presented. We assess the proposed approach on a simulation study, before using our model on the voting records from the lower house of the French Parliament, where our analysis brings out relevant groups of MPs and texts, together with a sensible interpretation of the behavior of non-voters.

Semisupervised Association Learning Based on Partial Differential Equations for Sparse Representation of Image Class Attributes

Semisupervised learning is an idea that addresses how to use a large number of unlabeled samples and a limited number of labeled samples to learn decision knowledge together. In this paper, we propose a multitask multiview semisupervised learning model based on partial differential equation random field and Hilbert independent standard probability image genus attribute model, i.e., shared semantics. In the framework of the image-like genus attribute model, data from different data sources are generated by their shared hidden space representation. Different from the traditional model, this paper uses the Hilbert independence criterion to inscribe the shared relationship of hidden expressions. Meanwhile, to exploit the correlations between labels in the label space as well, this paper uses the partial differential equation random field to inscribe the correlations between different kinds of labels in the label space and the correlations between hidden features and labels. Using the variational expectation-maximization algorithm, the whole generative process model can be inferred. To verify the effectiveness of the model, two artificial datasets and three real datasets are tested in this paper, and the experimental results verify the effectiveness of the algorithm in the paper. On the one hand, it not only improves the classification accuracy of the multiclassification problem and the multilabel problem; it also outputs the association structure between different kinds of labels and between hidden features and labels.

Empirical Bayes estimation of pairwise maximum entropy model for nonlinear brain state dynamics

The pairwise maximum entropy model (pMEM) has recently gained widespread attention to exploring the nonlinear characteristics of brain state dynamics observed in resting-state functional magnetic resonance imaging (rsfMRI). Despite its unique advantageous features, the practical application of pMEM for individuals is limited as it requires a much larger sample than conventional rsfMRI scans. Thus, this study proposes an empirical Bayes estimation of individual pMEM using the variational expectation-maximization algorithm (VEM-MEM). The performance of the VEM-MEM is evaluated for several simulation setups with various sample sizes and network sizes. Unlike conventional maximum likelihood estimation procedures, the VEM-MEM can reliably estimate the individual model parameters, even with small samples, by effectively incorporating the group information as the prior. As a test case, the individual rsfMRI of children with attention deficit hyperactivity disorder (ADHD) is analyzed compared to that of typically developed children using the default mode network, executive control network, and salient network, obtained from the Healthy Brain Network database. We found that the nonlinear dynamic properties uniquely established on the pMEM differ for each group. Furthermore, pMEM parameters are more sensitive to group differences and are better associated with the behavior scores of ADHD compared to the Pearson correlation-based functional connectivity. The simulation and experimental results suggest that the proposed method can reliably estimate the individual pMEM and characterize the dynamic properties of individuals by utilizing empirical information of the group brain state dynamics.

Probabilistic domain-knowledge modeling of disorder pathogenesis for dynamics forecasting of acute onset

Disease pathogenesis, a type of domain knowledge about biological mechanisms leading to diseases, has not been adequately encoded in machine-learning-based medical diagnostic models because of the inter-patient variabilities and complex dependencies of the underlying pathogenetic mechanisms. We propose 1) a novel pathogenesis probabilistic graphical model (PPGM) to quantify the dynamics underpinning patient-specific data and pathogenetic domain knowledge, 2) a Bayesian-based inference paradigm to answer the medical queries and forecast acute onsets. The PPGM model consists of two components: a Bayesian network of patient attributes and a temporal model of pathogenetic mechanisms. The model structure was reconstructed from expert knowledge elicitation, and its parameters were estimated using Variational Expectation-Maximization algorithms. We benchmarked our model with two well-established hidden Markov models (HMMs) – Input-output HMM (IO-HMM) and Switching Auto-Regressive HMM (SAR-HMM) – to evaluate the computational costs, forecasting performance, and execution time. Two case studies on Obstructive Sleep Apnea (OSA) and Paroxysmal Atrial Fibrillation (PAF) were used to validate the model. While the performance of the parameter learning step was equivalent to those of IO-HMM and SAR-HMM models, our model forecasting ability was outperforming those two models. The merits of the PPGM model are its representation capability to capture the dynamics of pathogenesis and perform medical inferences and its interpretability for physicians. The model has been used to perform medical queries and forecast the acute onset of OSA and PAF. Additional applications of the model include prognostic healthcare and preventive personalized treatments.