Variational Bayes Approximation Research Articles

Bayesian Empirical Likelihood Regression for Semiparametric Estimation of Optimal Dynamic Treatment Regimes.

We propose a semiparametric approach to Bayesian modeling of dynamic treatment regimes that is built on a Bayesian likelihood-based regression estimation framework. Methods based on this framework exhibit a probabilistic coherence property that leads to accurate estimation of the optimal dynamic treatment regime. Unlike most Bayesian estimation methods, our proposed method avoids strong distributional assumptions for the intermediate and final outcomes by utilizing empirical likelihoods. Our proposed method allows for either linear, or more flexible forms of mean functions for the stagewise outcomes. A variational Bayes approximation is used for computation to avoid common pitfalls associated with Markov Chain Monte Carlo approaches coupled with empirical likelihood. Through simulations and analysis of the STAR*D sequential randomized trial data, our proposed method demonstrates superior accuracy over Q-learning and parametric Bayesian likelihood-based regression estimation, particularly when the parametric assumptions of regression error distributions may be potentially violated.

Ultra-Efficient MCMC for Bayesian Longitudinal Functional Data Analysis

Functional mixed models are widely useful for regression analysis with dependent functional data, including longitudinal functional data with scalar predictors. However, existing algorithms for Bayesian inference with these models only provide either scalable computing or accurate approximations to the posterior distribution, but not both. We introduce a new MCMC sampling strategy for highly efficient and fully Bayesian regression with longitudinal functional data. Using a novel blocking structure paired with an orthogonalized basis reparameterization, our algorithm jointly samples the fixed effects regression functions together with all subject- and replicate-specific random effects functions. Crucially, the joint sampler optimizes sampling efficiency for these key parameters while preserving computational scalability. Perhaps surprisingly, our new MCMC sampling algorithm even surpasses state-of-the-art algorithms for frequentist estimation and variational Bayes approximations for functional mixed models—while also providing accurate posterior uncertainty quantification—and is orders of magnitude faster than existing Gibbs samplers. Simulation studies show improved point estimation and interval coverage in nearly all simulation settings over competing approaches. We apply our method to a large physical activity dataset to study how various demographic and health factors associate with intraday activity. Supplementary materials for this article are available online.

Nonlinearities in macroeconomic tail risk through the lens of big data quantile regressions

SummaryModeling and predicting extreme movements in GDP is notoriously difficult, and the selection of appropriate covariates and/or possible forms of nonlinearities are key in obtaining precise forecasts. In this paper, our focus is on using large datasets in quantile regression models to forecast the conditional distribution of US GDP growth. To capture possible nonlinearities, we include several nonlinear specifications. The resulting models will be huge dimensional, and we thus rely on a set of shrinkage priors. Since Markov chain Monte Carlo estimation becomes slow in these dimensions, we rely on fast variational Bayes approximations to the posterior distribution of the coefficients and the latent states. We find that our proposed set of models produces precise forecasts. These gains are especially pronounced in the tails. Using Gaussian processes to approximate the nonlinear component of the model further improves the good performance, in particular in the right tail.

Variational Bayesian Inference for Quantile Regression Models with Nonignorable Missing Data

Quantile regression models are remarkable structures for conducting regression analyses when the data are subject to missingness. Missing values occur because of various factors like missing completely at random, missing at random, or missing not at random. All these may result from system malfunction during data collection or human error during data preprocessing. Nevertheless, it is important to deal with missing values before analyzing data since ignoring or omitting missing values may result in biased or misinformed analysis. This paper studies quantile regressions from a Bayesian perspective. By proposing a hierarchical model framework, we develop an alternative approach based on deterministic variational Bayes approximations. Logistic and probit models are adopted to specify propensity scores for missing manifests and covariates, respectively. Bayesian variable selection method is proposed to recognize significant covariates. Several simulation studies and real examples illustrate the advantages of the proposed methodology and offer some possible future research directions.

An enriched mixture model for functional clustering

AbstractThere is an increasingly rich literature about Bayesian nonparametric models for clustering functional observations. Most recent proposals rely on infinite‐dimensional characterizations that might lead to overly complex cluster solutions. In addition, while prior knowledge about the functional shapes is typically available, its practical exploitation might be a difficult modeling task. Motivated by an application in e‐commerce, we propose a novel enriched Dirichlet mixture model for functional data. Our proposal accommodates the incorporation of functional constraints while bounding the model complexity. We characterize the prior process through a urn scheme to clarify the underlying partition mechanism. These features lead to a very interpretable clustering method compared to available techniques. Moreover, we employ a variational Bayes approximation for tractable posterior inference to overcome computational bottlenecks.

Variational Bayes approximation of factor stochastic volatility models

Estimation and prediction in high dimensional multivariate factor stochastic volatility models is an important and active research area, because such models allow a parsimonious representation of multivariate stochastic volatility. Bayesian inference for factor stochastic volatility models is usually done by Markov chain Monte Carlo methods (often by particle Markov chain Monte Carlo methods), which are usually slow for high dimensional or long time series because of the large number of parameters and latent states involved. Our article makes two contributions. The first is to propose a fast and accurate variational Bayes methods to approximate the posterior distribution of the states and parameters in factor stochastic volatility models. The second is to extend this batch methodology to develop fast sequential variational updates for prediction as new observations arrive. The methods are applied to simulated and real datasets, and shown to produce good approximate inference and prediction compared to the latest particle Markov chain Monte Carlo approaches, but are much faster.

Predicting Conversion from MCI to AD Combining Multi-Modality Data and Based on Molecular Subtype.

Alzheimer’s disease (AD) is a neurodegenerative brain disease in the elderly. Identifying patients with mild cognitive impairment (MCI) who are more likely to progress to AD is a key step in AD prevention. Recent studies have shown that AD is a heterogeneous disease. In this study, we propose a subtyping-based prediction strategy to predict the conversion from MCI to AD in three years according to MCI patient subtypes. Structural magnetic resonance imaging (sMRI) data and multi-omics data, including genotype data and gene expression profiling derived from peripheral blood samples, from 125 MCI patients were used in the Alzheimer’s Disease Neuroimaging Initiative (ADNI)-1 dataset and from 98 MCI patients in the ADNI-GO/2 dataset. A variational Bayes approximation model based on the multiple kernel learning method was constructed to predict whether an MCI patient will progress to AD within three years. In internal fivefold cross-validation within ADNI-1, we achieved an overall AUC of 0.83 (79.20% accuracy, 81.25% sensitivity, 77.92% specificity) compared to the model without subtyping, which achieved an AUC of 0.78 (76.00% accuracy, 77.08% sensitivity, 75.32% specificity). In external validation using ADNI-1 as a training set and ADNI-GO/2 as an independent test set, we attained an AUC of 0.78 (74.49% accuracy, 74.19% sensitivity, 74.63% specificity). Identifying MCI patient subtypes with omics data would improve the accuracy of predicting the conversion from MCI to AD. In addition to evaluating statistics, obtaining the significant sMRI, single nucleotide polymorphism (SNP) and mRNA expression data from peripheral blood of MCI patients is noninvasive and cost-effective for predicting conversion from MCI to AD.

Bayesian estimation of maximum entropy model for individualized energy landscape analysis of brain state dynamics.

The pairwise maximum entropy model (MEM) for resting state functional MRI (rsfMRI) has been used to generate energy landscape of brain states and to explore nonlinear brain state dynamics. Researches using MEM, however, has mostly been restricted to fixed‐effect group‐level analyses, using concatenated time series across individuals, due to the need for large samples in the parameter estimation of MEM. To mitigate the small sample problem in analyzing energy landscapes for individuals, we propose a Bayesian estimation of individual MEM using variational Bayes approximation (BMEM). We evaluated the performances of BMEM with respect to sample sizes and prior information using simulation. BMEM showed advantages over conventional maximum likelihood estimation in reliably estimating model parameters for individuals with small sample data, particularly utilizing the empirical priors derived from group data. We then analyzed individual rsfMRI of the Human Connectome Project to show the usefulness of MEM in differentiating individuals and in exploring neural correlates for human behavior. MEM and its energy landscape properties showed high subject specificity comparable to that of functional connectivity. Canonical correlation analysis identified canonical variables for MEM highly associated with cognitive scores. Inter‐individual variations of cognitive scores were also reflected in energy landscape properties such as energies, occupation times, and basin sizes at local minima. We conclude that BMEM provides an efficient method to characterize dynamic properties of individuals using energy landscape analysis of individual brain states.

Assessment and Adjustment of Approximate Inference Algorithms Using the Law of Total Variance

A common method for assessing validity of Bayesian sampling or approximate inference methods makes use of simulated data replicates for parameters drawn from the prior. Under continuity assumptions, quantiles of functions of the simulated parameter values for corresponding posterior distributions are uniformly distributed. Checking for uniformity when a posterior density is approximated numerically provides a diagnostic for algorithm validity. Furthermore, adjustments to achieve uniformity can improve the quality of approximate inference methods. The present article develops a moment-based alternative to the conventional checking and adjustment methods using quantiles. The new approach relates prior and posterior expectations and covariances through the tower property of conditional expectation and the law of total variance. For adjustment, approximate inferences are modified so that the correct prior to posterior relationships hold. We illustrate the method in three examples. The first uses an auxiliary model in a likelihood-free inference problem. The second considers corrections for variational Bayes approximations in a deep neural network generalized linear mixed model. Our final application considers a deep neural network surrogate for approximating Gaussian process regression predictive inference. Supplementary files for this article are available online.

PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models.

Datasets displaying temporal dependencies abound in science and engineering applications, with Markov models representing a simplified and popular view of the temporal dependence structure. In this paper, we consider Bayesian settings that place prior distributions over the parameters of the transition kernel of a Markov model, and seek to characterize the resulting, typically intractable, posterior distributions. We present a Probably Approximately Correct (PAC)-Bayesian analysis of variational Bayes (VB) approximations to tempered Bayesian posterior distributions, bounding the model risk of the VB approximations. Tempered posteriors are known to be robust to model misspecification, and their variational approximations do not suffer the usual problems of over confident approximations. Our results tie the risk bounds to the mixing and ergodic properties of the Markov data generating model. We illustrate the PAC-Bayes bounds through a number of example Markov models, and also consider the situation where the Markov model is misspecified.

Strong Tracking PHD Filter Based on Variational Bayesian with Inaccurate Process and Measurement Noise Covariance.

Assuming that the measurement and process noise covariances are known, the probability hypothesis density (PHD) filter is effective in real-time multi-target tracking; however, noise covariance is often unknown and time-varying for an actual scene. To solve this problem, a strong tracking PHD filter based on Variational Bayes (VB) approximation is proposed in this paper. The measurement noise covariance is described in the linear system by the inverse Wishart (IW) distribution. Then, the fading factor in the strong tracking principle uses the optimal measurement noise covariance at the previous moment to control the state prediction covariance in real-time. The Gaussian IW (GIW) joint distribution adopts the VB approximation to jointly return the measurement noise covariance and the target state covariance. The simulation results show that, compared with the traditional Gaussian mixture PHD (GM-PHD) and the VB-adaptive PHD, the proposed algorithm has higher tracking accuracy and stronger robustness in a more reasonable calculation time.

Variational Bayes for High-Dimensional Linear Regression With Sparse Priors

We study a mean-field spike and slab variational Bayes (VB) approximation to Bayesian model selection priors in sparse high-dimensional linear regression. Under compatibility conditions on the design matrix, oracle inequalities are derived for the mean-field VB approximation, implying that it converges to the sparse truth at the optimal rate and gives optimal prediction of the response vector. The empirical performance of our algorithm is studied, showing that it works comparably well as other state-of-the-art Bayesian variable selection methods. We also numerically demonstrate that the widely used coordinate-ascent variational inference algorithm can be highly sensitive to the parameter updating order, leading to potentially poor performance. To mitigate this, we propose a novel prioritized updating scheme that uses a data-driven updating order and performs better in simulations. The variational algorithm is implemented in the R package sparsevb. Supplementary materials for this article are available online.

Seemingly unrelated regression with measurement error: estimation via Markov Chain Monte Carlo and mean field variational Bayes approximation.

Linear regression with measurement error in the covariates is a heavily studied topic, however, the statistics/econometrics literature is almost silent to estimating a multi-equation model with measurement error. This paper considers a seemingly unrelated regression model with measurement error in the covariates and introduces two novel estimation methods: a pure Bayesian algorithm (based on Markov chain Monte Carlo techniques) and its mean field variational Bayes (MFVB) approximation. The MFVB method has the added advantage of being computationally fast and can handle big data. An issue pertinent to measurement error models is parameter identification, and this is resolved by employing a prior distribution on the measurement error variance. The methods are shown to perform well in multiple simulation studies, where we analyze the impact on posterior estimates for different values of reliability ratio or variance of the true unobserved quantity used in the data generating process. The paper further implements the proposed algorithms in an application drawn from the health literature and shows that modeling measurement error in the data can improve model fitting.

Toward security monitoring of industrial Cyber-Physical systems via hierarchically distributed intrusion detection

Industrial Cyber-physical systems (ICPSs), integrating communication, computation and control of industrial processes are referred to as a core technology to approach the Industry 4.0. Ensuring the ICPS security is of paramount importance in smart manufacturing. Considering the characteristics of large-scale, geographically-dispersed and multi-dimensional heterogeneous, federated and life-critical natures of ICPSs, this paper investigates a hierarchically distributed intrusion detection scheme that seeks to achieve the all-round safety protection of ICPSs according to the system structure and attacking types of each ICPS layer. For physical system-relevant perceptual executive layer, potential and covert attacks are detected by the clustered sensory system state residual anomaly monitoring based on a process noise and measurement noise-adaptive Kalman filter (PNMN-AKF). PNMN-AKF can perform a joint recursive estimation of dynamic system states, time-varying process and measurement noise covariance matrices by the variational Bayes approximation framework. In cyberspace, potential cyber-attacks are detected by the anomaly monitoring of the statistical distribution of the network transmission characteristics of data transmission layer by introducing a forgetting factor-induced recursive Gaussian mixture model (FF-RGMM). In the application control layer, a regularized sparse deep belief network model is introduced to characterize the misuse behavior for detecting potential attacks. Extensive validation and comparative experiments have been conducted on a numerical simulation system and a comprehensive ICPS simulation platform by using OPNET and a commonly-used benchmark simplified Tennessee Eastman process (STEP) based on Matlab/Simulink. Experimental results demonstrate that the proposed hierarchically distributed intrusion detection method can efficiently recognize potential and covert cyber-attacks in each ICPSs link with low false alarm rate and missing detection rate, which lays a foundation for the overall security monitoring of ICPSs.

Bayesian transfer learning between Student-t filters

The problem of sequentially transferring a data-predictive probability distribution from a source to a target Bayesian filter is addressed in this paper. In many practical settings, this transfer is incompletely modelled, since the stochastic dependence structure between the filters typically cannot be fully specified. We therefore adopt fully probabilistic design to select the optimal transfer mechanism. We relax the target observation model via a scale-mixing parameter, which proves vital in successfully transferring the first and second moments of the source data predictor. This sensitivity to the transferred second moment ensures that imprecise predictors are rejected, achieving robust transfer. Indeed, Student-t state and observation models are adopted for both learning processes, in order to handle outliers in all hidden and observed variables. A recursive outlier-robust Bayesian transfer learning algorithm is recovered via a local variational Bayes approximation. The outlier rejection and positive transfer properties of the resulting algorithm are clearly demonstrated in a simulated planar position-velocity system, as is the key property of imprecise knowledge rejection (robust transfer), unavailable in current Bayesian transfer algorithms. Performance comparison with particle filter variants demonstrates the successful convergence of our robust variational Bayes transfer learning algorithm in sequential processing.

A Variational Approximations-DIC Rubric for Parameter Estimation and Mixture Model Selection Within a Family Setting

Mixture model-based clustering has become an increasingly popular data analysis technique since its introduction over fifty years ago, and is now commonly utilized within a family setting. Families of mixture models arise when the component parameters, usually the component covariance (or scale) matrices, are decomposed and a number of constraints are imposed. Within the family setting, model selection involves choosing the member of the family, i.e., the appropriate covariance structure, in addition to the number of mixture components. To date, the Bayesian information criterion (BIC) has proved most effective for model selection, and the expectation-maximization (EM) algorithm is usually used for parameter estimation. In fact, this EM-BIC rubric has virtually monopolized the literature on families of mixture models. Deviating from this rubric, variational Bayes approximations are developed for parameter estimation and the deviance information criteria (DIC) for model selection. The variational Bayes approach provides an alternate framework for parameter estimation by constructing a tight lower bound on the complex marginal likelihood and maximizing this lower bound by minimizing the associated Kullback-Leibler divergence. The framework introduced, which we refer to as VB-DIC, is applied to the most commonly used family of Gaussian mixture models, and real and simulated data are used to compared with the EM-BIC rubric.

Asymptotic consistency of loss‐calibrated variational Bayes

This paper establishes the asymptotic consistency of the loss‐calibrated variational Bayes (LCVB) method. LCVB is a method for approximately computing Bayesian posterior approximations in a “loss aware” manner. This methodology is also highly relevant in general data‐driven decision‐making contexts. Here, we establish the asymptotic consistency of both the loss‐ calibrated approximate posterior and the resulting decision rules. We also establish the asymptotic consistency of decision rules obtained from a “naive” two‐stage procedure that first computes a standard variational Bayes approximation and then uses this in the decision‐making procedure.

Carbon Dioxide Emissions and Economic Activities: A Mean Field Variational Bayes Semiparametric Panel Data Model with Random Coefficients

This paper proposes semiparametric estimation of the relationship between emissions and economic activities for a panel of 81 countries observed over the period 1991-2015. The observed differentiated behaviors by country reveal strong heterogeneity as well as different trends across countries and years. This is the motivation behind using a mixed fixed- and random-coefficients panel data model to estimate this relationship. Following Lee and Wand (2016a), we apply a mean field variational Bayes approximation to estimate a log model with structural breaks between emissions per capita and GDP per capita including control covariates such as energy intensity and use, energy consumption, population density, urbanization and trade. Results reveal a strong CO2 emissions - GDP elasticity, close to one, confirming the increasing but complex link between these two variables. The use of this methodology enriches the estimates of climate change models underlining a large diversity of responses across variables and countries.

Computationally efficient Bayesian estimation of high-dimensional Archimedean copulas with discrete and mixed margins

Estimating copulas with discrete marginal distributions is challenging, especially in high dimensions, because computing the likelihood contribution of each observation requires evaluating $$2^{J}$$ terms, with J the number of discrete variables. Our article focuses on the estimation of Archimedean copulas, for example, Clayton and Gumbel copulas. Currently, data augmentation methods are used to carry out inference for discrete copulas and, in practice, the computation becomes infeasible when J is large. Our article proposes two new fast Bayesian approaches for estimating high-dimensional Archimedean copulas with discrete margins, or a combination of discrete and continuous margins. Both methods are based on recent advances in Bayesian methodology that work with an unbiased estimate of the likelihood rather than the likelihood itself, and our key observation is that we can estimate the likelihood of a discrete Archimedean copula unbiasedly with much less computation than evaluating the likelihood exactly or with current simulation methods that are based on augmenting the model with latent variables. The first approach builds on the pseudo-marginal method that allows Markov chain Monte Carlo simulation from the posterior distribution using only an unbiased estimate of the likelihood. The second approach is based on a variational Bayes approximation to the posterior and also uses an unbiased estimate of the likelihood. We show that the two new approaches enable us to carry out Bayesian inference for high values of J for the Archimedean copulas where the computation was previously too expensive. The methodology is illustrated through several real and simulated data examples.

Estimation and testing of stochastic frontier models using variational Bayes

We show how a wide range of stochastic frontier models can be estimated relatively easily using variational Bayes. We derive approximate posterior distributions and point estimates for parameters and inefficiency effects for (a) time invariant models with several alternative inefficiency distributions, (b) models with time varying effects, (c) models incorporating environmental effects, and (d) models with more flexible forms for the regression function and error terms. Despite the abundance of stochastic frontier models, there have been few attempts to test the various models against each other, probably due to the difficulty of performing such tests. One advantage of the variational Bayes approximation is that it facilitates the computation of marginal likelihoods that can be used to compare models. We apply this idea to test stochastic frontier models with different inefficiency distributions. Estimation and testing is illustrated using three examples.