Neural Autoregressive Flows Based Variational Bayes Model Averaging

Applying Bayesian Model Averaging to mechanistic models: An example and comparison of methods

Variable selection is one of the most important and controversial issues in modern data analysis. In the study of relationships between biological communities and environmental conditions, variable selection is especially important as it guides decisions about environmental management. Using a case study from the Great Lakes we use Bayesian Model Averaging (BMA) to select interesting subsets of environmental variables (such as water chemistry, depth, etc), that can affect the abundance of benthic microinvertebrates taxa. We implement BMA for a multivariate technique called Canonical Correspondence Analysis (CCA) and use its results to represent sites, species and selected environmental variables on ordination diagrams (biplots) along with “error bars” representing uncertainty due to both sampling variability and model selection. BMA provides data analysts with an efficient tool for discovering promising models and obtaining estimates of their posterior probabilities via Markov chain Monte Carlo (MCMC). These probabilities are further used as weights for model averaged predictions and estimates of the parameters of interest. As a result, variance components due to model selection can be estimated and accounted for, contrary to the practice of conventional data analysis. In our study we adopt an approach to BMA called Model Composition Markov Chain Monte Carlo (Madigan and Raffcery, 1994) and we implement the BMA methodology by treating CCA within a general framework of reduced rank regression for which we develop a Bayes Information Criterion (BIC) approximation to posterior model probabilities in the spirit of MC3.

The software reliability modeling is of great significance in improving software quality and managing the software development process. However, the existing methods are not able to accurately model software reliability improvement behavior because existing single model methods rely on restrictive assumptions and combination models cannot well deal with model uncertainties. In this article, we propose a Bayesian model averaging (BMA) method to model software reliability. First, the existing reliability modeling methods are selected as the candidate models, and the Bayesian theory is used to obtain the posterior probabilities of each reliability model. Then, the posterior probabilities are used as weights to average the candidate models. Both Markov Chain Monte Carlo (MCMC) algorithm and the Expectation–Maximization (EM) algorithm are used to evaluate a candidate model's posterior probability and for comparison purpose. The results show that the BMA method has superior performance in software reliability modeling, and the MCMC algorithm performs better than EM algorithm when they are used to estimate the parameters of BMA method.

Choosing the subset of covariates to use in regression or generalized linear models is a ubiquitous problem. The Bayesian paradigm addresses the problem of model uncertainty by considering models corresponding to all possible subsets of the covariates, where the posterior distribution over models is used to select models or combine them via Bayesian model averaging (BMA). Although conceptually straightforward, BMA is often difficult to implement in practice, since either the number of covariates is too large for enumeration of all subsets, calculations cannot be done analytically, or both. For orthogonal designs with the appropriate choice of prior, the posterior probability of any model can be calculated without having to enumerate the entire model space and scales linearly with the number of predictors, p. In this article we extend this idea to a much broader class of nonorthogonal design matrices. We propose a novel method which augments the observed nonorthogonal design by at most p new rows to obtain a design matrix with orthogonal columns and generate the “missing” response variables in a data augmentation algorithm. We show that our data augmentation approach keeps the original posterior distribution of interest unaltered, and develop methods to construct Rao–Blackwellized estimates of several quantities of interest, including posterior model probabilities of any model, which may not be available from an ordinary Gibbs sampler. Our method can be used for BMA in linear regression and binary regression with nonorthogonal design matrices in conjunction with independent “spike and slab” priors with a continuous prior component that is a Cauchy or other heavy tailed distribution that may be represented as a scale mixture of normals. We provide simulated and real examples to illustrate the methodology. Supplemental materials for the manuscript are available online.

This article examines Bayesian model averaging as a means of addressing predictive performance in Bayesian structural equation models. The current approach to addressing the problem of model uncertainty lies in the method of Bayesian model averaging. We expand the work of Madigan and his colleagues by considering a structural equation model as a special case of a directed acyclic graph. We then provide an algorithm that searches the model space for submodels and obtains a weighted average of the submodels using posterior model probabilities as weights. Our simulation study provides a frequentist evaluation of our Bayesian model averaging approach and indicates that when the true model is known, Bayesian model averaging does not yield necessarily better predictive performance compared to nonaveraged models. However, our case study using data from an international large-scale assessment reveals that the model-averaged submodels provide better posterior predictive performance compared to the initially specified model.

A special technique that measures the uncertainties embedded in model selection processes is Bayesian Model Averaging (BMA) which depends on the appropriate choices of model and parameter priors. Inspite the importance of the parameter priors' specification in BMA, the existing parameter priors give exitremely low Posterior Model Probability (PMP). Therefore, this paper elicits modified g-parameter priors to improve the performance of the PMP and predictive ability of the model with an application to the Water Pollution of Asejire in Ibadan. The modified g-parameter priors g j = , established the consistency conditions and asymptotic properties using the models in the literature. The results show that the PMP with the best prior (g j = ) had the least standard deviations (0.0411 at n=100,000 and 0:000 at n=1000) for models 1 & 2 respectively; and had the highest posterior means (0.9577 at n=100,000 and 1.000 at n=1000) for models 1 & 2 respectively. The point and overall predictive performances for the best prior were 2.357 at n=50 and 2.335 at n=100,000 when compared with the BMA Log Predictive Score threshold of 2.335. Applying this best g-parameter prior in modeling the Asejire river, it indicates that the dissolved solids (mg/l) and total solids (mg/l) are the most important pollutants in the river model with their PIP of 6.14% and 6.1% respectively. Keywords : Posterior Inclusion Probability (PIP), Log-Predictive Score, Model Uncertainty, Dissolved Solids DOI : 10.7176/JEES/9-11-06 Publication date: November 30 th 2019

We investigate model uncertainty associated with predictive regressions employed in asset return forecasting research. We use simple combination and Bayesian model averaging (BMA) techniques to compare the performance of these forecasting approaches in short-vs. long-run horizons of S&P500 monthly excess returns. Simple averaging involves an equally-weighted averaging of the forecasts from alternative combinations of factors used in the predictive regressions, whereas BMA involves computing the predictive probability that each model is the true model and uses these predictive probabilities as weights in combing the forecasts from different models. From a given set of multiple factors, we evaluate all possible pricing models to the extent, which they describe the data as dictated by the posterior model probabilities. We find that, while simple averaging compares quite favorably to forecasts derived from a random walk model with drift (using a 10-year out-of-sample iterative period), BMA outperforms simple averaging in longer compared to shorter forecast horizons. Moreover, we find further evidence of the latter when the predictive Bayesian model includes shorter, rather than longer lags of the predictive factors. An interesting outcome of this study tends to illustrate the power of BMA in suppressing model uncertainty through model as well as parameter shrinkage, especially when applied to longer predictive horizons.

Bayesian model averaging (BMA) ranks the plausibility of alternative conceptual models according to Bayes' theorem. A prior belief about each model's adequacy is updated to a posterior model probability based on the skill to reproduce observed data and on the principle of parsimony. The posterior model probabilities are then used as model weights for model ranking, selection, or averaging. Despite the statistically rigorous BMA procedure, model weights can become uncertain quantities due to measurement noise in the calibration data set or due to uncertainty in model input. Uncertain weights may in turn compromise the reliability of BMA results. We present a new statistical concept to investigate this weighting uncertainty, and thus, to assess the significance of model weights and the confidence in model ranking. Our concept is to resample the uncertain input or output data and then to analyze the induced variability in model weights. In the special case of weighting uncertainty due to measurement noise in the calibration data set, we interpret statistics of Bayesian model evidence to assess the distance of a model's performance from the theoretical upper limit. To illustrate our suggested approach, we investigate the reliability of soil-plant model selection following up on a study by Wöhling et al. (2015). Results show that the BMA routine should be equipped with our suggested upgrade to (1) reveal the significant but otherwise undetected impact of measurement noise on model ranking results and (2) to decide whether the considered set of models should be extended with better performing alternatives.

Application of Bayesian model averaging in modeling long-term wind speed distributions

Tumor is one of the deadly diseases which is frequently to be found in animals. However, identifying whether an animal has a tumor still becomes a big challenge. Classification of tumor disease can be done through gene expression, which consists of hundreds of genes, but only a small number of samples is taken. This data structure is called microarray data having the characteristic of high-dimensional data. The choice of a single model can be a problem for high-dimensional data because it ignores model uncertainty. This research proposed to use Bayesian Model Averaging (BMA) to model the uncertainty model by averaging the posterior distribution of all best models, weighted by their posterior model probabilities. Selecting relevant genes to diagnose animal tumors is very important; hence, variable selection needs to be carried out. The selection of predictor variables is carried out by using the iterative BMA algorithm. The BMA results showed that from 335 gene expressions, 12 genes were selected to be relevant genes for classifying whether the animals have a tumor or normal. Moreover, from 2335 possible models formed, 12 of the best models are selected. The accuracy of BMA results is assessed using the Brier Score, resulting from a value indicating that the BMA model is good enough to classify animals, whether they have a tumor or not. This research has proven that BMA with logistic performance has very good predictability; hence, the method can be applied to classify other diseases.

We review the use of Bayesian model averaging in astrophysics. We first introduce the statistical basis of Bayesian model selection and model averaging. We discuss methods to calculate the model‐averaged posteriors, including Markov chain Monte Carlo (MCMC), nested sampling, population Monte Carlo, and reversible jump MCMC (RJMCMC). We then review some applications of Bayesian model averaging in astrophysics, including measurements of the dark energy and primordial power spectrum parameters in cosmology, cluster weak lensing, and Sunyaev–Zel'dovich effect data, estimating distances to Cepheids and classifying variable stars. © 2013 Wiley Periodicals, Inc. Statistical Analysis and Data Mining 6: 3–14, 2013

As all kinds of physics-based and data-driven models are emerging in the fields of hydrologic and hydraulic engineering, Bayesian model averaging (BMA) is one of the popular multi-model methods used to account for the various uncertainty sources in the flood modeling process and generate robust ensemble predictions based on multiple competitive candidate models. The reliability of BMA parameters (weights and variances) determines the accuracy of BMA predictions. However, the uncertainty in the BMA parameters with fixed values, which are usually obtained from the Expectation-Maximization (EM) algorithm, has not been adequately investigated in BMA-related applications over the past few decades. Given the limitations of the commonly used EM algorithm, the Metropolis-Hastings (M-H) algorithm, which is one of the most widely used algorithms in the Markov Chain Monte Carlo (MCMC) method, is proposed to estimate the BMA parameters and quantify their associated uncertainty. Both numerical experiments and the one-dimensional HEC-RAS models are employed to examine the applicability of the M-H algorithm with multiple independent Markov chains. The performances of the EM and M-H algorithms in the BMA analysis are compared based on the daily water stage predictions from 10 model configurations. The results show that the BMA weights estimated from both algorithms are comparable, while the BMA variances obtained from the M-H MCMC algorithm are closer to the given variances in the numerical experiment. Moreover, the normal proposal distribution used in the M-H algorithm can yield narrower distributions for the BMA weights than those from the uniform prior. Overall, the MCMC approach with multiple chains can provide more information associated with the uncertainty of BMA parameters and its prediction performance is better than the default EM algorithm in terms of multiple evaluation metrics as well as algorithm flexibility.

Accurate estimation of the mechanical property of aging pipes is critical to maintain the safety and to scheduling maintenance. Destructive testing for mechanical properties measurement is very expensive and sometime impossible. Inference methods are needed for estimating the bulk properties by multimodality surface material measurements from nondestructive testing, such as chemical composition, volume fraction and hardness. Bayesian network modeling is utilized to integrate the information from various types of surface measurements for a more accurate bulk mechanical property estimation. To improve the approximation of the actual underlying model and avoid the risk of overfitting, Bayesian model averaging (BMA) of Bayesian networks is implemented to account for Bayesian network model uncertainty. The models considered are weighted based on the posterior model probability. Markov Chain Monte Carlo sampling provides an effective way for numerically computing the marginal likelihoods, which are essential for obtaining the posterior model probabilities. The predictive performance of single best model and BMA are compared by logarithmic scoring rule. The predictive capability of the proposed method is evaluated. It is shown that the Bayesian network and model averaging approach can provide more reliable results in predicting the bulk mechanical properties of the pipelines.

This paper describes a new procedure of model selection in linear regression analysis to efficiently model and predict the formation permeability in non-cored intervals. The simplest way of model selection in regression model is to adopt stepwise elimination that depends on the probability of null hypotheses. However, the new method is the Bayesian Model Averaging (BMA) that selects the most appropriate model for a given outcome variable based on Bayes factors. Model selection process in BMA considers model's posterior probability and Bayesian Information Criterion (BIC). BMA produces a posterior distribution of the outcome factor that represents the weighted average of the posterior distributions of that factor for each likely model. Only five models are selected that their posterior probabilities sum to be equal to one. The best model selection has the maximum posterior probability and minimum Bayesian Information Criterion; nevertheless, the model subset selection is determined when the probability of a non-zero predictor's coefficient is more than 90 % for the best sampled model. That means this predictor has an effect on the outcome and should be included in the regression model. All these variables are shown in the computed Occam's window. The results showed that MBA algorithm has to achieve perfect t-test and analysis of variance especially regarding the rejection of null hypotheses for all the subselected independent variables. In addition, the reduced BMA model has been evaluated through Mean Square Prediction Error (MSPE) and residual analysis that clearly shows uncertainty reduction in permeability estimation. Therefore, the predicted permeability values for the entire formation depth are much more compatible with the measured data than the linear regression model.

This article provides a review of Bayesian model averaging as a means of optimizing the predictive performance of common statistical models applied to large-scale educational assessments. The Bayesian framework recognizes that in addition to parameter uncertainty, there is uncertainty in the choice of models themselves. A Bayesian approach to addressing the problem of model uncertainty is the method of Bayesian model averaging. Bayesian model averaging searches the space of possible models for a set of submodels that satisfy certain scientific principles and then averages the coefficients across these submodels weighted by each model's posterior model probability (PMP). Using the weighted coefficients for prediction has been shown to yield optimal predictive performance according to certain scoring rules. We demonstrate the utility of Bayesian model averaging for prediction in education research with three examples: Bayesian regression analysis, Bayesian logistic regression, and a recently developed approach for Bayesian structural equation modeling. In each case, the model-averaged estimates are shown to yield better prediction of the outcome of interest than any submodel based on predictive coverage and the log-score rule. Implications for the design of large-scale assessments when the goal is optimal prediction in a policy context are discussed.