True Posterior Research Articles

Probabilistic Forward Modeling of Galaxy Catalogs with Normalizing Flows

Evaluating the accuracy and calibration of the redshift posteriors produced by photometric redshift (photo-z) estimators is vital for enabling precision cosmology and extragalactic astrophysics with modern wide-field photometric surveys. Evaluating photo-z posteriors on a per-galaxy basis is difficult, however, as real galaxies have a true redshift but not a true redshift posterior. We introduce PZFlow, a Python package for the probabilistic forward modeling of galaxy catalogs with normalizing flows. For catalogs simulated with PZFlow, there is a natural notion of “true” redshift posteriors that can be used for photo-z validation. We use PZFlow to simulate a photometric galaxy catalog where each galaxy has a redshift, noisy photometry, shape information, and a true redshift posterior. We also demonstrate the use of an ensemble of normalizing flows for photo-z estimation. We discuss how PZFlow will be used to validate the photo-z estimation pipeline of the Dark Energy Science Collaboration, and the wider applicability of PZFlow for statistical modeling of any tabular data.

Bayesian Analysis of Exponential Random Graph Models Using Stochastic Gradient Markov Chain Monte Carlo

The exponential random graph model (ERGM) is a popular model for social networks, which is known to have an intractable likelihood function. Sampling from the posterior for such a model is a long-standing problem in statistical research. We analyze the performance of the stochastic gradient Langevin dynamics (SGLD) algorithm (also known as noisy Longevin Monte Carlo) in tackling this problem, where the stochastic gradient is calculated via running a short Markov chain (the so-called inner Markov chain in this paper) at each iteration. We show that if the model size grows with the network size slowly enough, then SGLD converges to the true posterior in 2-Wasserstein distance as the network size and iteration number become large regardless of the length of the inner Markov chain performed at each iteration. Our study provides a scalable algorithm for analyzing large-scale social networks with possibly high-dimensional ERGMs.

Bernstein flows for flexible posteriors in variational Bayes

Black-box variational inference (BBVI) is a technique to approximate the posterior of Bayesian models by optimization. Similar to MCMC, the user only needs to specify the model; then, the inference procedure is done automatically. In contrast to MCMC, BBVI scales to many observations, is faster for some applications, and can take advantage of highly optimized deep learning frameworks since it can be formulated as a minimization task. In the case of complex posteriors, however, other state-of-the-art BBVI approaches often yield unsatisfactory posterior approximations. This paper presents Bernstein flow variational inference (BF-VI), a robust and easy-to-use method flexible enough to approximate complex multivariate posteriors. BF-VI combines ideas from normalizing flows and Bernstein polynomial-based transformation models. In benchmark experiments, we compare BF-VI solutions with exact posteriors, MCMC solutions, and state-of-the-art BBVI methods, including normalizing flow-based BBVI. We show for low-dimensional models that BF-VI accurately approximates the true posterior; in higher-dimensional models, BF-VI compares favorably against other BBVI methods. Further, using BF-VI, we develop a Bayesian model for the semi-structured melanoma challenge data, combining a CNN model part for image data with an interpretable model part for tabular data, and demonstrate, for the first time, the use of BBVI in semi-structured models.

BPMB: BayesCNNs with perturbed multi-branch structure for robust facial expression recognition

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets.

Addressing the Binning Problem in Calibration Assessment through Scalar Annotations

Abstract Computational linguistics models commonly target the prediction of discrete—categorical—labels. When assessing how well-calibrated these model predictions are, popular evaluation schemes require practitioners to manually determine a binning scheme: grouping labels into bins to approximate true label posterior. The problem is that these metrics are sensitive to binning decisions. We consider two solutions to the binning problem that apply at the stage of data annotation: collecting either distributed (redundant) labels or direct scalar value assignment. In this paper, we show that although both approaches address the binning problem by evaluating instance-level calibration, direct scalar assignment is significantly more cost-effective. We provide theoretical analysis and empirical evidence to support our proposal for dataset creators to adopt scalar annotation protocols to enable a higher-quality assessment of model calibration.

Novel Screw Placement Method for Extremely Small Lumbar Pedicles in Scoliosis.

Study Design: Consecutive case series. Objective: To propose a screw placement method in patients with extremely small lumbar pedicles (ESLPs) (<2 mm) to maintain screw density and correction power, without relying on the O-arm navigation system. Summary of Background Data: In scoliosis surgery, ESLPs can hinder probe passage, resulting in exclusion or substitution of the pedicle screws with a hook. Screw density affects correction power, making it necessary to maximize the number of screw placements, especially in the lumbar curve. Limited studies provide technical guidelines for screw placement in patients with ESLPs, independent of the O-arm navigation system. Methods: We enrolled 19 patients who underwent scoliosis correction surgery using our novel screw placement method for ESLPs. Clinical, radiological, and surgical parameters were assessed. After posterior exposure of the spine, the C-arm fluoroscope was rotated to obtain a true posterior-anterior view and both pedicles were symmetrically visualized. An imaginary pedicle outline was presumed based on the elliptical or linear shadow from the pedicle. The screw entry point was established at a 2 (or 10) o'clock position in the presumed pedicle outline. After adjusting the gear-shift convergence, both cortices of the transverse process were penetrated and the tip was advanced towards the lateral vertebral body wall, where an extrapedicular screw was placed with tricortical fixation. Results: Out of 90 lumbar screws in 19 patients, 33 screws were inserted using our novel method, without correction loss or complications during an average follow-up period of 28.44 months, except radiological loosening of one screw. Conclusions: Our new extrapedicular screw placement method into the vertebral body provides an easy, accurate, and safe alternative for scoliosis patients with ESLPs without relying on the O-arm navigation system. Surgeons must consider utilizing this method to enhance correction power in scoliosis surgery, regardless of the small size of the lumbar pedicle.

Neural parameter calibration and uncertainty quantification for epidemic forecasting.

The recent COVID-19 pandemic has thrown the importance of accurately forecasting contagion dynamics and learning infection parameters into sharp focus. At the same time, effective policy-making requires knowledge of the uncertainty on such predictions, in order, for instance, to be able to ready hospitals and intensive care units for a worst-case scenario without needlessly wasting resources. In this work, we apply a novel and powerful computational method to the problem of learning probability densities on contagion parameters and providing uncertainty quantification for pandemic projections. Using a neural network, we calibrate an ODE model to data of the spread of COVID-19 in Berlin in 2020, achieving both a significantly more accurate calibration and prediction than Markov-Chain Monte Carlo (MCMC)-based sampling schemes. The uncertainties on our predictions provide meaningful confidence intervals e.g. on infection figures and hospitalisation rates, while training and running the neural scheme takes minutes where MCMC takes hours. We show convergence of our method to the true posterior on a simplified SIR model of epidemics, and also demonstrate our method's learning capabilities on a reduced dataset, where a complex model is learned from a small number of compartments for which data is available.

Reduced-order model-based variational inference with normalizing flows for Bayesian elliptic inverse problems

We propose a reduced-order model-based variational inference method with normalizing flows for Bayesian elliptic inverse problems. The coefficient of the elliptic PDE is represented by a finite number of parameters. We aim to estimate the posterior distributions of these parameters in the framework of variational inference, and the approximation of the posterior distribution is constructed through a normalizing flow. Moreover, as data of inputs and outputs of the forward problem are accumulated during the training of the normalizing flow, we can naturally exploit the low-dimensional intrinsic structure of the forward elliptic PDE using reduced-order models based on these data, in which information of the true posterior is incorporated. We construct a low-dimensional set of data-driven basis functions in the solution space using proper orthogonal decomposition (POD) and train a neural network that maps the parameters to the coefficients of these data-driven basis functions. The surrogate forward map, which is the combination of the reduced-order model and the parameter-to-coefficient neural network, is then applied to the Bayesian elliptic inverse problem, where in each iteration the forward problem is evaluated in the space spanned by the POD basis functions using the surrogate forward map. Thus the inversion will be significantly accelerated as the surrogate forward map is essentially a low-dimensional model. We present numerical examples to show the accuracy and efficiency of the proposed method.

Gaussian Variational Approximations for High-dimensional State Space Models

We consider a Gaussian variational approximation of the posterior density in high-dimensional state space models. The number of parameters in the covariance matrix of the variational approximation grows as the square of the number of model parameters, so it is necessary to find simple yet effective parametrisations of the covariance structure when the number of model parameters is large. We approximate the joint posterior density of the state vectors by a dynamic factor model, having Markovian time dependence and a factor covariance structure for the states. This gives a reduced description of the dependence structure for the states, as well as a temporal conditional independence structure similar to that in the true posterior. We illustrate the methodology on two examples. The first is a spatio-temporal model for the spread of the Eurasian collared-dove across North America. Our approach compares favorably to a recently proposed ensemble Kalman filter method for approximate inference in high-dimensional hierarchical spatio-temporal models. Our second example is a Wishart-based multivariate stochastic volatility model for financial returns, which is outside the class of models the ensemble Kalman filter method can handle.

Constraining cosmological parameters from N-body simulations with variational Bayesian neural networks

Introduction: Methods based on deep learning have recently been applied to recover astrophysical parameters, thanks to the ability of these techniques to capture information from complex data. One of these schemes is the approximate Bayesian neural network (BNN), which has demonstrated to yield a posterior distribution into the parameter space that is extremely helpful for uncertainty quantification. However, modern neural networks tend to produce overly confident uncertainty estimates and introduce bias when applying BNNs to data.Method: In this work, we implement multiplicative normalizing flows (MNFs), a family of approximate posteriors for the parameters of BNNs with the purpose of enhancing the flexibility of the variational posterior distribution, to extract Ωm, h, and σ8 from the QUIJOTE simulations. We compared the latter method with the standard BNNs and the Flipout estimator.Results: We have found that the use of MNFs consistently outperforms the standard BNNs with a percent difference in the mean squared error of 21%, in addition to high-accuracy extraction of σ8 (r2 = 0.99), with precise and consistent uncertainty estimates.Discussions: These findings imply that MNFs provide a more realistic predictive distribution closer to the true posterior, mitigating the bias introduced by the variational approximation and allowing us to work with well-calibrated networks.

Hierarchical Graph Neural Network Based on Semi-Implicit Variational Inference

Graph neural network(GNN) has obtained outstanding achievements in relational data. However, these data have uncertain properties, for example, spurious edges may be included. Recently, Variational graph autoencoder(VGAE) has been proposed to solve this problem. However, the distributional assumptions in the variational family restrict the variational inference (VI) flexibility and they define variational families using mean-field, which can not capture complex posterior distributional. To solve the above question, in this paper, we proposed a novel GNN model based on semi-implicit variational inference (SIVI), which can embed the node to the latent space to improve VI flexibility and enhance VI expressiveness with mixing distribution. Specifically, to approximate the true posterior, a variational posterior was given utilizing a semi-implicit hierarchical variational framework, which can model complex posterior. Moreover, an iterative decoder is used to better capture graph properties. Besides, due to the hierarchical structure in our model, it can incorporation neighbour information between nodes. Experiments on multiple data sets, our method has achieved state-of-the-art results compared to other similar methods. Particularly, on the citation dataset Citeseer without features, our method outperforms VGAE by nine percentage.

Dynamical Non-compensatory Multidimensional IRT Model Using Variational Approximation.

Multidimensional item response theory (MIRT) is a statistical test theory that precisely estimates multiple latent skills of learners from the responses in a test. Both compensatory and non-compensatory models have been proposed for MIRT: the former assumes that each skill can complement other skills, whereas the latter assumes they cannot. This non-compensatory assumption is convincing in many tests that measure multiple skills; therefore, applying non-compensatory models to such data is crucial for achieving unbiased and accurate estimation. In contrast to tests, latent skills will change over time in daily learning. To monitor the growth of skills, dynamical extensions of MIRT models have been investigated. However, most of them assumed compensatory models, and a model that can reproduce continuous latent states of skills under the non-compensatory assumption has not been proposed thus far. To enable accurate skill tracing under the non-compensatory assumption, we propose a dynamical extension of non-compensatory MIRT models by combining a linear dynamical system and a non-compensatory model. This results in a complicated posterior of skills, which we approximate with a Gaussian distribution by minimizing the Kullback-Leibler divergence between the approximated posterior and the true posterior. The learning algorithm for the model parameters is derived through Monte Carlo expectation maximization. Simulation studies verify that the proposed method is able to reproduce latent skills accurately, whereas the dynamical compensatory model suffers from significant underestimation errors. Furthermore, experiments on an actual data set demonstrate that our dynamical non-compensatory model can infer practical skill tracing and clarify differences in skill tracing between non-compensatory and compensatory models.

Gaussian Mixture Filtering with Nonlinear Measurements Minimizing Forward Kullback-Leibler Divergence

A Gaussian mixture filter is proposed for the state estimation of dynamical systems with nonlinear measurements. The filter is derived by solving an assumed density filtering problem where Kullback-Leibler (KL) divergence from the assumed posterior, which is a Gaussian mixture, to the true posterior (which we call forward KL divergence) is minimized. The approximate solution to this problem gives an iterative measurement update by which the weights, means and covariances of the assumed posterior are optimized. The resulting Gaussian mixture filter is shown to be a generalization of the (damped) posterior linearization filter to Gaussian mixture posteriors. The performance of the proposed filter is illustrated and compared to alternatives on a challenging example of target tracking in a sensor network and on an example of tracking with a radar. The results show that the proposed filter can outperform Gaussian filters as well as the Gaussian sum filter and the Gaussian sum particle filter yielding results very close to a bootstrap particle filter when the number of components in the assumed posterior is sufficiently large.

Divide-and-conquer Bayesian inference in hidden Markov models

Divide-and-conquer Bayesian methods consist of three steps: dividing the data into smaller computationally manageable subsets, running a sampling algorithm in parallel on all the subsets, and combining parameter draws from all the subsets. These methods use the combined parameter draws for efficient posterior inference in massive data settings. Existing divide-and-conquer methods have a major limitation in that their first two steps assume that the observations are independent. We address this problem by developing a divide-and-conquer method for Bayesian inference in parametric hidden Markov models, where the state space is known and finite. Our main contributions are two-fold. First, we partition the data into smaller blocks of consecutive observations and modify the likelihood on every time block. For any time block, the posterior distribution computed using the modified likelihood is such that its variance has the same asymptotic order as that of the true posterior. Second, suppose the number of subsets is chosen appropriately depending on the mixing properties of the hidden Markov chain. In that case, we show that the subset posterior distributions defined using the modified likelihood are asymptotically normal as the subset sample size tends to infinity. This result facilitates using any existing combination algorithm in the third step. We show that the combined posterior distribution obtained using one such algorithm is close to the true posterior distribution in 1-Wasserstein distance under widely used regularity assumptions. Our numerical results show that the proposed method provides an accurate approximation of the true posterior distribution than its competitors in simulation studies and a real data analysis.

Rapid parameter estimation for an all-sky continuous gravitational wave search using conditional variational auto-encoders

All-sky searches for continuous gravitational waves are generally model dependent and computationally costly to run. By contrast, SOAP is a model-agnostic search that rapidly returns candidate signal tracks in the time-frequency plane. In this work we extend the SOAP search to return broad Bayesian posteriors on the astrophysical parameters of a specific signal model. These constraints drastically reduce the volume of parameter space that any follow-up search needs to explore, so increasing the speed at which candidates can be identified and confirmed. Our method uses a machine learning technique, specifically a conditional variational auto-encoder, and delivers a rapid estimation of the posterior distribution of the four Doppler parameters of a continuous wave signal. It does so without requiring a clear definition of a likelihood function, or being shown any true Bayesian posteriors in training. We demonstrate how the Doppler parameter space volume can be reduced by a factor of $\mathcal{O}(10^{-7})$ for signals of SNR 100.

Learning Summary Statistics for Bayesian Inference with Autoencoders

For stochastic models with intractable likelihood functions, approximate Bayesian computation offers a way of approximating the true posterior through repeated comparisons of observations with simulated model outputs in terms of a small set of summary statistics. These statistics need to retain the information that is relevant for constraining the parameters but cancel out the noise. They can thus be seen as thermodynamic state variables, for general stochastic models. For many scientific applications, we need strictly more summary statistics than model parameters to reach a satisfactory approximation of the posterior. Therefore, we propose to use a latent representation of deep neural networks based on Autoencoders as summary statistics. To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information on the noise that has been used to generate the training data. We validate the approach empirically on two types of stochastic models.

Macroscopic Analysis of Vector Approximate Message Passing in a Model-Mismatched Setting

In this study, macroscopic properties of the vector approximate message passing (VAMP) algorithm for inference of generalized linear models are investigated using a non-rigorous heuristic method of statistical mechanics when the true posterior cannot be used and the measurement matrix is a sample from rotation-invariant random matrix ensembles. The focus is on the correspondence between the non-rigorous replica analysis of statistical mechanics and the performance assessment of VAMP in the model-mismatched setting. The correspondence of this kind is well-known when the measurement matrix has independent and identically distributed entries. However, when the measurement matrix follows a general rotation-invariant matrix ensemble, the correspondence has been validated only under limited cases, such as the Bayes optimal inference or the convex empirical risk minimization. The result presented in this paper is to extend the scope of such correspondence. Herein, we heuristically derive the explicit formula of state-evolution equations, which macroscopically describe VAMP dynamics for the current model-mismatched case, and show that their fixed point is generally consistent with the replica symmetric solution obtained by the replica method of statistical mechanics. We also show that the fixed point of VAMP can exhibit a microscopic instability, which indicates that message variables continue to move by VAMP while their macroscopically summarized quantities converge to fixed values. The critical condition the for microscopic instability agrees with that for breaking the replica symmetry that is derived within the non-rigorous replica analysis. The results of the numerical experiments cross-check our findings.

A Probabilistic Framework for Land Deformation Prediction (Student Abstract)

The development of InSAR (satellite Interferometric Synthetic Aperture Radar) enables accurate monitoring of land surface deformations, and has led to advances of deformation forecast for preventing landslide, which is one of the severe geological disasters. Despite the unparalleled success, existing spatio-temporal models typically make predictions on static adjacency relationships, simplifying the conditional dependencies and neglecting the distributions of variables. To overcome those limitations, we propose a Distribution Aware Probabilistic Framework (DAPF), which learns manifold embeddings while maintaining the distribution of deformations. We obtain a dynamic adjacency matrix upon which we approximate the true posterior while emphasizing the spatio-temporal characteristics. Experimental results on real-world dataset validate the superior performance of our method.

Bayesian Distance Weighted Discrimination

Distance weighted discrimination (DWD) is a linear discrimination method that is particularly well-suited for classification tasks with high-dimensional data. The DWD coefficients minimize an intuitive objective function, which can solved efficiently using state-of-the-art optimization techniques. However, DWD has not yet been cast into a model-based framework for statistical inference. In this article we show that DWD identifies the mode of a proper Bayesian posterior distribution, that results from a particular link function for the class probabilities and a shrinkage-inducing proper prior distribution on the coefficients. We describe a relatively efficient Markov chain Monte Carlo (MCMC) algorithm to simulate from the true posterior under this Bayesian framework. We show that the posterior is asymptotically normal and derive the mean and covariance matrix of its limiting distribution. Through several simulation studies and an application to breast cancer genomics we demonstrate how the Bayesian approach to DWD can be used to (a) compute well-calibrated posterior class probabilities, (b) assess uncertainty in the DWD coefficients and resulting sample scores, (c) improve power via semisupervised analysis when not all class labels are available, and (d) automatically determine a penalty tuning parameter within the model-based framework. R code to perform Bayesian DWD is available at https://github.com/lockEF/BayesianDWD. Supplementary materials for this article are available online.

Fast increased fidelity samplers for approximate Bayesian Gaussian process regression.

Gaussian processes (GPs) are common components in Bayesian non-parametric models having a rich methodological literature and strong theoretical grounding. The use of exact GPs in Bayesian models is limited to problems containing several thousand observations due to their prohibitive computational demands. We develop a posterior sampling algorithm using -matrix approximations that scales at . We show that this approximation's Kullback-Leibler divergence to the true posterior can be made arbitrarily small. Though multidimensional GPs could be used with our algorithm, d-dimensional surfaces are modeled as tensor products of univariate GPs to minimize the cost of matrix construction and maximize computational efficiency. We illustrate the performance of this fast increased fidelity approximate GP, FIFA-GP, using both simulated and non-synthetic data sets.