Approximation Of Posterior Distribution Research Articles

A collaborative filtering recommendation algorithm based on embedding representation

This aims to propose a collaborative filtering algorithm based on embedding representation and word embedding techniques, namely UI2vec. According to the joint feature extraction network designed in this paper, UI2vec embeds users and items on the potential space at the same time, and uses the item similarity between them to predict the user’s content of interest. Then a generative model VUI2vec with more stable performance is proposed based on UI2vec, which maps users and items as independent Gaussian distributions and obtains the approximate posterior distribution of both by variational inference. The recommendation performance of UI2vec and VUI2vec is evaluated on TaFeng, Movielens, and Netflix datasets. The impact of important superparameters within the model on performance is investigated. The experimental results show that compared with the baseline model, the proposed methods performs consistently well.

Likelihood-free Hamiltonian Monte Carlo for modeling piping degradation and remaining useful life prediction using the mixed gamma process

The gamma process is a popular model to characterize the temporal uncertainty in degradation processes. According to this model, degradation occurs in tiny independent increments creating a monotonic non-decreasing trend. Mechanical components or units are often repetitively observed over time for measuring the extent of degradation; apparently, they degrade at different rates although operating in the same environment. The mixed gamma degradation process is suitable for capturing these unexplained differences. However, statistical estimation of its parameters and subsequent prediction of remaining useful life (RUL) turn challenging with limited and noisy measurements.While limited observations bring uncertainties to the model parameters, noisy data turns the likelihood function intractable. In such cases, approximate Bayesian computation (ABC) methods are widely applicable for their ability to bypass the likelihood evaluation and generate samples from an approximate target posterior distribution. The currently available Markov chain Monte Carlo (MCMC) based ABC method shows promising results, although it requires significant thinning due to poor mixing of samples. A new ABC method is derived in this paper which imitates the sampling process of the Hamiltonian Monte Carlo (HMC) method. The proposed ABC-HMC algorithm explores a target probability space more effectively — generating better mixing of samples and requiring less thinning than the MCMC based ABC method. Proof of convergence of ABC-HMC is shown by fulfilling the detailed balance condition. The application of the method is shown using real degradation data of piping components of the heat transport system of a nuclear reactor.

Emulation of greenhouse‐gas sensitivities using variational autoencoders

AbstractFlux inversion is the process by which sources and sinks of a gas are identified from observations of gas mole fraction. The inversion often involves running a Lagrangian particle dispersion model (LPDM) to generate simulations of the gas movement over a domain of interest. The LPDM must be run backward in time for every gas measurement, and this can be computationally prohibitive. To address this problem, here we develop a novel spatio‐temporal emulator for LPDM sensitivities that is built using a convolutional variational autoencoder (CVAE, a two‐piece neural network capable of condensing and reconstructing images). With the encoder segment of the CVAE, we obtain approximate (variational) posterior distributions over latent variables in a low‐dimensional space. We then use a spatio‐temporal Gaussian process emulator on the low‐dimensional space to emulate new variables at prediction locations and time points. Emulated variables are then passed through the decoder segment of the CVAE to yield emulated sensitivities. We show that our CVAE‐based emulator outperforms the more traditional emulator built using empirical orthogonal functions and that it can be used with different LPDMs. We conclude that our emulation‐based approach can be used to reliably reduce the computing time needed to generate LPDM outputs for use in high‐resolution flux inversions.

Graph Neural Network Aided MU-MIMO Detectors

Multi-user multiple-input multiple-output (MU-MIMO) systems can be used to meet high throughput requirements of 5G and beyond networks. A base station serves many users in an uplink MU-MIMO system, leading to a substantial multi-user interference (MUI). Designing a high-performance detector for dealing with a strong MUI is challenging. This paper analyses the performance degradation caused by the posterior distribution approximation used in the state-of-the-art message passing (MP) detectors in the presence of high MUI. We develop a graph neural network based framework to fine-tune the MP detectors’ cavity distributions and thus improve the posterior distribution approximation in the MP detectors. We then propose two novel neural network based detectors which rely on the expectation propagation (EP) and Bayesian parallel interference cancellation (BPIC), referred to as the GEPNet and GPICNet detectors, respectively. The GEPNet detector maximizes detection performance, while GPICNet detector balances the performance and complexity. We provide proof of the permutation equivariance property, allowing the detectors to be trained only once, even in the systems with dynamic changes of the number of users. The simulation results show that the proposed GEPNet detector performance approaches maximum likelihood performance in various configurations and GPICNet detector doubles the multiplexing gain of BPIC detector.

Probabilistic Identification of Multi-DOF Structures Subjected to Ground Motion Using Manifold-Constrained Gaussian Processes

Bayesian uncertainty quantification has a pivotal role in structural identification, yet the posterior distribution estimation of unknown parameters and system responses is still a challenging task. This study explores a novel method, named manifold-constrained Gaussian processes (GPs), for the probabilistic identification of multi-DOF structural dynamical systems, taking shear-type frames subjected to ground motion as a demonstrative paradigm. The key idea of the method is to restrict the GPs (priorly defined over system responses) on a manifold that satisfies the equation of motion of the structural system. In contrast to widely used Bayesian probabilistic model updating methods, the manifold-constrained GPs avoid the numerical integration when formulating the joint probability density function of unknown parameters and system responses, hence achieving an accurate and computationally efficient inference for the posterior distributions. An eight-storey shear-type frame is analyzed as a case study to demonstrate the effectiveness of the manifold-constrained GPs. The results indicate the posterior distributions of system responses, and unknown parameters can be successfully identified, and reliable probabilistic model updating can be achieved.

Bayesian origin-destination estimation in networked transit systems using nodal in- and outflow counts

We propose a Bayesian inference approach for static Origin-Destination (OD)-estimation in large-scale networked transit systems. The approach finds posterior distribution estimates of the OD-coefficients, which describe the relative proportions of passengers travelling between origin and destination locations, via a Hamiltonian Monte Carlo sampling procedure. We suggest two different inference model formulations, the instantaneous-balance and average-delay model.The average-delay model is generally more robust in determining accurate and precise coefficient posteriors across various combinations of observation properties. The instantaneous-balance model, however, requires lower resolution count observations and produces estimates comparable to the average-delay model, pending that certain count observation properties are met. We demonstrate that the Bayesian posterior distribution estimates provide quantifiable measures of the estimation uncertainty and prediction quality of the model. Moreover, the Bayesian approach is at least as accurate as existing optimisation approaches and proves robust in scaling to high-dimensional underdetermined problems without suffering from the curse of dimensionality. The Bayesian instantaneous-balance model is applied to the New York City subway network, with several years of entry and exit count observations recorded at several hundred station turnstiles across the network. The posterior distribution estimates provide intuitive demand patterns and are projected to be more valuable than point estimates, since they allow for robust transport network designs that account for the uncertainty of network parameters.

Implicit optimal variational collaborative filtering

Recently, variational autoencoder (VAE), a model that supports Bayesian inference and variational posterior distribution approximation, has been used to address the prevailing challenge of inefficient modelling of non-linear user-item interactions in collaborative filtering (CF). However, the choices of prior distributions in these VAE-based CF models are inadequate to determine the underlying distributions, leading to skewed maximum likelihood estimation of model parameters. In this paper, we present an Implicit Optimal Variational autoencoder model for collaborative filtering (IOVA-CF) to alleviate this challenge. Specifically, IOVA-CF utilizes a novel implicit optimal prior, which aids in generating excellent latent representations. This implicit optimal prior (IoP) is an aggregated posterior prior (AP), that is, the posterior’s expectation of the data distribution. However, unlike previous AP, IoP utilizes the density ratio technique to compute the Kullback-Leibler (KL) divergence without explicitly capturing the aggregated posterior. Besides, we can calculate the KL divergence between the aggregated posterior and the inference network in a closed-form using the density ratio technique, making IoP an optimal prior for maximizing the objective function. Additionally, unlike earlier VAE-based recommendation models, IOVA-CF significantly alleviates the over-regularization issue. Furthermore, IOVA-CF can adequately capture the latent space’s uncertainty. Finally, empirical evaluations with several competitive baseline models on four (4) real-world datasets reveal the superior performance of IOVA-CF.

Variational Bootstrap for Classification

Carrying out Bayesian inference over parameters of statistical models is intractable when the likelihood and the prior are non-conjugate. Variational bootstrap provides a way to obtain samples from the posterior distribution over model parameters, where each sample is the solution of a task where the labels are perturbed. For Bayesian linear regression with a Gaussian likelihood, variational bootstrap yields samples from the exact posterior, whereas for nonlinear models with a Gaussian likelihood some guarantees of approaching the true posterior can be established. In this work, we extend variational bootstrap to the Bernoulli likelihood to tackle classification tasks. We use a transformation of the labels which allows us to turn the classification task into a regression one, and then we apply variational bootstrap to obtain samples from an approximate posterior distribution over the parameters of the model. Variational bootstrap allows us to employ advanced gradient optimization techniques which provide fast convergence. We provide experimental evidence that the proposed approach allows us to achieve classification accuracy and uncertainty estimation comparable with MCMC methods at a fraction of the cost.

Complex Variational Inference Network for PolSAR Classification

For maintaining the phase information in images, complex neural networks have been widely applied to PolSAR classification. However, due to constant weights of neurons, the networks may lack randomness and be potentially overfitting for complicated imaging mechanisms and random speckle noise in PolSAR images. Thus, this letter proposes a complex variational inference network (CVIN) where complex Gaussian probability distributions are introduced into the weights of neurons in complex neural networks. In CVIN, a novel evidence lower bound (ELBO) for complex network is designed to infer the variational approximation of weights through backpropagation. After training, CVIN propagates the approximate posterior distributions given the data and makes the prediction of the labels. Thus, CVIN is an ensemble of flexible models with infinite weights, where the complex weights are regularized by the Gaussian distributions. Experiments on real PolSAR images verify the feasibility of CVIN and illustrate the potential of CVIN to serve as a competitive method for PolSAR classification.

Conditional Variational Autoencoder for Learned Image Reconstruction

Learned image reconstruction techniques using deep neural networks have recently gained popularity and have delivered promising empirical results. However, most approaches focus on one single recovery for each observation, and thus neglect information uncertainty. In this work, we develop a novel computational framework that approximates the posterior distribution of the unknown image at each query observation. The proposed framework is very flexible: it handles implicit noise models and priors, it incorporates the data formation process (i.e., the forward operator), and the learned reconstructive properties are transferable between different datasets. Once the network is trained using the conditional variational autoencoder loss, it provides a computationally efficient sampler for the approximate posterior distribution via feed-forward propagation, and the summarizing statistics of the generated samples are used for both point-estimation and uncertainty quantification. We illustrate the proposed framework with extensive numerical experiments on positron emission tomography (with both moderate and low-count levels) showing that the framework generates high-quality samples when compared with state-of-the-art methods.

Detecting conflicting summary statistics in likelihood-free inference

Bayesian likelihood-free methods implement Bayesian inference using simulation of data from the model to substitute for intractable likelihood evaluations. Most likelihood-free inference methods replace the full data set with a summary statistic before performing Bayesian inference, and the choice of this statistic is often difficult. The summary statistic should be low-dimensional for computational reasons, while retaining as much information as possible about the parameter. Using a recent idea from the interpretable machine learning literature, we develop some regression-based diagnostic methods which are useful for detecting when different parts of a summary statistic vector contain conflicting information about the model parameters. Conflicts of this kind complicate summary statistic choice, and detecting them can be insightful about model deficiencies and guide model improvement. The diagnostic methods developed are based on regression approaches to likelihood-free inference, in which the regression model estimates the posterior density using summary statistics as features. Deletion and imputation of part of the summary statistic vector within the regression model can remove conflicts and approximate posterior distributions for summary statistic subsets. A larger than expected change in the estimated posterior density following deletion and imputation can indicate a conflict in which inferences of interest are affected. The usefulness of the new methods is demonstrated in a number of real examples.

A new framework for experimental design using Bayesian Evidential Learning: The case of wellhead protection area

Groundwater management practices, such as sustainable drinking water extraction or contamination protection, can have significant socioeconomic impacts. A complete uncertainty analysis should ideally be performed to anticipate all possible outcomes and assess any risk. Uncertainties arise as a result of our limited understanding of the physical processes involved, as well as a scarcity of measurement data, whether directly or indirectly related to the physical parameters of interest. In this paper, we use a small number of tracing experiments (predictor) to predict the wellhead protection area (WHPA, target), the shape and extent of which are influenced by the distribution of hydraulic conductivity (K). Our first goal is to make stochastic predictions of the WHPA within the Bayesian Evidential Learning (BEL) framework, which uses machine learning to find a direct relationship between predictor and target. This relationship is learned using a small number of training models (400) drawn from the prior distribution of K. Forward modelling is used to obtain the 400 pairs of simulated predictors and targets. Newly collected field data can then be used directly to predict the approximate posterior distribution of the corresponding WHPA, obviating the need for the traditional step of data inversion. The number and location of data sources (injection wells) influence the posterior WHPA distribution's uncertainty range. Our second goal is to extend BEL to determine the optimal design of data source locations that minimises the WHPA's posterior uncertainty. Because the BEL model, once trained, allows the computation of the posterior uncertainty corresponding to any new input data, experimental design can be done explicitly, without averaging or approximating. We estimate the WHPA's posterior uncertainty range using the Modified Hausdorff Distance (MHD) and Structural Similarity (SSIM) index metrics. Because the breakthrough curves store information on a large area of the K field surrounding the pumping well, increasing the number of injection wells reduces the derived posterior WHPA uncertainty. Our method can also estimate which injection wells are more informative than others, as demonstrated by a k-fold cross-validation procedure. Overall, the application of BEL to experimental design allows for identifying data sources that maximise the information content of any measurement data while keeping budget constraints and computational costs to a minimum.

Non-Gaussian estimates of tensions in cosmological parameters

We discuss how to efficiently and reliably estimate the level of agreement and disagreement on parameter determinations from different experiments, fully taking into account non-Gaussianities in the parameter posteriors. We develop two families of scalable algorithms that allow us to perform this type of calculations in an increasing number of dimensions and for different levels of tensions. One family of algorithms rely on kernel density estimates of posterior distributions while the other relies on machine learning modeling of the posterior distribution with normalizing flows. We showcase their effectiveness and accuracy with a set of benchmark examples and find both methods agree with each other and the true tension within $0.5\ensuremath{\sigma}$ in difficult cases and generally to $0.2\ensuremath{\sigma}$ or better. This allows us to study the level of internal agreement between different measurements of the clustering of cosmological structures from the Dark Energy Survey and their agreement with measurements of the cosmic microwave background from the Planck satellite.

Accelerating Bayesian Estimation for Network Poisson Models Using Frequentist Variational Estimates

Abstract This work is motivated by the analysis of ecological interaction networks. Poisson stochastic block models are widely used in this field to decipher the structure that underlies a weighted network, while accounting for covariate effects. Efficient algorithms based on variational approximations exist for frequentist inference, but without statistical guaranties as for the resulting estimates. In the absence of variational Bayes estimates, we show that a good proxy of the posterior distribution can be straightforwardly derived from the frequentist variational estimation procedure, using a Laplace approximation. We use this proxy to sample from the true posterior distribution via a sequential Monte Carlo algorithm. As shown in the simulation study, the efficiency of the posterior sampling is greatly improved by the accuracy of the approximate posterior distribution. The proposed procedure can be easily extended to other latent variable models. We use this methodology to assess the influence of available covariates on the organization of several ecological networks, as well as the existence of a residual interaction structure.

Principled Decision-Making Workflow with Hierarchical Bayesian Models of High-Throughput Dose-Response Measurements.

We present a case study applying hierarchical Bayesian estimation on high-throughput protein melting-point data measured across the tree of life. We show that the model is able to impute reasonable melting temperatures even in the face of unreasonably noisy data. Additionally, we demonstrate how to use the variance in melting-temperature posterior-distribution estimates to enable principled decision-making in common high-throughput measurement tasks, and contrast the decision-making workflow against simple maximum-likelihood curve-fitting. We conclude with a discussion of the relative merits of each workflow.

Conditional Prior Probabilistic Generative Model With Similarity Measurement for ISAR Imaging

The higher bandwidth inverse synthetic aperture radar (ISAR) can obtain the higher resolution radar images, which can provide more target information and help improve radar target detection and recognition. It is essential to study how to achieve a precise high-resolution (HR) ISAR image utilizing limited measurement echoes. The existing neural-network-based ISAR imaging methods extract features only from limited measurement echoes, and the common features in HR ISAR images are not utilized sufficiently, which limits the imaging performance improvement. Moreover, in their loss functions, there are no explicit constraints on the correct recovery of strong scattering points, which are important in reflecting the target characteristics. In this letter, we propose a conditional probabilistic generative model to achieve the HR ISAR imaging. By optimizing the well-designed Kullback-Leibler (KL) divergence between conditional prior and approximate posterior probability distribution in the loss function, the common features contained in training HR radar images can be learned, and a suitable prior probability distribution for the latent variable can be obtained. To accurately recover the positions and relative amplitudes of strong scattering points, we blend a similarity measurement that is sensitive to the large values' locations in a vector with the adversarial loss. Both visual and numerical results of extensive experiments prove that the proposed model can obtain enhanced effectiveness and efficiency compared with some counterparts.

Bayesian signal processing: A practical guide to particle filtering design and performance

When uncertain acoustic processes can no longer be characterized by Gaussian or for that matter unimodal (single peak) distributions along with the fact that the underlying phenomenology is nonstationary (time-varying) and nonlinear, then more general Bayesian processors must be applied to solve the underlying signal enhancement/extraction problem. A particle filter provides a solution to this multimodal (multiple peaks) posterior distribution estimation problem in noisy acoustic environments. A particle filter is a sequential Markov chain Monte Carlo processor capable of providing reasonable performance for data evolving from a multimodal distribution by estimating a nonparametric representation of the posterior distribution from which a multitude of meaningful statistics can be retrieved. However, the question of evaluating its performance can be challenging even for the simplest of processes. For instance, it is well-known that Kalman filter optimality can be obtained only when the resulting error residuals (innovations) are zero-mean and white. It is not that simple for the particle filter. Once characterized, the performance of the particle filter must be analyzed for it to be of practical value. Here a set of design and analysis criteria is discussed and applied to demonstrate their ability to quantify particle filtering performance.

Probabilistic dipole inversion for adaptive quantitative susceptibility mapping

A learning-based posterior distribution estimation method, Probabilistic Dipole Inversion (PDI), is proposed to solve the quantitative susceptibility mapping (QSM) inverse problem in MRI with uncertainty estimation. In PDI, a deep convolutional neural network (CNN) is used to represent the multivariate Gaussian distribution as the approximate posterior distribution of susceptibility given the input measured field. Such CNN is first trained on healthy subjects via posterior density estimation, where the training dataset contains samples from the true posterior distribution. Domain adaptations are then deployed on patient datasets with new pathologies not included in pre-training, where PDI updates the pre-trained CNN’s weights in an unsupervised fashion by minimizing the Kullback-Leibler divergence between the approximate posterior distribution represented by CNN and the true posterior distribution from the likelihood distribution of a known physical model and pre-defined prior distribution. Based on our experiments, PDI provides additional uncertainty estimation compared to the conventional MAP approach, while addressing the potential issue of the pre-trained CNN when test data deviates from training. Our code is available at https://github.com/Jinwei1209/Bayesian_QSM.

A Remote Sensing Image Registration Algorithm Based on Multiple Constraints and a Variational Bayesian Framework

ABSTRACT Aiming at complicated remote sensing image registration problem, we proposed a multi-constraint registration algorithm based on variational inference. Gaussian mixture model is used to construct the entire Bayesian network, which facilitates the estimation of heteroscedasticity noise in mapping function regression, and distribution of outliers in scene point density estimation. We use a variational parameter proxy to approximate complex posterior distribution. In addition, while maintaining the stability of global structure, similarity of local structure is restricted. In order to limit the cost of image conversion, we added constraints on spatial curvature for image. Accuracy and smoothness of the spatial transformation can be maintained by minimizing the curvature in horizontal and vertical directions. Experimental results show that compared with other point set matching algorithms, this method can achieve better results in robustness and matching accuracy.

Joint Bayesian Inference for Near‐Surface Explosion Yield and Height‐of‐Burst

AbstractForensic capabilities to understand chemical and nuclear explosions are greatly aided by an accurate estimate of explosive yield with uncertainty. The relationship between explosive size and geophysical observations of seismic, acoustic, and optical waves can be exploited to provide an estimate of yield. Any near‐surface yield estimate is complicated by the surface interaction, so an estimate for the explosion height‐of‐burst is necessarily included in the relationship. The relationship dictates a trade‐off between estimates of yield and height‐of‐burst. Fortunately, the surface interaction for each type of observation is different, which breaks the trade‐off, and the inclusion of height‐of‐burst with multiple data types improves yield estimation. We define simple parametric forward models to relate seismoäcoustoöptic observations from a data set of known explosive yields and height‐of‐bursts. The parameters of the models and a prediction for the yield and height‐of‐burst of a new event can then be estimated given new observations via Bayesian inference. We report posterior distribution estimates of the parametric models using a Markov chain Monte Carlo sampling technique. These models are then used to predict the yield and height‐of‐burst of SUGAR, a historical near‐surface nuclear explosion, using its reported historical observations. The reported yield of 1.2 ktonne Trinitrotoluene (TNT)‐equivalent (Department of Energy, 2015) is within the estimated posterior. Yield uncertainty can be estimated from the spread of the posterior, which is between 0.9 and 2.1 ktonne TNT‐equivalent. The posterior for height‐of‐burst has a wider range between 10 m below and 8 m above ground that includes the true height‐of‐burst of 1 m.