Posterior Marginals Research Articles

Graphics processing unit–accelerated computation of the Markov random fields and loopy belief propagation algorithms for hyperspectral image classification

We propose a commodity graphics processing units (GPUs)–based massively parallel efficient computation for spectral-spatial classification of hyperspectral images. The spectral-spatial classification framework is based on the marginal probability distribution which uses all of the information in the hyperspectral data. In this framework, first, the posterior class probability is modeled with discriminative random field in which the association potential is linked with a multinomial logistic regression (MLR) classifier and the interaction potential modeling the spatial information is linked to a Markov random field multilevel logistic (MLL) prior. Second, the maximizers of the posterior marginals are computed via the loopy belief propagation (LBP) method. In addition, the regressors of the multinominal logistic regression classifier are inferred by the logistic regression via variable splitting and augmented Lagrangian (LORSAL) algorithm. Although the spectral-spatial classification framework exhibited state-of-the-art accuracy performance with regard to similar approaches, its computational complexity is very high. We take advantage of the massively parallel computing capability of NVIDIA Tesla C2075 with the compute unified device architecture including a set of GPU-accelerated linear algebra libraries (CULA) to dramatically improve the computation speed of this hyperspectral image classification framework. The shared memory and the asynchronous transfer techniques are also used for further computationally efficient optimization. Real hyperspectral data sets collected by the National Aeronautics and Space Administration’s airborne visible infrared imaging spectrometer and the reflective optics system imaging spectrometer system are used for effectiveness evaluation. The results show that we achieved a speedup of 92-fold on LORSAL, 69-fold on MLR, 127-fold on MLL, 160-fold on LBP, and 73-fold on the whole spectral-spatial classification framework as compared with its single-core central processing unit counterpart, respectively.

Joint inference of misaligned irregular time series with application to Greenland ice core data

Abstract. Ice cores provide insight into the past climate over many millennia. Due to ice compaction, the raw data for any single core are irregular in time. Multiple cores have different irregularities; and when considered together, they are misaligned in time. After processing, such data are made available to researchers as regular time series: a data product. Typically, these cores are independently processed. This paper considers a fast Bayesian method for the joint processing of multiple irregular series. This is shown to be more efficient than the independent alternative. Furthermore, our explicit framework permits a reliable modelling of the impact of the multiple sources of uncertainty. The methodology is illustrated with the analysis of a pair of ice cores. Our data products, in the form of posterior marginals or joint distributions on an arbitrary temporal grid, are finite Gaussian mixtures. We can also produce process histories to study non-linear functionals of interest. More generally, the concept of joint analysis via hierarchical Gaussian process models can be widely extended, as the models used can be viewed within the larger context of continuous space–time processes.

3D EM/MPM Image Segmentation Using an FPGA Embedded Design Implementation

This paper presents a Field Programmable Gate Array (FPGA) based embedded system which is used to achieve high speed segmentation of 3D images. Segmentation is performed using Expectation-Maximization (EM) with Maximization of Posterior Marginals (MPM) Bayesian algorithm. This algorithm segments the 3D image using neighboring pixels based on a Markov Random Field (MRF) model. In this system, the embedded processor controls a custom circuit which performs the MPM and portions of the EM algorithm. The embedded processor completes the EM algorithm and also controls image data transmission between host computer and on-board memory. The whole system has been implemented on Xilinx Virtex 6 FPGA and achieved over 100 times processing improvement compared to standard desktop computer. Three new techniques were the key to achieve this speed: Pipelined computational cores, sixteen parallel data paths and a novel memory interface for maximizing the external memory bandwidth.

Spatial Data Analysis withR-INLAwith Some Extensions

The integrated nested Laplace approximation (INLA) provides an interesting way of approximating the posterior marginals of a wide range of Bayesian hierarchical models. This approximation is based on conducting a Laplace approximation of certain functions and numerical integration is extensively used to integrate some of the models parameters out. The R-INLA package offers an interface to INLA, providing a suitable framework for data analysis. Although the INLA methodology can deal with a large number of models, only the most relevant have been implemented within R-INLA. However, many other important models are not available for R-INLA yet. In this paper we show how to fit a number of spatial models with R-INLA, including its interaction with other R packages for data analysis. Secondly, we describe a novel method to extend the number of latent models available for the model parameters. Our approach is based on conditioning on one or several model parameters and fit these conditioned models with R-INLA. Then these models are combined using Bayesian model averaging to provide the final approximations to the posterior marginals of the model. Finally, we show some examples of the application of this technique in spatial statistics. It is worth noting that our approach can be extended to a number of other fields, and not only spatial statistics.

Bayesian inference of a lake water quality model by emulating its posterior density

AbstractWe use a Gaussian stochastic process emulator to interpolate the posterior probability density of a computationally demanding application of the biogeochemical‐ecological lake model BELAMO to accelerate statistical inference of deterministic model and error model parameters. The deterministic model consists of a mechanistic description of key processes influencing the mass balance of nutrients, dissolved oxygen, organic particles, and phytoplankton and zooplankton in the lake. This model is complemented by a Gaussian stochastic process to describe the remaining model bias and by Normal, independent observation errors. A small subsample of the Markov chain representing the posterior of the model parameters is propagated through the full model to get model predictions and uncertainty estimates. We expect this approximation to be more accurate at only slightly higher computational costs compared to using a Normal approximation to the posterior probability density and linear error propagation to the results as we did in an earlier paper. The performance of the two techniques is compared for a didactical example as well as for the lake model. As expected, for the didactical example, the use of the emulator led to posterior marginals of the model parameters that are closer to those calculated by Markov chain simulation using the full model than those based on the Normal approximation. For the lake model, the new technique proved applicable without an excessive increase in computational requirements, but we faced challenges in the choice of the design data set for emulator calibration. As the posterior is a scalar function of the parameters, the suggested technique is an alternative to the emulation of a potentially more complex, structured output of the simulation model that allows for the use of a less case‐specific emulator. This is at the cost that still the full model has to be used for prediction (which can be done with a smaller, approximately independent subsample of the Markov chain).

Comparing three-dimensional Bayesian segmentations for images with low signal-to-noise ratio (SNR<1) and strong attenuation

This paper examines three Bayesian statistical segmentation techniques with an innovative attenuation compensation on synthetic data and breast ultrasound medical images. All use expectation maximization for estimating the Gaussian model parameters and segment the data using a three-dimensional (3-D) Markov random field pixel neighborhood. This paper compares three Bayesian segmentation techniques: maximum a posteriori simulated annealing (MAP-SA), MAP iterated conditional modes (MAP-ICM), and maximization of posterior marginals (MPM). We conclude that because of the high speckle noise and adverse attenuation challenges of breast ultrasound, the MPM algorithm has the best performance. This is due to better localized segmentation than the other MAP techniques. We present results first with synthetic images then with breast ultrasound. Our new contributions for a 3-D breast ultrasound produce improved results using a model of the noise, in which the Gaussian mean is proportional to the image attenuation with depth, combined with a new prior probability model.

Spatial Latent Gaussian Models: Application to House Prices Data in Tehran City

Latent Gaussian models are flexible models that are applied in several statistical applications. When posterior marginals or full conditional distributions in hierarchical Bayesian inference from these models are not available in closed form, Markov chain Monte Carlo methods are implemented. The component dependence of the latent field usually causes increase in computational time and divergence of algorithms. In this paper, an integrated nested Laplace approximation is used to solve these problems, in which the Laplace approximation and the numerical integration methods are combined in an efficient way so that hard simulations are replaced by fast computation and accurate approximation. Finally the relationship between house price data, floor size, age, number of rooms, building frame, type of proprietorship and facilities such as electricity, landline, water, gas, central heating and cooling system, kitchen goods, bath and toilet are modeled by using spatial latent Gaussian models. The fitted model can be used for predicting the house price in Tehran city.

Approximate Bayesian inference for spatial econometrics models

In this paper we explore the use of the Integrated Laplace Approximation (INLA) for Bayesian inference in some widely used models in Spatial Econometrics. Bayesian inference often relies on computationally intensive simulation methods, such as Markov Chain Monte Carlo. When only marginal inference is needed, INLA provides a fast and accurate estimate of the posterior marginals of the parameters in the model.Furthermore, we have compared the results provided by these models to those obtained with a more general class of Generalised Linear Models with random effects. In these models, spatial autocorrelation is modelled by means of correlated Gaussian random effects.We also discuss a procedure to extend the class of models that the R-INLA software can fit. This approach is based on conditioning on one or more parameters so that the resulting models can be fitted with R-INLA across sets of values of the fixed parameters. The posterior marginals of these parameters of interest are then obtained by combining the marginal likelihoods (which are conditioned on the values of the parameters fixed) of the fitted models and a prior on these parameters. This approach can also be used to fit even more general models.Finally, we discuss the use of all these models on two datasets based on median housing prices for census tracts in Boston and the probability of business re-opening in New Orleans in the aftermath of hurricane Katrina.

Three-dimensional characterization of the permeability of W–Cu composites using a new “TriBeam” technique

Large three-dimensional microstructural datasets have been gathered for two W–Cu composites of 10 and 15wt.% Cu using the TriBeam system via in situ femtosecond laser sectioning in a scanning electron microscope. Laser ablation was performed on W–Cu samples along a 90° edge, milling parallel with the imaging surface. Secondary electron images for 1000 two-dimensional slices were segmented into binary images representing Cu and W components using EM/MPM (expectation–maximization/maximization of the posterior marginals) image-processing algorithms. A statistically random volume sampling approach has been employed to evaluate the microstructural and property volume element sizes necessary for the assessment of volume fraction, surface-area to volume ratio and permeability, respectively. This approach also characterizes the mean values and variability in microstructure and properties for volume elements ranging from 10μm to 160μm on edge. The converged values of the volume fractions of Cu closely match experimental values measured by the Archimedes technique.

Bayesian scanning of spatial disease rates with integrated nested Laplace approximation (INLA)

Among the many tools suited to detect local clusters in group-level data, Kulldorff–Nagarwalla’s spatial scan statistic gained wide popularity (Kulldorff and Nagarwalla in Stat Med 14(8):799–810, 1995). The underlying assumptions needed for making statistical inference feasible are quite strong, as counts in spatial units are assumed to be independent Poisson distributed random variables. Unfortunately, outcomes in spatial units are often not independent of each other, and risk estimates of areas that are close to each other will tend to be positively correlated as they share a number of spatially varying characteristics. We therefore introduce a Bayesian model-based algorithm for cluster detection in the presence of spatially autocorrelated relative risks. Our approach has been made possible by the recent development of new numerical methods based on integrated nested Laplace approximation, by which we can directly compute very accurate approximations of posterior marginals within short computational time (Rue et al. in JRSS B 71(2):319–392, 2009). Simulated data and a case study show that the performance of our method is at least comparable to that of Kulldorff–Nagarwalla’s statistic.

Replica Analysis of Multiuser Detection for Code Division Multiple Access withM-Ary Phase-Shift Keying

We analyze the performance of the maximizer of the posterior marginals (MPM) detector for code division multiple access (CDMA) multiuser detection with \(M\)-ary phase shift keying (\(M\)-ary PSK) in the large system limit by the replica method. The obtained theory agrees with computer simulation reasonably well. We also derive the theory in the case of the large \(M\) limit and discuss the dependence of the properties of \(M\)-ary PSK CDMA communication on \(M\). We show that the waterfall phenomenon occurs for both the finite and infinite values of \(M\). We also show that a value of \(M\) for which the decoded phase information on the original user symbol becomes minimum exists. Furthermore, we discuss the relationship between the theory based on the replica method and that based on self-consistent signal-to-noise analysis (SCSNA).

Bayesian computing with INLA: New features

The INLA approach for approximate Bayesian inference for latent Gaussian models has been shown to give fast and accurate estimates of posterior marginals and also to be a valuable tool in practice via the R-package R-INLA. New developments in the R-INLA are formalized and it is shown how these features greatly extend the scope of models that can be analyzed by this interface. The current default method in R-INLA to approximate the posterior marginals of the hyperparameters using only a modest number of evaluations of the joint posterior distribution of the hyperparameters, without any need for numerical integration, is discussed.

Bayesian Nonparametric Regression and Density Estimation Using Integrated Nested Laplace Approximations

Integrated nested Laplace approximations (INLA) are a recently proposed approximate Bayesian approach to fit structured additive regression models with latent Gaussian field. INLA method, as an alternative to Markov chain Monte Carlo techniques, provides accurate approximations to estimate posterior marginals and avoid time-consuming sampling. We show here that two classical nonparametric smoothing problems, nonparametric regression and density estimation, can be achieved using INLA. Simulated examples and R functions are demonstrated to illustrate the use of the methods. Some discussions on potential applications of INLA are made in the paper.

MPM SAR Image Segmentation Using Feature Extraction and Context Model

A new synthetic aperture radar (SAR) image segmentation method based on a maximization of posterior marginals (MPM) algorithm with feature extraction and context model is proposed in this letter. First, Gabor wavelet and texture descriptor are used to extract features, which enhance intraclass similarities and interclass differences. Second, the number of regions within the same class is reduced in order to improve the reliability of the regional statistical characteristics. Finally, the MPM of each region combined with the context model is calculated by considering both the intralayer correlation and interlayer correlation. The experimental results show that the proposed method is efficient and effective for SAR image segmentation.

Class-specific weighting for Markov random field estimation: Application to medical image segmentation

Many estimation tasks require Bayesian classifiers capable of adjusting their performance (e.g. sensitivity/specificity). In situations where the optimal classification decision can be identified by an exhaustive search over all possible classes, means for adjusting classifier performance, such as probability thresholding or weighting the a posteriori probabilities, are well established. Unfortunately, analogous methods compatible with Markov random fields (i.e. large collections of dependent random variables) are noticeably absent from the literature. Consequently, most Markov random field (MRF) based classification systems typically restrict their performance to a single, static operating point (i.e. a paired sensitivity/specificity). To address this deficiency, we previously introduced an extension of maximum posterior marginals (MPM) estimation that allows certain classes to be weighted more heavily than others, thus providing a means for varying classifier performance. However, this extension is not appropriate for the more popular maximum a posteriori (MAP) estimation. Thus, a strategy for varying the performance of MAP estimators is still needed. Such a strategy is essential for several reasons: (1) the MAP cost function may be more appropriate in certain classification tasks than the MPM cost function, (2) the literature provides a surfeit of MAP estimation implementations, several of which are considerably faster than the typical Markov Chain Monte Carlo methods used for MPM, and (3) MAP estimation is used far more often than MPM. Consequently, in this paper we introduce multiplicative weighted MAP (MWMAP) estimation-achieved via the incorporation of multiplicative weights into the MAP cost function-which allows certain classes to be preferred over others. This creates a natural bias for specific classes, and consequently a means for adjusting classifier performance. Similarly, we show how this multiplicative weighting strategy can be applied to the MPM cost function (in place of the strategy we presented previously), yielding multiplicative weighted MPM (MWMPM) estimation. Furthermore, we describe how MWMAP and MWMPM can be implemented using adaptations of current estimation strategies such as iterated conditional modes and MPM Monte Carlo. To illustrate these implementations, we first integrate them into two separate MRF-based classification systems for detecting carcinoma of the prostate (CaP) on (1) digitized histological sections from radical prostatectomies and (2) T2-weighted 4 Tesla ex vivo prostate MRI. To highlight the extensibility of MWMAP and MWMPM to estimation tasks involving more than two classes, we also incorporate these estimation criteria into a MRF-based classifier used to segment synthetic brain MR images. In the context of these tasks, we show how our novel estimation criteria can be used to arbitrarily adjust the sensitivities of these systems, yielding receiver operator characteristic curves (and surfaces).

Direct fitting of dynamic models using integrated nested Laplace approximations — INLA

Inference in state-space models usually relies on recursive forms for filtering and smoothing of the state vectors regarding the temporal structure of the observations, an assumption that is, from our view point, unnecessary if the dataset is fixed, that is, completely available before analysis. In this paper, we propose a computational framework to perform approximate full Bayesian inference in linear and generalized dynamic linear models based on the Integrated Nested Laplace Approximation (INLA) approach. The proposed framework directly approximates the posterior marginals of interest disregarding the assumption of recursive updating/estimation of the states and hyperparameters in the case of fixed datasets and, therefore, enable us to do fully Bayesian analysis of complex state-space models more easily and in a short computational time. The proposed framework overcomes some limitations of current tools in the dynamic modeling literature and is vastly illustrated with a series of simulated as well as well known real-life examples from the literature, including realistically complex models with correlated error structures and models with more than one state vector, being mutually dependent on each other. R code is available online for all the examples presented.

Bayesian multiscale analysis of images modeled as Gaussian Markov random fields

A Bayesian multiscale technique for the detection of statistically significant features in noisy images is proposed. The prior is defined as a stationary intrinsic Gaussian Markov random field on a toroidal graph, which enables efficient computation of the relevant posterior marginals. Hence the method is applicable to large images produced by modern digital cameras. The technique is demonstrated in two examples from medical imaging.

Weighted Maximum Posterior Marginals for Random Fields Using an Ensemble of Conditional Densities From Multiple Markov Chain Monte Carlo Simulations

The ability of classification systems to adjust their performance (sensitivity/specificity) is essential for tasks in which certain errors are more significant than others. For example, mislabeling cancerous lesions as benign is typically more detrimental than mislabeling benign lesions as cancerous. Unfortunately, methods for modifying the performance of Markov random field (MRF) based classifiers are noticeably absent from the literature, and thus most such systems restrict their performance to a single, static operating point (a paired sensitivity/specificity). To address this deficiency we present weighted maximum posterior marginals (WMPM) estimation, an extension of maximum posterior marginals (MPM) estimation. Whereas the MPM cost function penalizes each error equally, the WMPM cost function allows misclassifications associated with certain classes to be weighted more heavily than others. This creates a preference for specific classes, and consequently a means for adjusting classifier performance. Realizing WMPM estimation (like MPM estimation) requires estimates of the posterior marginal distributions. The most prevalent means for estimating these--proposed by Marroquin--utilizes a Markov chain Monte Carlo (MCMC) method. Though Marroquin's method (M-MCMC) yields estimates that are sufficiently accurate for MPM estimation, they are inadequate for WMPM. To more accurately estimate the posterior marginals we present an equally simple, but more effective extension of the MCMC method (E-MCMC). Assuming an identical number of iterations, E-MCMC as compared to M-MCMC yields estimates with higher fidelity, thereby 1) allowing a far greater number and diversity of operating points and 2) improving overall classifier performance. To illustrate the utility of WMPM and compare the efficacies of M-MCMC and E-MCMC, we integrate them into our MRF-based classification system for detecting cancerous glands in (whole-mount or quarter) histological sections of the prostate.

Unsupervised 4D myocardium segmentation with a Markov Random Field based deformable model

A stochastic deformable model is proposed for the segmentation of the myocardium in Magnetic Resonance Imaging. The segmentation is posed as a probabilistic optimization problem in which the optimal time-dependent surface is obtained for the myocardium of the heart in a discrete space of locations built upon simple geometric assumptions. For this purpose, first, the left ventricle is detected by a set of image analysis tools gathered from the literature. Then, the segmentation solution is obtained by the Maximization of the Posterior Marginals for the myocardium location in a Markov Random Field framework which optimally integrates temporal-spatial smoothness with intensity and gradient related features in an unsupervised way by the Maximum Likelihood estimation of the parameters of the field. This scheme provides a flexible and robust segmentation method which has been able to generate results comparable to manually segmented images for some derived cardiac function parameters in a set of 43 patients affected in different degrees by an Acute Myocardial Infarction.

Approximate Marginals in Latent Gaussian Models

We consider the problem of improving the Gaussian approximate posterior marginals computed by expectation propagation and the Laplace method in latent Gaussian models and propose methods that are s...