Gibbs Sampling Research Articles

Mixing times and hitting times for general Markov processes

The hitting and mixing times are two fundamental quantities associated with Markov chains. In Peres and Sousi [PS15] and Oliveira [Oli12], the authors show that the mixing times and “worst-case” hitting times of reversible Markov chains on finite state spaces are equal up to some universal multiplicative constant. We use tools from nonstandard analysis to extend this result to reversible Markov chains on general state spaces that satisfy the strong Feller property. Finally, we show that this asymptotic equivalence can be used to find bounds on the mixing times of a large class of Markov chains used in MCMC, such as typical Gibbs samplers and Metropolis–Hastings chains, even though they usually do not satisfy the strong Feller property.

Estimated genetic parameters for mature cow live weights during lactation in a multi-breed dairy cattle population

ABSTRACT Live weight (LWT) is an important trait for dairy production systems as it influences both maintenance feed requirements and beef revenue from culled cows and from offspring surplus to replacements. The breeding goal for the New Zealand dairy industry includes LWT. This study estimated genetic parameters for LWT during lactation from some animals in the New Zealand multi-breed dairy cattle population. A total of 31,922 LWT records from 11,222 lactating dairy cattle born from 1988 to 2015 were used for the analysis. Estimation of (co)variance components used Gibbs sampling from pedigree-based models. The posterior mean heritability estimates using scale weights were consistently high across six lactations and ranged from 0.57 to 0.64. There were high genetic correlations between the scale weights measured during different lactations (>0.95). We further estimated the posterior mean heritability for subjective liveweight scores recorded from first lactation using trained inspectors and the estimate was 0.36, lower than those heritability estimates from scaled weight. Further, the genetic correlations between subjective liveweight score and scale weights at various lactations ranged from 0.91-0.92. It is recommended that only scale weights be used for national genetic evaluation.

Sparse Bayesian inference with regularized Gaussian distributions *

Regularization is a common tool in variational inverse problems to impose assumptions on the parameters of the problem. One such assumption is sparsity, which is commonly promoted using lasso and total variation-like regularization. Although the solutions to many such regularized inverse problems can be considered as points of maximum probability of well-chosen posterior distributions, samples from these distributions are generally not sparse. In this paper, we present a framework for implicitly defining a probability distribution that combines the effects of sparsity imposing regularization with Gaussian distributions. Unlike continuous distributions, these implicit distributions can assign positive probability to sparse vectors. We study these regularized distributions for various regularization functions including total variation regularization and piecewise linear convex functions. We apply the developed theory to uncertainty quantification for Bayesian linear inverse problems and derive a Gibbs sampler for a Bayesian hierarchical model. To illustrate the difference between our sparsity-inducing framework and continuous distributions, we apply our framework to small-scale deblurring and computed tomography examples.

The reciprocal elastic net

A new penalized likelihood method (reciprocal elastic net) is put forward for regularization and variable selection. Our proposal is based on a new class of reciprocal penalty functions, combining the strengths of the reciprocal LASSO regularization and ridge regression. We formulate the reciprocal elastic net problem as an equivalent reciprocal LASSO problem on augmented data, facilitating a direct utilization of the reciprocal LASSO algorithm to generate the entire reciprocal elastic net solution path. We further present the reciprocal adaptive elastic net, fusing the merits of ridge regression with the adaptively weighted reciprocal LASSO regularization. These methods, illustrated through simulated examples and real data analysis, demonstrate satisfactory performance in various diversified scenarios compared to published methods. Finally, we propose Bayesian methods to solve the reciprocal elastic net and reciprocal adaptive elastic net models using Gibbs samplers.

Hyper Markov law in undirected graphical models with its applications

By exploring the prime decomposition of undirected graphs, this work investigates the hyper Markov property within the framework of arbitrary undirected graph, which can be seen as the generalization of that for decomposable graphical models proposed by Dawid and Laurizten. The proposed hyper Markov properties of this article can be used to characterize the conditional independence of a distribution or a statistical quantity and helpful to simplify the likelihood ratio functions for statistical test for two different graphs obtained by removing or adding one edge. As an application of these properties, the G-Wishart law is introduced as a prior law for graphical Gaussian models for Bayesian posterior updating, and a hypothesis test for precision matrix is designed to determine the model structures. Our simulation experiments are implemented by using the Gibbs Sampler algorithm and the results show that it performs better in convergence speed.

Reliability assessment of civil structures with incomplete probability distribution information

The probability distribution of random variables is usually incomplete owing to a limited amount of data, making it a challenging problem for structural reliability assessments. This paper presents a novel reliability analysis method for civil structures with incomplete data. The probability distributions of the random variables were estimated using an improved Bayesian inference method. In the proposed inference method, the frequencies and mode shapes were selected as the observation data. The likelihood function is replaced by homogeneous chaos, and the computational efficiency of Bayesian inference is improved. The affine invariant ensemble algorithm was used to generate samplers for the posterior distribution. After estimating the probability distribution of the incomplete random variables, a reliability analysis was performed, and the failure probability of the structure was estimated. Numerical studies on a two-span, two-story reinforced concrete building and a five-span bridge are presented. Experimental studies were also performed on an eight-story steel frame structure. The results show that the probability distributions of random variables can be significantly updated using the proposed technique and modal properties. The proposed technique was more efficient than the Gibbs sampling method. The failure probabilities of the structures were estimated using the estimated probability distributions.

A Gibbs-INLA algorithm for multidimensional graded response model analysis.

In this paper, we propose a novel Gibbs-INLA algorithm for the Bayesian inference of graded response models with ordinal response based on multidimensional item response theory. With the combination of the Gibbs sampling and the integrated nested Laplace approximation (INLA), the new framework avoids the cumbersome tuning which is inevitable in classical Markov chain Monte Carlo (MCMC) algorithm, and has low computing memory, high computational efficiency with much fewer iterations, and still achieve higher estimation accuracy. Therefore, it has the ability to handle large amount of multidimensional response data with different item responses. Simulation studies are conducted to compare with the Metroplis-Hastings Robbins-Monro (MH-RM) algorithm and an application to the study of the IPIP-NEO personality inventory data is given to assess the performance of the new algorithm. Extensions of the proposed algorithm for application on more complicated models and different data types are also discussed.

Truncated two‐parameter Poisson–Dirichlet approximation for Pitman–Yor process hierarchical models

AbstractIn this paper, we construct an approximation to the Pitman–Yor process by truncating its two‐parameter Poisson–Dirichlet representation. The truncation is based on a decreasing sequence of random weights, thus having a lower approximation error compared to the popular truncated stick‐breaking process. We develop an exact simulation algorithm to sample from the approximation process and provide an alternative MCMC algorithm for the parameter regime where the exact simulation algorithm becomes slow. The effectiveness of the simulation algorithms is demonstrated by the estimation of the functionals of a Pitman–Yor process. Then we adapt the approximation process into a Pitman–Yor process mixture model and devise a blocked Gibbs sampler for posterior inference.

Root and community inference on the latent growth process of a network

Abstract Many statistical models for networks overlook the fact that most real-world networks are formed through a growth process. To address this, we introduce the Preferential Attachment Plus Erdős–Rényi model, where we let a random network G be the union of a preferential attachment (PA) tree T and additional Erdős–Rényi (ER) random edges. The PA tree captures the underlying growth process of a network where vertices/edges are added sequentially, while the ER component can be regarded as noise. Given only one snapshot of the final network G, we study the problem of constructing confidence sets for the root node of the unobserved growth process; the root node can be patient zero in an infection network or the source of fake news in a social network. We propose inference algorithms based on Gibbs sampling that scales to networks with millions of nodes and provide theoretical analysis showing that the size of the confidence set is small if the noise level of the ER edges is not too large. We also propose variations of the model in which multiple growth processes occur simultaneously, reflecting the growth of multiple communities; we use these models to provide a new approach to community detection.

A composite Bayesian approach for quantile curve fitting with non-crossing constraints

To fit a set of quantile curves, Bayesian simultaneous quantile curve fitting methods face some challenges in properly specifying a feasible formulation and efficiently accommodating the non-crossing constraints. In this article, we propose a new minimization problem and develop its corresponding Bayesian analysis. The new minimization problem imposes two penalties to control not only the smoothness of fitted quantile curves but also the differences between quantile curves. This enables a direct inference on differences of quantile curves and facilitates improved information sharing among quantiles. After adopting B-spline approximation for the positive smoothing functions in the minimization problem, we specify the pseudo composite asymmetric Laplace likelihood and derive its priors. The computation algorithm, including partially collapsed Gibbs sampling for model parameters and Monte Carlo Expectation-Maximization algorithm for penalty parameters, are provided to carry out the proposed approach. The extensive simulation studies show that, compared with other candidate methods, the proposed approach yields more robust estimation. More advantages of the proposed approach are observed for the extreme quantiles, heavy-tailed random errors, and inference on the differences of quantiles. We also demonstrate the relative performances of the proposed approach and other competing methods through two real data analyses.

Maximum Likelihood Estimation of Hierarchical Linear Models from Incomplete Data: Random Coefficients, Statistical Interactions, and Measurement Error

We consider two-level models where a continuous response R and continuous covariates C are assumed missing at random. Inferences based on maximum likelihood or Bayes are routinely made by estimating their joint normal distribution from observed data and . However, if the model for R given C includes random coefficients, interactions, or polynomial terms, their joint distribution will be nonstandard. We propose a family of unique factorizations involving selected “provisionally known random effects” u such that is normally distributed and u is a low-dimensional normal random vector; we approximate via adaptive Gauss-Hermite quadrature. For polynomial models, the approximation is exact but, in any case, can be made as accurate as required given sufficient computation time. The model incorporates random effects as explanatory variables, reducing bias due to measurement error. By construction, our factorizations solve problems of compatibility among fully conditional distributions that have arisen in Bayesian imputation based on the Gibbs Sampler. We spell out general rules for selecting u, and show that our factorizations can support fully compatible Bayesian methods of imputation using the Gibbs Sampler. Supplementary materials for this article are available online.

Estimation of genetic parameters for the tick and hemoparasite burden in Angus cattle.

The study was conducted with the objective of estimating genetic and phenotypic parameters for tick (CRM) and Babesia bigemina (IBBi), Babesia bovis (IBBo), and Anaplasma marginale (IAM) burden in Angus female breed in Brazil. The sample group was composed of Angus females raised in herds located in a region of endemic instability for cattle tick fever in the state of Rio Grande Sul (RS), Brazil. The variance components were estimated using Bayesian inference and Gibbs sampling algorithm, considering a multi-trait animal model. Heritability estimates showed values of low magnitude, ranging from 0.03 (IBBo) to 0.16 (CRM), while repeatability estimates ranged between 0.07 (IBBo) and 0.21 (CRM). Regarding the genetic correlation estimates, the values showed low (-0.01 for IBBo × IAM) to moderate (0.55 between IBBi × IAM) magnitudes. The results indicate that it is possible to use tick count and hemoparasite infection levels as selection criteria, with small genetic gains.

A Bayesian framework for pathway-guided identification of cancer subgroups by integrating multiple types of genomic data.

In recent years, comprehensive cancer genomics platforms, such as The Cancer Genome Atlas (TCGA), provide access to an enormous amount of high throughput genomic datasets for each patient, including gene expression, DNA copy number alterations, DNA methylation, and somatic mutation. While the integration of these multi-omics datasets has the potential to provide novel insights that can lead to personalized medicine, most existing approaches only focus on gene-level analysis and lack the ability to facilitate biological findings at the pathway-level. In this article, we propose Bayes-InGRiD (Bayesian Integrative Genomics Robust iDentification of cancer subgroups), a novel pathway-guided Bayesian sparse latent factor model for the simultaneous identification of cancer patient subgroups (clustering) and key molecular features (variable selection) within a unified framework, based on the joint analysis of continuous, binary, and count data. By utilizing pathway (gene set) information, Bayes-InGRiD does not only enhance the accuracy and robustness of cancer patient subgroup and key molecular feature identification, but also promotes biological understanding and interpretation. Finally, to facilitate an efficient posterior sampling, an alternative Gibbs sampler for logistic and negative binomial models is proposed using Pólya-Gamma mixtures of normal to represent latent variables for binary and count data, which yields a conditionally Gaussian representation of the posterior. The R package "INGRID" implementing the proposed approach is currently available in our research group GitHub webpage (https://dongjunchung.github.io/INGRID/).

A fast asynchronous Markov chain Monte Carlo sampler for sparse Bayesian inference

Abstract We propose a very fast approximate Markov chain Monte Carlo sampling framework that is applicable to a large class of sparse Bayesian inference problems. The computational cost per iteration in several regression models is of order O(n(s+J)), where n is the sample size, s is the underlying sparsity of the model, and J is the size of a randomly selected subset of regressors. This cost can be further reduced by data sub-sampling when stochastic gradient Langevin dynamics are employed. The algorithm is an extension of the asynchronous Gibbs sampler of Johnson et al. [(2013). Analyzing Hogwild parallel Gaussian Gibbs sampling. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS’13) (Vol. 2, pp. 2715–2723)], but can be viewed from a statistical perspective as a form of Bayesian iterated sure independent screening [Fan, J., Samworth, R., & Wu, Y. (2009). Ultrahigh dimensional feature selection: Beyond the linear model. Journal of Machine Learning Research, 10, 2013–2038]. We show that in high-dimensional linear regression problems, the Markov chain generated by the proposed algorithm admits an invariant distribution that recovers correctly the main signal with high probability under some statistical assumptions. Furthermore, we show that its mixing time is at most linear in the number of regressors. We illustrate the algorithm with several models.

Locally adaptive Bayesian isotonic regression using half shrinkage priors

AbstractIsotonic regression or monotone function estimation is a problem of estimating function values under monotonicity constraints, which appears naturally in many scientific fields. This paper proposes a new Bayesian method with global–local shrinkage priors for estimating monotone function values. Specifically, we introduce half shrinkage priors for positive valued random variables and assign them for the first‐order differences of function values. We also develop fast and simple Gibbs sampling algorithms for full posterior analysis. By incorporating advanced shrinkage priors, the proposed method is adaptive to local abrupt changes or jumps in target functions. We show this adaptive property theoretically by proving that the posterior mean estimators are robust to large differences and that asymptotic risk for unchanged points can be improved. Finally, we demonstrate the proposed methods through simulations and applications to a real data set.

Odd log-logistic generalised Lindley distribution with properties and applications

In this paper, a new three-parameter lifetime model, called the odd log-logistic generalised Lindley distribution, is introduced. Some structural properties of the new distribution including ordinary and incomplete moments, quantile and generating functions and order statistics are obtained. The new density function can be expressed as a linear mixture of exponetiated Lindley densities. Different methods are discussed to estimate the model parameters and a simulation study is carried out to show the performance of the new distribution. The importance and flexibility of the new model are also illustrated empirically by means of two real data sets. Finally, Bayesian analysis and Gibbs sampling are performed based on the two real data sets.

Exploring the capabilities of Gibbs sampling in pulsar timing arrays

Exploring the capabilities of Gibbs sampling in pulsar timing arrays

Multi-View Scholar Clustering With Dynamic Interest Tracking

Scholar clustering has garnered increasing attention due to the explosive growth of scholar data. Although researchers have proposed many algorithms to cluster scholars, they typically focus on clustering scholars from the intrinsic view (scholars' contents). These algorithms may lead to inaccurate and biased clustering results because they ignore the extrinsic view (scholar's specialty) and the changeability of scholars' interest in each view. In this paper, we propose a multi-view scholar clustering topic model (MSCT), which integrates complementary information from both intrinsic and extrinsic views while considering dynamic scholar interests. Specifically, MSCT involves two novel schemes. The first one is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multi-view integration</i> , where MSCT collaboratively tracks scholars' time-varying topic distribution from two views: <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> intrinsic view</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">extrinsic view</i> . The former exploits the details of different academic degrees in the title and information in the abstract; the latter leverages the specialty of different categories in the corresponding research field and research discipline. The second one is <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">dynamic interest tracking</i> , which dynamically models each scholar's interest distribution in terms of the current scholar texts and previously estimated distribution through a newly designed collapsed Gibbs sampling algorithm. Experimental results demonstrate that MSCT can significantly outperform state-of-the-art algorithms.

Genetic evaluation of sperm abnormality traits in Holstein Friesian crossbred bulls using random regression models

Genetic parameters of sperm abnormality traits were evaluated in HF crossbred bulls using Bayesian univariate, bivariate and random regression models. Data from 161 bulls on 1,527 ejaculates collected from 2010 to 2020 at BAIF (Bharatiya Agro Industries Foundation) was utilized for this study. Four sperm abnormality traits viz. total abnormalities (A), head abnormality (HA), mid-piece abnormality (MA) and tail abnormality (TA) were included in the analysis. Gibbs sampling was done to implement a Bayesian framework. Means for A, HA, MA and TA were 10.77%, 2.99%. 2.61% and 5.10% respectively. Heritability and repeatability estimates obtained from univariateanalysis were below 0.1. Genetic correlations obtained for the total abnormalities with semen production traits like ejaculate volume, sperm concentration, initial sperm motility, post-thaw motility and the total number of spermatozoa were negative. Heritability and repeatability estimates were higher in the case of random regression models where the values tend to increase with the age of the bulls. Heritability ranged from 0.025 (3 years) to 0.139 (10 years) for total abnormality estimates. In general, the estimates of genetic parameters for abnormality were low and found to increase with age.

Rates of convergence for Gibbs sampling in the analysis of almost exchangeable data

Motivated by de Finetti’s representation theorem for almost exchangeable arrays, we want to sample p∈[0,1]d from a distribution with density proportional to exp(−A2∑i<jcij(pi−pj)2), where A is large and cij’s are non-negative weights. We analyze the rate of convergence of a coordinate Gibbs sampler used to simulate from these measures. We show that for every non-zero fixed matrix C=(cij), and large enough A, mixing happens in Θ(A2) steps in a suitable Wasserstein distance. The upper and lower bounds are explicit and depend on the matrix C through few relevant spectral parameters.