Improper Priors Research Articles

Recurrence and transience of a Markov chain on $\mathbb Z$ + and evaluation of prior distributions for a Poisson mean

AbstractEaton (1992) considered a general parametric statistical model paired with an improper prior distribution for the parameter and proved that if a certain Markov chain, constructed using the model and the prior, is recurrent, then the improper prior is strongly admissible, which (roughly speaking) means that the generalized Bayes estimators derived from the corresponding posterior distribution are admissible. Hobert and Robert (1999) proved that Eaton’s Markov chain is recurrent if and only if its so-called conjugate Markov chain is recurrent. The focus of this paper is a family of Markov chains that contains all of the conjugate chains that arise in the context of a Poisson model paired with an arbitrary improper prior for the mean parameter. Sufficient conditions for recurrence and transience are developed and these are used to establish new results concerning the strong admissibility of non-conjugate improper priors for the Poisson mean.

Empirical Bayes factors for common hypothesis tests.

Bayes factors for composite hypotheses have difficulty in encoding vague prior knowledge, as improper priors cannot be used and objective priors may be subjectively unreasonable. To address these issues I revisit the posterior Bayes factor, in which the posterior distribution from the data at hand is re-used in the Bayes factor for the same data. I argue that this is biased when calibrated against proper Bayes factors, but propose adjustments to allow interpretation on the same scale. In the important case of a regular normal model, the bias in log scale is half the number of parameters. The resulting empirical Bayes factor is closely related to the widely applicable information criterion. I develop test-based empirical Bayes factors for several standard tests and propose an extension to multiple testing closely related to the optimal discovery procedure. When only a P-value is available, an approximate empirical Bayes factor is 10p. I propose interpreting the strength of Bayes factors on a logarithmic scale with base 3.73, reflecting the sharpest distinction between weaker and stronger belief. This provides an objective framework for interpreting statistical evidence, and realises a Bayesian/frequentist compromise.

On admissible estimation of a mean vector when the scale is unknown

We consider admissibility of generalized Bayes estimators of the mean of a multivariate normal distribution when the scale is unknown under quadratic loss. The priors considered put the improper invariant prior on the scale while the prior on the mean has a hierarchical normal structure conditional on the scale. This conditional hierarchical prior is essentially that of Maruyama and Strawderman (2021, Biometrika) (MS21) which is indexed by a hyperparameter $a$. In that paper $a$ is chosen so this conditional prior is proper which corresponds to $a>-1$. This paper extends MS21 by considering improper conditional priors with $a$ in the closed interval $[-2, -1]$, and establishing admissibility for such $a$. The authors, in Maruyama and Strawderman (2017, JMVA), have earlier shown that such conditional priors with $a < -2$ lead to inadmissible estimators. This paper therefore completes the determination of admissibility/inadmissibility for this class of priors. It establishes the the boundary as $a = -2$, with admissibility holding for $a\geq -2$ and inadmissibility for $a < -2$. This boundary corresponds exactly to that in the known scale case for these conditional priors, and which follows from Brown (1971, AOMS). As a notable benefit of this enlargement of the class of admissible generalized Bayes estimators, we give admissible and minimax estimators in all dimensions greater than $2$ as opposed to MS21 which required the dimension to be greater than $4$. In one particularly interesting special case, we establish that the joint Stein prior for the unknown scale case leads to a minimax admissible estimator in all dimensions greater than $2$.

Marginal Likelihood Computation for Model Selection and Hypothesis Testing: An Extensive Review

This is an up-to-date introduction to, and overview of, marginal likelihood computation for model selection and hypothesis testing. Computing normalizing constants of probability models (or ratios of constants) is a fundamental issue in many applications in statistics, applied mathematics, signal processing, and machine learning. This article provides a comprehensive study of the state of the art of the topic. We highlight limitations, benefits, connections, and differences among the different techniques. Problems and possible solutions with the use of improper priors are also described. Some of the most relevant methodologies are compared through theoretical comparisons and numerical experiments.

On the safe use of prior densities for Bayesian model selection

AbstractThe application of Bayesian inference for the purpose of model selection is very popular nowadays. In this framework, models are compared through their marginal likelihoods, or their quotients, called Bayes factors. However, marginal likelihoods depend on the prior choice. For model selection, even diffuse priors can be actually very informative, unlike for the parameter estimation problem. Furthermore, when the prior is improper, the marginal likelihood of the corresponding model is undetermined. In this work, we discuss the issue of prior sensitivity of the marginal likelihood and its role in model selection. We also comment on the use of uninformative priors, which are very common choices in practice. Several practical suggestions are discussed and many possible solutions, proposed in the literature, to design objective priors for model selection are described. Some of them also allow the use of improper priors. The connection between the marginal likelihood approach and the well‐known information criteria is also presented. We describe the main issues and possible solutions by illustrative numerical examples, providing also some related code. One of them involving a real‐world application on exoplanet detection.This article is categorized under:Statistical Models &gt; Bayesian ModelsStatistical Models &gt; Fitting ModelsStatistical Models &gt; Model Selection

An objective Bayes factor with improper priors

A new look at the use of improper priors in Bayes factors for model comparison is presented. As is well known, in such a case, the Bayes factor is only defined up to an arbitrary constant. Most current methods overcome the problem by using part of the sample to train the Bayes factor (Fractional Bayes Factor) or to transform the improper prior in to a proper distribution (Intrinsic Bayes Factors) and use the remainder of the sample for the model comparison. It is provided an alternative approach which relies on matching divergences between density functions so as to establish a value for the constant appearing in the Bayes factor. These are the Kullback–Leibler divergence and the Fisher information divergence; the latter being crucial as it does not depend on an unknown normalizing constant. Demonstrations of the performance of the proposed method are provided through numerous illustrations and comparisons, showing that the main advantage over existing ones is that it does not require any input from the experimenter; it is fully automated.

A New Statistic for Bayesian Hypothesis Testing

A new Bayesian–inspired statistic for hypothesis testing is proposed which compares two posterior distributions; the observed posterior and the expected posterior under the null model. The Kullback–Leibler divergence between the two posterior distributions yields a test statistic which can be interpreted as a penalized log–Bayes factor with the penalty term converging to a constant as the sample size increases. Hence, asymptotically, the statistic behaves as a Bayes factor. Viewed as a penalized Bayes factor, this approach solves the long standing issue of using improper priors with the Bayes factor, since only posterior summaries are needed for the new statistic. Further motivation for the new statistic is a minimal move from the Bayes factor which requires no tuning nor splitting of data into training and inference, and can use improper priors. Critical regions for the test can be assessed using frequentist notions of Type I error.

Analyzing relevance vector machines using a single penalty approach

AbstractRelevance vector machine (RVM) is a popular sparse Bayesian learning model typically used for prediction. Recently it has been shown that improper priors assumed on multiple penalty parameters in RVM may lead to an improper posterior. Currently in the literature, the sufficient conditions for posterior propriety of RVM do not allow improper priors over the multiple penalty parameters. In this article, we propose a single penalty relevance vector machine (SPRVM) model in which multiple penalty parameters are replaced by a single penalty and we consider a semi‐Bayesian approach for fitting the SPRVM. The necessary and sufficient conditions for posterior propriety of SPRVM are more liberal than those of RVM and allow for several improper priors over the penalty parameter. Additionally, we also prove the geometric ergodicity of the Gibbs sampler used to analyze the SPRVM model and hence can estimate the asymptotic standard errors associated with the Monte Carlo estimate of the means of the posterior predictive distribution. Such a Monte Carlo standard error cannot be computed in the case of RVM, since the rate of convergence of the Gibbs sampler used to analyze RVM is not known. The predictive performance of RVM and SPRVM is compared by analyzing two simulation examples and three real life datasets.

Shrinkage Priors for Nonparametric Bayesian Prediction of Nonhomogeneous Poisson Processes

We consider nonparametric Bayesian estimation and prediction for nonhomogeneous Poisson process models with unknown intensity functions. We propose a class of improper priors for intensity functions. Nonparametric Bayesian inference with kernel mixture based on the class improper priors is shown to be useful, although improper priors have not been widely used for nonparametric Bayes problems. Several theorems corresponding to those for finite-dimensional independent Poisson models hold for nonhomogeneous Poisson process models with infinite-dimensional parameter spaces. Bayesian estimation and prediction based on the improper priors are shown to be admissible under the Kullback-Leibler loss. Numerical methods for Bayesian inference based on the priors are investigated.

Improper priors and improper posteriors

AbstractWhat is a good prior? Actual prior knowledge should be used, but for complex models this is often not easily available. The knowledge can be in the form of symmetry assumptions, and then the choice will typically be an improper prior. Also more generally, it is quite common to choose improper priors. Motivated by this we consider a theoretical framework for statistics that includes both improper priors and improper posteriors. Knowledge is then represented by a possibly unbounded measure with interpretation as explained by Rényi in 1955. The main mathematical result here is a constructive proof of existence of a transformation from prior to posterior knowledge. The posterior always exists and is uniquely defined by the prior, the observed data, and the statistical model. The transformation is, as it should be, an extension of conventional Bayesian inference as defined by the axioms of Kolmogorov. It is an extension since the novel construction is valid also when replacing the axioms of Kolmogorov by the axioms of Rényi for a conditional probability space. A concrete case based on Markov Chain Monte Carlo simulations and data for different species of tropical butterflies illustrate that an improper posterior may appear naturally and is useful. The theory is also exemplified by more elementary examples.

A Discrete Density Approach to Bayesian Quantile and Expectile Regression with Discrete Responses

For decades, regression models beyond the mean for continuous responses have attracted great attention in the literature. These models typically include quantile regression and expectile regression. But there is little research on these regression models for discrete responses, particularly from a Bayesian perspective. By forming the likelihood function based on suitable discrete probability mass functions, this paper introduces a discrete density approach for Bayesian inference of these regression models with discrete responses. Bayesian quantile regression for discrete responses is first developed, and then this method is extended to Bayesian expectile regression for discrete responses. The posterior distribution under this approach is shown not only coherent irrespective of the true distribution of the response, but also proper with regarding to improper priors for the unknown model parameters. The performance of the method is evaluated via extensive Monte Carlo simulation studies and one real data analysis.

Prior Sample Size Extensions for Assessing Prior Impact and Prior-Likelihood Discordance

AbstractThis paper outlines a framework for quantifying the prior’s contribution to posterior inference in the presence of prior-likelihood discordance, a broader concept than the usual notion of prior-likelihood conflict. We achieve this dual purpose by extending the classic notion of prior sample size, M, in three directions: (I) estimating M beyond conjugate families; (II) formulating M as a relative notion that is as a function of the likelihood sample size k, M(k), which also leads naturally to a graphical diagnosis; and (III) permitting negative M, as a measure of prior-likelihood conflict, that is, harmful discordance. Our asymptotic regime permits the prior sample size to grow with the likelihood data size, hence making asymptotic arguments meaningful for investigating the impact of the prior relative to that of likelihood. It leads to a simple asymptotic formula for quantifying the impact of a proper prior that only involves computing a centrality and a spread measure of the prior and the posterior. We use simulated and real data to illustrate the potential of the proposed framework, including quantifying how weak is a ‘weakly informative’ prior adopted in a study of lupus nephritis. Whereas we take a pragmatic perspective in assessing the impact of a prior on a given inference problem under a specific evaluative metric, we also touch upon conceptual and theoretical issues such as using improper priors and permitting priors with asymptotically non-vanishing influence.

Anna Karenina and the two envelopes problem

SummaryThe Anna Karenina principle is named after the opening sentence in the eponymous novel: Happy families are all alike; every unhappy family is unhappy in its own way. The two envelopes problem (TEP) is a much‐studied paradox in probability theory, mathematical economics, logic and philosophy. Time and again a new analysis is published in which an author claims finally to explain what actually goes wrong in this paradox. Each author (the present author included) emphasises what is new in their approach and concludes that earlier approaches did not get to the root of the matter. We observe that though a logical argument is only correct if every step is correct, an apparently logical argument which goes astray can be thought of as going astray at different places. This leads to a comparison between the literature on TEP and a successful movie franchise: it generates a succession of sequels, and even prequels, each with a different director who approaches the same basic premise in a personal way. We survey resolutions in the literature with a view to synthesis, correct common errors, and give a new theorem on order properties of an exchangeable pair of random variables, at the heart of most TEP variants and interpretations. A theorem on asymptotic independence between the amount in your envelope and the question whether it is smaller or larger shows that the pathological situation of improper priors or infinite expectation values has consequences as we merely approach such a situation.

Posterior impropriety of some sparse Bayesian learning models

Sparse Bayesian learning models are typically used for prediction in datasets with significantly greater number of covariates than observations. Such models often take a reproducing kernel Hilbert space (RKHS) approach to carry out the task of prediction and can be implemented using either proper or improper priors. In this article we show that a few sparse Bayesian learning models in the literature, when implemented using improper priors, lead to improper posteriors.

Block Gibbs samplers for logistic mixed models: Convergence properties and a comparison with full Gibbs samplers

The logistic linear mixed model (LLMM) is one of the most widely used statistical models. Generally, Markov chain Monte Carlo algorithms are used to explore the posterior densities associated with the Bayesian LLMMs. Polson, Scott and Windle's (2013) Polya-Gamma data augmentation (DA) technique can be used to construct full Gibbs (FG) samplers for the LLMMs. Here, we develop efficient block Gibbs (BG) samplers for Bayesian LLMMs using the Polya-Gamma DA method. We compare the FG and BG samplers in the context of a real data example, as the correlation between the fixed effects and the random effects changes as well as when the dimensions of the design matrices vary. These numerical examples demonstrate superior performance of the BG samplers over the FG samplers. We also derive conditions guaranteeing geometric ergodicity of the BG Markov chain when the popular improper uniform prior is assigned on the regression coefficients, and proper or improper priors are placed on the variance parameters of the random effects. This theoretical result has important practical implications as it justifies the use of asymptotically valid Monte Carlo standard errors for Markov chain based estimates of the posterior quantities.

On Posterior Properties of the Two Parameter Gamma Family of Distributions.

The gamma distribution has been extensively used in many areas of applications. In this paper, considering a Bayesian analysis we provide necessary and sufficient conditions to check whether or not improper priors lead to proper posterior distributions. Further, we also discuss sufficient conditions to verify if the obtained posterior moments are finite. An interesting aspect of our findings are that one can check if the posterior is proper or improper and also if its posterior moments are finite by looking directly in the behavior of the proposed improper prior. To illustrate our proposed methodology these results are applied in different objective priors.

Optional Stopping with Bayes Factors: A Categorization and Extension of Folklore Results, with an Application to Invariant Situations

It is often claimed that Bayesian methods, in particular Bayes factor methods for hypothesis testing, can deal with optional stopping. We first give an overview, using elementary probability theory, of three different mathematical meanings that various authors give to this claim: (1) stopping rule independence, (2) posterior calibration and (3) (semi-) frequentist robustness to optional stopping. We then prove theorems to the effect that these claims do indeed hold in a general measure-theoretic setting. For claims of type (2) and (3), such results are new. By allowing for non-integrable measures based on improper priors, we obtain particularly strong results for the practically important case of models with nuisance parameters satisfying a group invariance (such as location or scale). We also discuss the practical relevance of (1)–(3), and conclude that whether Bayes factor methods actually perform well under optional stopping crucially depends on details of models, priors and the goal of the analysis.

Posterior properties of the Weibull distribution for censored data

The Weibull distribution is one of the most used tools in reliability analysis. In this paper, assuming a Bayesian approach, we propose necessary and sufficient conditions to verify when improper priors lead to proper posteriors for the parameters of the Weibull distribution in the presence of complete or right-censored data. Additionally, we proposed sufficient conditions to verify if the obtained posterior moments are finite. These results can be achieved by checking the behavior of the improper priors, which are applied in different objective priors to illustrate the usefulness of the new results. As an application of our theorem, we prove that if the improper prior leads to a proper posterior, the posterior mean, as well as other higher moments of the scale parameter, are not finite and, therefore, should not be used.

A Flexible Skewed Link Model for Ordinal Outcomes: An Application to Infertility

BACKGROUND: An important issue in modeling categorical response data is the choice of the links. The commonly used complementary log-log link is inclined to link misspecification due to its positive and fixed skewness parameter.&#x0D; AIM: The objective of this paper is to introduce a flexible skewed link function for modeling ordinal data with some covariates.&#x0D; METHODS: We introduce a flexible skewed link model for the cumulative ordinal regression model based on Chen model.&#x0D; RESULTS: The main advantage suggested by the proposed links is the skewed link provide much more identifiable than the existing skewed links. The propriety of posterior distributions under proper and improper priors is explored in detail. An efficient Markov chain Monte Carlo algorithm is developed for sampling from the posterior distribution.&#x0D; CONCLUSION: The proposed methodology is motivated and illustrated by ovary hyperstimulation syndrome data.

Bayesian inference for an ordered multiple linear regression with skew normal errors

This paper studies a Bayesian ordered multiple linear regression model with skew normal error. It is reasonable that the kind of inherent information available in an applied regression requires some constraints on the coefficients to be estimated. In addition, the assumption of normality of the errors is sometimes not appropriate in the real data. Therefore, to explain such situations more flexibly, we use the skew-normal distribution given by Sahu et al. (The Canadian Journal of Statistics, 31, 129-150, 2003) for error-terms including normal distribution. For Bayesian methodology, the Markov chain Monte Carlo method is employed to resolve complicated integration problems. Also, under the improper priors, the propriety of the associated posterior density is shown. Our Bayesian proposed model is applied to NZAPB’s apple data. For model comparison between the skew normal error model and the normal error model, we use the Bayes factor and deviance information criterion given by Spiegelhalter et al. (Journal of the Royal Statistical Society Series B (Statistical Methodology), 64, 583-639, 2002). We also consider the problem of detecting an influential point concerning skewness using Bayes factors. Finally, concluding remarks are discussed.