Lindley's Paradox Research Articles

Bayesian hypothesis tests with diffuse priors: Can we have our cake and eat it too?

SummaryWe propose a new class of priors for Bayesian hypothesis testing, which we name ‘cake priors’. These priors circumvent the Jeffreys–Lindley paradox (also called Bartlett's paradox) a problem associated with the use of diffuse priors leading to nonsensical statistical inferences. Cake priors allow the use of diffuse priors (having one's cake) while achieving theoretically justified inferences (eating it too). We demonstrate this methodology for Bayesian hypotheses tests for various common scenarios. The resulting Bayesian test statistic takes the form of a penalised likelihood ratio test statistic. Under typical regularity conditions, we show that Bayesian hypothesis tests based on cake priors are Chernoff consistent, that is, achieve zero type I and II error probabilities asymptotically. We also discuss Lindley's paradox and argue that the paradox occurs with small and vanishing probability as sample size increases.

The Jeffreys–Lindley paradox: an exchange

This Editorial reports an exchange in form of a comment and reply on the article “History and Nature of the Jeffreys–Lindley Paradox” (Arch Hist Exact Sci 77:25, 2023) by Eric-Jan Wagenmakers and Alexander Ly.

History and nature of the Jeffreys–Lindley paradox

The Jeffreys–Lindley paradox exposes a rift between Bayesian and frequentist hypothesis testing that strikes at the heart of statistical inference. Contrary to what most current literature suggests, the paradox was central to the Bayesian testing methodology developed by Sir Harold Jeffreys in the late 1930s. Jeffreys showed that the evidence for a point-null hypothesis H0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {H}}_0$$\\end{document} scales with n\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sqrt{n}$$\\end{document} and repeatedly argued that it would, therefore, be mistaken to set a threshold for rejecting H0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {H}}_0$$\\end{document} at a constant multiple of the standard error. Here, we summarize Jeffreys’s early work on the paradox and clarify his reasons for including the n\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\sqrt{n}$$\\end{document} term. The prior distribution is seen to play a crucial role; by implicitly correcting for selection, small parameter values are identified as relatively surprising under H1\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$${\\mathcal {H}}_1$$\\end{document}. We highlight the general nature of the paradox by presenting both a fully frequentist and a fully Bayesian version. We also demonstrate that the paradox does not depend on assigning prior mass to a point hypothesis, as is commonly believed.

Kenneth (Ken) Ronald Walter Brewer (1931–2021)

Kenneth (Ken) Ronald Walter Brewer (1931–2021)

A Bayesian machine learning approach for online detection of railway wheel defects using track-side monitoring

Wheel condition assessment is of great significance to ensure the operation safety of trains and metro systems. This study is intended to develop a Bayesian probabilistic method for online and quantitative assessment of railway wheel conditions using track-side strain-monitoring data. The proposed method is a fully data-driven, nonparametric approach without the need of a physical model. To enable defect identification using only response measurement, the measured dynamic strain responses of rail tracks during the passage of trains are processed to elicit the normalized cumulative distribution function values representative of the effect of individual wheels, which in conjunction with the frequency points are used to formulate a probabilistic reference model in terms of sparse Bayesian learning. Through cleverly realizing sparsity by introducing hyper-parameters and their priors, the sparse Bayesian learning makes the resulting model to exempt from overfitting and generalize well on unseen data. Only the monitoring data in healthy state are needed in formulating the reference model. A novel Bayesian null hypothesis significance testing in terms of scale-invariant intrinsic Bayes factor, which does not suffer from the Jeffreys–Lindley paradox, is then pursued in the presence of new monitoring data collected from possibly defective wheel(s) to detect wheel defects and quantitatively assess wheel condition. The proposed method in fully Bayesian inference framework is verified by utilizing the real-world monitoring data acquired by a distributed fiber Bragg grating–based track-side monitoring system and comparing with the offline inspection results.

On one-sample Bayesian tests for the mean

ABSTRACTThis paper deals with a new Bayesian approach to the standard one-sample z- and t- tests. More specifically, let be an independent random sample from a normal distribution with mean μ and variance . The goal is to test the null hypothesis against all possible alternatives. The approach is based on using the well-known formula of the Kullbak–Leibler divergence between two normal distributions (sampling and hypothesized distributions selected in an appropriate way). The change of the distance from a priori to a posteriori is compared through the relative belief ratio (a measure of evidence). Eliciting the prior, checking for prior-data conflict and bias are also considered. Many theoretical properties of the procedure have been developed. Besides it's simplicity, and unlike the classical approach, the new approach possesses attractive and distinctive features such as giving evidence in favour of the null hypothesis. It also avoids several undesirable paradoxes, such as Lindley's paradox that may be encountered by some existing Bayesian methods. The use of the approach has been illustrated through several examples.

Bayes Factor with Lindley Paradox and Tow Standard Methods in Model

For any statistical analysis, Model selection is necessary and required. In many cases of selection, Bayes factor is one of the important basic elements. For the unilateral hypothesis testing problem, we extend the harmony of frequency and Bayesian evidence to the generalized p-value of unilateral hypothesis testing problem, and study the harmony of generalized P-value and posterior probability of original hypothesis. For the problem of single point hypothesis testing, the posterior probability of the Bayes evidence under the traditional Bayes testing method, that is, the Bayes factor or the single point original hypothesis is established, is analyzed, a phenomenon known as the Lindley paradox, which is at odds with the classical frequency evidence of p-value. At this point, many statisticians have been worked for this from both frequentist and Bayesian perspective. In this paper, I am going to focus on Bayesian approach to model selection, starting from Bayes factors and going within Lindley Paradox, which also briefly talks about partial and fractional Bayes factor. Trying to use a simple way to consider this paradox is the thing what I want to do in the paper. On the other hand, a detailed derivation of BIC and AIC is given in Section 4. The guiding principle of selecting the optimal model is to investigate from two aspects: one is to maximize the likelihood function, the other is to minimize the number of unknown parameters in the model. The larger the likelihood function value, the better the model fitting, but we can not simply measure the model fitting accuracy, which leads to more and more unknown parameters in the model, and the model that becomes more and more complex would have caused an overmatch. Therefore, a good model should be the combination of the fitting accuracy and the number of unknown parameters to optimize the configuration.

On Analysis of the Behrens-Fisher Problem Based on Bayesian Evidence

In this paper we have demonstrated the ability of the new Bayesian measure of evidence of Yin (2012, Computational Statistics, 27: 237-249) to solve both the Behrens-Fisher problem and Lindley's paradox. We have provided a general proof that for any prior which yields a linear combination of two independent t random variables as posterior distribution of the di erence of means, the new Bayesian measure of evidence given that prior will solve Lindleys' paradox thereby serving as a general proof for the works of Yin and Li (2014, Journal of Applied Mathematics, 2014(978691)) and Goltong and Doguwa (2018, Open Journal of Statistics, 8: 902-914). Using the Pareto prior as an example, we have shown by the use of simulation results that the new Bayesian measure of evidence solves Lindley's paradox.

Should significance testing be abandoned in machine learning?

Significance testing has become a mainstay in machine learning, with the p value being firmly embedded in the current research practice. Significance tests are widely believed to lend scientific rigor to the interpretation of empirical findings; however, their problems have received only scant attention in the machine learning literature so far. Here, we investigate one particular problem, the Jeffreys–Lindley paradox. This paradox describes a statistical conundrum: the p value can be close to zero, convincing us that there is overwhelming evidence against the null hypothesis. At the same time, however, the posterior probability of the null hypothesis being true can be close to 1, convincing us of the exact opposite. In experiments with synthetic data sets and a subsequent thought experiment, we demonstrate that this paradox can have severe repercussions for the comparison of multiple classifiers over multiple benchmark data sets. Our main result suggests that significance tests should not be used in such comparative studies. We caution that the reliance on significance tests might lead to a situation that is similar to the reproducibility crisis in other fields of science. We offer for debate four avenues that might alleviate the looming crisis.

A reverse to the Jeffreys–Lindley paradox

In this paper the seminal Jeffreys–Lindley paradox is regarded from a mathematical point of view. We show that in certain scenarios the paradox may emerge in a reverse direction.

On the mathematics of the Jeffreys–Lindley paradox

ABSTRACTThis paper is concerned with the well known Jeffreys–Lindley paradox. In a Bayesian set up, the so-called paradox arises when a point null hypothesis is tested and an objective prior is sought for the alternative hypothesis. In particular, the posterior for the null hypothesis tends to one when the uncertainty, i.e., the variance, for the parameter value goes to infinity. We argue that the appropriate way to deal with the paradox is to use simple mathematics, and that any philosophical argument is to be regarded as irrelevant.

A conditional Lindley paradox in Bayesian linear models

The development of prior distributions for Bayesian regression has traditionally been driven by the goal of achieving sensible model selection and parameter estimation. The formalization of properties that characterize good performance has led to the development and popularization of thick-tailed mixtures of g priors such as the Zellner–Siow and hyper-g priors. In this paper we introduce a conditional information asymptotic regime that is motivated by the common data analysis setting where at least one regression coefficient is much larger than the others. We analyse existing mixtures of g priors under this limit and reveal two new phenomena, essentially least-squares estimation and the conditional Lindley paradox, and argue that these are, in general, undesirable. The driver behind both is the use of a single, latent scale parameter common to all coefficients.

The Lindley paradox in optical interferometry

The so-called Lindley paradox is a counterintuitive statistical effect where the Bayesian and frequentist approaches to hypothesis testing give radically different answers, depending on the choice of the prior distribution. In this paper we address the occurrence of the Lindley paradox in optical interferometry and discuss its implications for high-precision measurements. In particular, we focus on phase estimation by Mach–Zehnder interferometers and show how to mitigate the conflict between the two approaches by using suitable priors.

Erratum to: The Jeffreys–Lindley paradox and discovery criteria in high energy physics

Erratum to: The Jeffreys–Lindley paradox and discovery criteria in high energy physics

The Jeffreys–Lindley paradox and discovery criteria in high energy physics

The Jeffreys-Lindley paradox displays how the use of a p-value (or number of standard deviations z) in a frequentist hypothesis test can lead to an inference that is radically different from that of a Bayesian hypothesis test in the form advocated by Harold Jeffreys in the 1930s and common today. The setting is the test of a well-specified null hypothesis (such as the Standard Model of elementary particle physics, possibly with "nuisance parameters") versus a composite alternative (such as the Standard Model plus a new force of nature of unknown strength). The p-value, as well as the ratio of the likelihood under the null hypothesis to the maximized likelihood under the alternative, can strongly disfavor the null hypothesis, while the Bayesian posterior probability for the null hypothesis can be arbitrarily large. The academic statistics literature contains many impassioned comments on this paradox, yet there is no consensus either on its relevance to scientific communication or on its correct resolution. The paradox is quite relevant to frontier research in high energy physics. This paper is an attempt to explain the situation to both physicists and statisticians, in the hope that further progress can be made.

A personal history of Bayesian statistics

The history of Bayesian statistics is traced, from a personal perspective, through various strands and via its re‐genesis during the 1960s to the current day. Emphasis is placed on broad‐sense Bayesian methodology that can be used to meaningfully analyze observed datasets. Over 750 people in science, medicine, and socioeconomics, who have influenced the evolution of the Bayesian approach into the powerful paradigm that it is today, are highlighted. The frequentist/Bayesian controversy is addressed, together with the ways in which many Bayesians combine the two ideologies as a Bayes/non‐Bayes compromise, e.g., when drawing inferences about unknown parameters or when investigating the choice of sampling model in relation to its real‐life background. A number of fundamental issues are discussed and critically examined, and some elementary explanations for nontechnical readers and some personal reminiscences are included. Some of the Bayesian contributions of the 21st century are subjected to more detailed critique, so that readers may learn more about the quality and relevance of the ongoing research. A recent resolution of Lindley's paradox by Baskurt and Evans is reported. The axioms of subjective probability are reassessed, some state‐of‐the‐art alternatives to Leonard Savage's axioms of utility are discussed, and Deborah Mayo and Michael Evan's refutation of Allan Birnbaum's 1962 justification of the likelihood principle in terms of the sufficiency and conditionality principles is addressed.WIREs Comput Stat2014, 6:80–115. doi: 10.1002/wics.1293This article is categorized under:Statistical and Graphical Methods of Data Analysis > Bayesian Methods and Theory

RECOMMENDATIONS FOR USING MSBAYES TO INCORPORATE UNCERTAINTY IN SELECTING AN ABC MODEL PRIOR: A RESPONSE TO OAKS ET AL.

Prior specification is an essential component of parameter estimation and model comparison in Approximate Bayesian computation (ABC). Oaks et al. present a simulation-based power analysis of msBayes and conclude that msBayes has low power to detect genuinely random divergence times across taxa, and suggest the cause is Lindley's paradox. Although the predictions are similar, we show that their findings are more fundamentally explained by insufficient prior sampling that arises with poorly chosen wide priors that critically undersample nonsimultaneous divergence histories of high likelihood. In a reanalysis of their data on Philippine Island vertebrates, we show how this problem can be circumvented by expanding upon a previously developed procedure that accommodates uncertainty in prior selection using Bayesian model averaging. When these procedures are used, msBayes supports recent divergences without support for synchronous divergence in the Oaks et al. data and we further present a simulation analysis that demonstrates that msBayes can have high power to detect asynchronous divergence under narrower priors for divergence time. Our findings highlight the need for exploration of plausible parameter space and prior sampling efficiency for ABC samplers in high dimensions. We discus potential improvements to msBayes and conclude that when used appropriately with model averaging, msBayes remains an effective and powerful tool.

Bayes factors for testing inequality constrained hypotheses: Issues with prior specification

Several issues are discussed when testing inequality constrained hypotheses using a Bayesian approach. First, the complexity (or size) of the inequality constrained parameter spaces can be ignored. This is the case when using the posterior probability that the inequality constraints of a hypothesis hold, Bayes factors based on non-informative improper priors, and partial Bayes factors based on posterior priors. Second, the Bayes factor may not be invariant for linear one-to-one transformations of the data. This can be observed when using balanced priors which are centred on the boundary of the constrained parameter space with a diagonal covariance structure. Third, the information paradox can be observed. When testing inequality constrained hypotheses, the information paradox occurs when the Bayes factor of an inequality constrained hypothesis against its complement converges to a constant as the evidence for the first hypothesis accumulates while keeping the sample size fixed. This paradox occurs when using Zellner's g prior as a result of too much prior shrinkage. Therefore, two new methods are proposed that avoid these issues. First, partial Bayes factors are proposed based on transformed minimal training samples. These training samples result in posterior priors that are centred on the boundary of the constrained parameter space with the same covariance structure as in the sample. Second, a g prior approach is proposed by letting g go to infinity. This is possible because the Jeffreys-Lindley paradox is not an issue when testing inequality constrained hypotheses. A simulation study indicated that the Bayes factor based on this g prior approach converges fastest to the true inequality constrained hypothesis.

The importance of prior choice in model selection: a density dependence example

Summary It is important to discern the magnitude of density dependence a species exhibits, as well as the time lag over which it operates. Knowledge of a species' likely response to natural as well as synthetic shocks will assist in effective species management. Statistically this is a challenging problem which does not usually admit closed‐form mathematical analysis. Consequently, many people have used Bayesian methods to fit state space models of density dependence to many different species, of which we take eleven species of North American duck as our motivating examples. A Bayesian analysis requires a choice of model and parameter prior. The latter is difficult to do without inducing bias in model selection, and we attempt to address this problem. Our priors will be obtained by considering which parameter values are representative of features we expect to see in the data, and which would produce unnatural behaviour. To fit the models, we use a novel sequential Monte Carlo method (particle learning) not previously applied to ecological data sets. We show that existing analyses on the duck data may have been susceptible to a common problem in Bayesian model selection (Lindley's paradox), and suggest methods for prior selection which mitigate this issue. We also discover that although it is possible to detect the existence of density dependence, it is unrealistic to expect to determine the time lag over which it operates without a great deal of data, even if said data are simulated from the model. We demonstrate that prior choices motivated by the above considerations can lead to substantially increased predictive accuracy over surprisingly long time scales whether model selection is of primary concern or not. We conclude from our analysis of real‐world data that there is little evidence of density dependence in many duck species, suggesting that such effects, if present, are likely to be small in magnitude.

Test procedures based on combination of Bayesian evidences for $H_{0}$

We introduce two procedures for testing which are based on pooling the posterior evidence for the null hypothesis provided by the full Bayesian significance test and the posterior probability for the null hypothesis. Although the proposed procedures can be used in more general situations, we focus attention in tests for a precise null hypothesis. We prove that the proposed procedure based on the linear operator is a Bayes rule. We also verify that it does not lead to the Jeffreys–Lindley paradox. For a precise null hypothesis, we prove that the procedure based on the logarithmic operator is a generalization of Jeffreys test. We apply the results to some well-known probability families. The empirical results show that the proposed procedures present good performances. As a by-product we obtain tests for normality under the skew-normal one.