Classical P-values Research Articles

Relationships Between p-values and Pearson Correlation Coefficients, Type 1 Errors and Effect Size Errors, Under a True Null Hypothesis

The American Statistical Association (ASA) published a statement in 2016 in The American Statistician for “researchers, practitioners and science writers who are not primarily statisticians” on the proper use and interpretation of p-values. Three years later, the ASA acknowledged that telling researchers what not to do with p-values was insufficient. Consequently, 3 years later an open access, special issue appeared with 43 papers proposing various novel and sophisticated alternatives to classical p-values for use with scientific methods in the twenty-first century. In the opening remarks, the editors stated that “no p-value can reveal the plausibility, presence, truth, or importance of an association or effect” and banned statistical significance: “don’t say it, don’t use it.” This paper questions both statements with a simulated data study. It is shown that p-values are strongly related to correlation coefficients under a true null hypothesis; hence, can reveal the “importance of an association or effect.” Furthermore, it demonstrates why a cut point for statistical significance is still a viable, ancillary tool for assessing the substantive significance of statistical effects with small sample sizes (n < 1000).

Second-Generation P-Values, Shrinkage, and Regularized Models

Second-generation p-values (SGPVs) are a novel and intuitive extension of classical p-values that better summarize the degree to which data support scientific hypotheses. SGPVs measure the overlap between an uncertainty interval for the parameter of interest and an interval null hypothesis that represents the set of null and practically null hypotheses. Although SGPVs are always in the unit interval, they are not formal probabilities. Rather, SGPVs are summary measures of when the data are compatible with null hypotheses (SGPV=1), compatible with alternative hypotheses (SGPV=0), or inconclusive (0<SGPV<1). Because second-generation p-values differentiate between inconclusive and null results, their Type I Error rate converges to zero along with the Type II Error rate. The SGPV approach is also inferentially agnostic: it can be applied to any uncertainty interval about a parameter of interest such as confidence intervals, likelihood support intervals, and Bayesian highest posterior density intervals. This paper revisits the motivation for using SGPVs and explores their long-run behavior under regularized models that provide shrinkage on point estimates. While shrinkage often results in a more desirable bias-variance trade-off, the impact of shrinkage on the error rates of SGPVs is not well understood. Through extensive simulations, we find that SPGVs based on shrunken estimates retain the desirable error rate behavior of SGPVs that we observe in classical models - albeit with a minor loss of power – while also retaining the benefits of bias-variance tradeoff.

The Bayes Factor for the Misclassified Categorical Data

This article addresses the issue of misclassification in a single categorical variable, that is, how to test whether the collected categorical data are misclassified. To tackle this issue, a pair of null and alternative hypotheses is proposed. A mixed Bayesian approach is taken to test these hypotheses. Specifically, a bias-adjusted cell proportion estimator is presented that accounts for the bias caused by classification errors in the observed categorical data. The chi-square test is then adjusted accordingly. To test the null hypothesis that the data are not misclassified under a specified multinomial distribution against the alternative hypothesis they are misclassified, the Bayes factor is calculated for the observed data and a comparison is made with the classical p-value.

Default \u201cGunel and Dickey\u201d Bayes factors for contingency tables

The analysis of R×C contingency tables usually features a test for independence between row and column counts. Throughout the social sciences, the adequacy of the independence hypothesis is generally evaluated by the outcome of a classical p-value null-hypothesis significance test. Unfortunately, however, the classical p-value comes with a number of well-documented drawbacks. Here we outline an alternative, Bayes factor method to quantify the evidence for and against the hypothesis of independence in R×C contingency tables. First we describe different sampling models for contingency tables and provide the corresponding default Bayes factors as originally developed by Gunel and Dickey (Biometrika, 61(3):545–557 (1974)). We then illustrate the properties and advantages of a Bayes factor analysis of contingency tables through simulations and practical examples. Computer code is available online and has been incorporated in the “BayesFactor” R package and the JASP program (jasp-stats.org).

Posterior predictive checking of multiple imputation models.

Multiple imputation is gaining popularity as a strategy for handling missing data, but there is a scarcity of tools for checking imputation models, a critical step in model fitting. Posterior predictive checking (PPC) has been recommended as an imputation diagnostic. PPC involves simulating "replicated" data from the posterior predictive distribution of the model under scrutiny. Model fit is assessed by examining whether the analysis from the observed data appears typical of results obtained from the replicates produced by the model. A proposed diagnostic measure is the posterior predictive "p-value", an extreme value of which (i.e., a value close to 0 or 1) suggests a misfit between the model and the data. The aim of this study was to evaluate the performance of the posterior predictive p-value as an imputation diagnostic. Using simulation methods, we deliberately misspecified imputation models to determine whether posterior predictive p-values were effective in identifying these problems. When estimating the regression parameter of interest, we found that more extreme p-values were associated with poorer imputation model performance, although the results highlighted that traditional thresholds for classical p-values do not apply in this context. A shortcoming of the PPC method was its reduced ability to detect misspecified models with increasing amounts of missing data. Despite the limitations of posterior predictive p-values, they appear to have a valuable place in the imputer's toolkit. In addition to automated checking using p-values, we recommend imputers perform graphical checks and examine other summaries of the test quantity distribution.

How Should an Economist Do Statistics?

Much of economists’ statistical work centers on testing hypotheses in which parameter values are partitioned between a null hypothesis and an alternative hypothesis in order to distinguish two views about the world. Our traditional procedures are based on the probabilities of a test statistic under the null but ignore what the statistics say about the probability of the test statistic under the alternative. Traditional procedures are not intended to provide evidence for the relative probabilities of the null versus alternative hypotheses, but are regularly treated as if they do. Unfortunately, when used to distinguish two views of the world, traditional procedures can lead to wildly misleading inference. In order to correctly distinguish between two views of the world, one needs to report the probabilities of the hypotheses given parameter estimates rather than the probability of the parameter estimates given the hypotheses. For most standard econometric estimators, it is not difficult to compute the proper probabilities using Bayes theorem. Doing so will often change empirical conclusions. Calculations that are respectful of Bayes Theorem ought to supplement, or perhaps supplant, classical test statistics and p-values in reports of empirical results.

Alternative-based thresholding with application to presurgical fMRI

Functional magnetic reasonance imaging (fMRI) plays an important role in pre-surgical planning for patients with resectable brain lesions such as tumors. With appropriately designed tasks, the results of fMRI studies can guide resection, thereby preserving vital brain tissue. The mass univariate approach to fMRI data analysis consists of performing a statistical test in each voxel, which is used to classify voxels as either active or inactive-that is, related, or not, to the task of interest. In cognitive neuroscience, the focus is on controlling the rate of false positives while accounting for the severe multiple testing problem of searching the brain for activations. However, stringent control of false positives is accompanied by a risk of false negatives, which can be detrimental, particularly in clinical settings where false negatives may lead to surgical resection of vital brain tissue. Consequently, for clinical applications, we argue for a testing procedure with a stronger focus on preventing false negatives. We present a thresholding procedure that incorporates information on false positives and false negatives. We combine two measures of significance for each voxel: a classical p-value, which reflects evidence against the null hypothesis of no activation, and an alternative p-value, which reflects evidence against activation of a prespecified size. This results in a layered statistical map for the brain. One layer marks voxels exhibiting strong evidence against the traditional null hypothesis, while a second layer marks voxels where activation cannot be confidently excluded. The third layer marks voxels where the presence of activation can be rejected.

A new Bayesian procedure for testing point null hypotheses

Testing point null hypotheses is a very common activity in various applied situations. However, the existing Bayesian testing procedure may give evidence which does not agree with the classical frequentist p-value in many point null testing situations. A typical example for this is the well known Lindley’s paradox (Lindley in Biometrika 44:187–192, 1957). In this paper we propose an alternative testing procedure in the Bayesian framework. It is shown that for many classical testing examples, the Bayesian evidence derived by our new testing procedure is not contradictory to its frequentist counterpart any more. In fact, the new Bayesian evidence under the noninformative prior is usually coincident with the frequentist observed significance level.

Searching for Footprints of Positive Selection in Whole-Genome SNP Data From Nonequilibrium Populations

A major goal of population genomics is to reconstruct the history of natural populations and to infer the neutral and selective scenarios that can explain the present-day polymorphism patterns. However, the separation between neutral and selective hypotheses has proven hard, mainly because both may predict similar patterns in the genome. This study focuses on the development of methods that can be used to distinguish neutral from selective hypotheses in equilibrium and nonequilibrium populations. These methods utilize a combination of statistics on the basis of the site frequency spectrum (SFS) and linkage disequilibrium (LD). We investigate the patterns of genetic variation along recombining chromosomes using a multitude of comparisons between neutral and selective hypotheses, such as selection or neutrality in equilibrium and nonequilibrium populations and recurrent selection models. We perform hypothesis testing using the classical P-value approach, but we also introduce methods from the machine-learning field. We demonstrate that the combination of SFS- and LD-based statistics increases the power to detect recent positive selection in populations that have experienced past demographic changes.

Classical and Bayesian interpretation of the Birge test of consistency and its generalized version for correlated results from interlaboratory evaluations

A well-known test of consistency in the results from an interlaboratory evaluation is the Birge test, named after its developer Raymond T Birge, a physicist. We show that the Birge test of consistency may be interpreted as a classical test of the null hypothesis that the variances of the results are less than or equal to their stated values against the alternative hypothesis that the variances of the results are greater than their stated values. A modern protocol for hypothesis testing is to calculate the classical p-value of the test statistic. The p-value is the maximum probability under the null hypothesis of realizing in conceptual replications a value of the test statistic equal to or larger than the realized (observed) value of the test statistic. The null hypothesis is rejected when the p-value is too small. We show that, interestingly, the classical p-value of the Birge test statistic is equal to the Bayesian posterior probability of the null hypothesis based on suitably chosen non-informative improper prior distributions for the unknown statistical parameters. Thus the Birge test may be interpreted also as a Bayesian test of the null hypothesis. The Birge test of consistency was developed for those interlaboratory evaluations where the results are uncorrelated. We present a general test of consistency for both correlated and uncorrelated results. Then we show that the classical p-value of the general test statistic is equal to the Bayesian posterior probability of the null hypothesis based on non-informative prior distributions. The general test makes it possible to check the consistency of correlated results from interlaboratory evaluations. The Birge test is a special case of the general test.

Reconciling Bayesian and Frequentist Evidence in the One-Sided Scale Parameter Testing Problem

An interesting but controversial problem arises when Bayesian as well as frequentist methodologies very often suggest similar solutions, thus creating a problem for the experimenter who wishes to make the best possible decision. For over a decade, there has been an effort by several authors to assess when Bayesian and frequentist methods provide exactly the same answers when employed. We encounter this situation in the problem of hypothesis testing, where Bayesian evidence, such as Bayes factors and prior or posterior predictive p-values are set against the classical p-values. All these measures provide a basis for rejection of a hypothesis or a model, but the selection of one over the other never comes without severe criticism. In this article, we develop prior predictive and posterior predictive p-values for one sided hypothesis testing for scale parameter problems. We reconcile Bayesian and frequentist evidence by showing that for many classes of prior distributions, the infimum of the prior predictive and posterior predictive p-values are equal to the classical p-value, for very general classes of distributions. The results are illustrated through examples relating to the one-sided testing problem for scale parameter.

Statistical strategies for avoiding false discoveries in metabolomics and related experiments

Many metabolomics, and other high-content or high-throughput, experiments are set up such that the primary aim is the discovery of biomarker metabolites that can discriminate, with a certain level of certainty, between nominally matched ‘case’ and ‘control’ samples. However, it is unfortunately very easy to find markers that are apparently persuasive but that are in fact entirely spurious, and there are well-known examples in the proteomics literature. The main types of danger are not entirely independent of each other, but include bias, inadequate sample size (especially relative to the number of metabolite variables and to the required statistical power to prove that a biomarker is discriminant), excessive false discovery rate due to multiple hypothesis testing, inappropriate choice of particular numerical methods, and overfitting (generally caused by the failure to perform adequate validation and cross-validation). Many studies fail to take these into account, and thereby fail to discover anything of true significance (despite their claims). We summarise these problems, and provide pointers to a substantial existing literature that should assist in the improved design and evaluation of metabolomics experiments, thereby allowing robust scientific conclusions to be drawn from the available data. We provide a list of some of the simpler checks that might improve one’s confidence that a candidate biomarker is not simply a statistical artefact, and suggest a series of preferred tests and visualisation tools that can assist readers and authors in assessing papers. These tools can be applied to individual metabolites by using multiple univariate tests performed in parallel across all metabolite peaks. They may also be applied to the validation of multivariate models. We stress in particular that classical p-values such as “p < 0.05”, that are often used in biomedicine, are far too optimistic when multiple tests are done simultaneously (as in metabolomics). Ultimately it is desirable that all data and metadata are available electronically, as this allows the entire community to assess conclusions drawn from them. These analyses apply to all high-dimensional ‘omics’ datasets.

Selecting "Significant" Differentially Expressed Genes from the Combined Perspective of the Null and the Alternative

In the search for genes associated with disease, statistical analysis yields a key towards reproducible results. To avoid a plethora of type I errors, classical gene selection procedures strike a balance between magnitude and precision of observed effects in terms of p-values. Protecting false discovery rates recovers some power but still ranks genes according to classical p-values. In contrast, we propose a selection procedure driven by the concern to detect well-specified important alternatives. By summarizing evidence from the perspective of both the null and such an alternative hypothesis, genes line up in a substantially different order with different genes yielding powerful signals. A cutoff point for a measure of relative evidence which balances the standard p-value, p0, with its counterpart, p1, derived from the perspective of the target alternative, determines our gene selection. We find the cutoff point that maximizes an expected specific gain. This yields an optimal decision which exploits gene-specific variances and thus involves different type I and type II errors across genes. We show the dramatic impact of this alternative perspective on the detection of differentially expressed genes in hereditary breast cancer. Our analysis does not rely on parametric assumptions on the data.

Significance and Impotence: Towards a Balanced View of the Null and the Alternative Hypotheses in Marker Selection for Plant Breeding

Summary The isolation of DNA markers that are linked to interesting genes helps plant breeders to select parent plants that transmit useful traits to future generations. Such ‘marker-assisted breeding and selection’ heavily leans on statistical testing of associations between markers and a well-chosen trait. Statistical association analysis is guided by classical p-values or the false discovery rate and thus relies predominantly on the null hypothesis. The main concern of plant breeders, however, is to avoid missing an important alternative. To judge evidence from this perspective, we complement the traditional p-value with a one-sided ‘alternative p-value’ which summarizes evidence against a target alternative in the direction of the null hypothesis. This p-value measures ‘impotence’ as opposed to significance: how likely is it to observe an outcome as extreme as or more extreme than the one that was observed when data stem from the alternative? We show how a graphical inspection of both p-values can guide marker selection when the null and the alternative hypotheses have a comparable importance. We derive formal decision tools with balanced properties yielding different rejection regions for different markers. We apply our approach to study rye-grass plants.

Reconciling Classical and Prior Predictive P-Values in the Two-Sided Location Parameter Testing Problem

Micheas and Dey (2003) reconciled classical and Bayesian p-values in the one-sided location parameter testing problem. In this article, the classical p-value is reconciled with the prior predictive p-value, for the two-sided location parameter testing problem, proving that the classical p-value coincides with the infimum of prior predictive p-values when the prior ranges in different classes of priors.

Asymptotic relationships between posterior probabilities and p-values using the hazard rate

In this paper the asymptotic relationship between the classical p-value and the infimum (over all unimodal and symmetric distributions) of the posterior probability in the point null hypothesis testing problem is analyzed. It is shown that the ratio between the infimum and the classical p-value has an equivalent asymptotic behavior to the hazard rate of the sample model.

Comparison of the P-value and posterior probability

For iid observations from a multivariate normal distribution in p dimensions with an unknown mean and a covariance matrix proportional to the Identity, we revisit the issue of the apparent irreconcilability of the classical test for a point null and the standard Bayesian formulation for testing such a point null. With appropriate families of priors on the alternative, we consider the threshold value of the a priori probability of the point null required for the smallest (over priors on the alternative) posterior probability and the classical P-value to coincide. We also consider, for an arbitrary but fixed a priori probability of the point null, the ratio of the minimum posterior probability and the classical P-value. The main results emphasize properties of the null distributions of these two quantities and their quartiles, etc. Among many theorems proved in the article are the results that regardless of the dimension p, the threshold prior probability as defined above has a median exactly equal to 0.5 in many cases, and the ratio as described above has a median exactly equal to twice the a priori probability assigned to the null. These and other results are an attempt to clarify the issue of typicality: how often the Bayes-classical conflict will arise and in what magnitude.

Noninformative Bayesian testing and neutral Bayes factors

In contrast with usual point estimation settings, there is no well-defined notion of noninformative Bayesian, procedures for tests of two-sided hypotheses. In this paper, we examine the behavior of the answers associated with the (point estimation) Jeffreys priors and show that, in some cases, they coincide numerically with the classical p-values. This partial equivalence between noninformative and classical answers is also exhibited with neutral Bayes factors. This new evaluation, which determines the cut-off point between acceptance and rejection of the null hypothesis, actually leads to a procedure similar to the (so-called) Jeffreys answere and thus supports the use of a noninformative improper prior. This interpretation thus brings a partial rehabilitation of some p-values as noninformative answers.

Understanding Unit Rooters: A Helicopter Tour

skewed asymptotically, while the likelihood and hence the posterior pdf remains symmetric. We show that no single prior can rationalize treating p-values as probabilities in these models, and we display examples of the sample-dependent priors that would do so. We argue that these results imply at a minimum that the usual test statistics and covariance matrices for autoregressions, which characterize the likelihood shape in dynamic models just as in static regression models, should be reported without any corrections for the special unit root distribution theory, even if the corrected classical p-values are reported as well.

Testing Precise Hypotheses

Testing of precise (point or small interval) hypotheses is reviewed, with special emphasis placed on exploring the dramatic conflict between conditional measures (Bayes factors and posterior probabilities) and the classical P-value (or observed significance level). This conflict is highlighted by finding lower bounds on the conditional measures over wide classes of priors, in normal and binomial situations, lower bounds, which are much larger than the P-value; this leads to the recommendation of several alternatives to P-values. Results are also given concerning the validity of approximating an interval null by a point null. The overall discussion features critical examination of issues such as the probability of objective testing and the possibility of testing from confidence sets.