Standard Bonferroni Research Articles

Mining Statistically Significant Patterns with High Utility

Statistically significant pattern mining (SSPM) is to mine patterns with significance based on hypothesis test. Under the constraint of statistical significance, our study aims to introduce a new preference relation into high utility patterns and to discover high utility and significant patterns (HUSPs) from transaction datasets, which has never been considered in existing SSPM problems. Our approach can be divided into two parts, HUSP-Mining and HUSP-Test. HUSP-Mining looks for HUSP candidates and HUSP-Test tests their significance. HUSP-Mining is not outputting all high utility itemsets (HUIs) as HUSP candidates; it is established based on candidate length and testable support requirements which can remove many insignificant HUIs early in the mining process; compared with the traditional HUIs mining algorithm, it can get candidates in a short time without losing the real HUSPs. HUSP-Test is to draw significant patterns from the results of HUSP-Mining based on Fisher’s test. We propose an iterative multiple testing procedure, which can alternately and efficiently reject a hypothesis and safely ignore the hypotheses that have less utility than the rejected hypothesis. HUSP-Test controls Family-wise Error Rate (FWER) under a user-defined threshold by correcting the test level which can find more HUSPs than standard Bonferroni’s control. Substantial experiments on real datasets show that our algorithm can draw HUSPs efficiently from transaction datasets with strong mathematical guarantee.

Synergistic impacts of co‐occurring invasive grasses cause persistent effects in the soil‐plant system after selective removal

AbstractHuman influence on the environment is so extensive that virtually all ecosystems on the planet are now affected by biological invasions. And, often, ecosystems are invaded by multiple co‐occurring non‐native species. Hence, it is important to understand the impacts these invasions are producing on biodiversity and ecosystem processes.Here, we present results of a 2‐year long field experiment where we tested the effects of co‐occurring invasive C4African grasses in a Cerrado area in central Brazil. We compared plant and arthropod communities, plant biomass, and soil nitrogen dynamics and soil chemical characteristics across five experimental treatments:Urochloa decumbensremoval;Melinis minutifloraremoval; bothU.decumbensandM.minutifloraremoval;U.decumbensandM.minutiflorainvaded plots; and uninvaded Cerrado. We hypothesized that selective removal of invasive grasses would have distinct effects on the native ecosystem structure and functioning. We expected that each invasive grass would produce a different type of impact on the native ecosystem and that their impacts would be synergistic when co‐occurring.Removal ofM.minutifloradoubled native plant diversity and biomass when compared to invaded plots, whereas removal ofU.decumbensdid not alter these parameters. Cerrado plots had four times more plant species than plots cleared of invasives. Removal of invasive grasses did not affect the species richness or community composition of soil epigeal fauna. Cerrado soils had lower fertility, organic matter content and pH than invaded soils. The effects were generally higher when both invasive grasses were removed, suggesting impacts were synergistic, butM.minutiflorahad greater effects on plants and soils thanU.decumbens. Both invasive species produced negative impacts, but a single species was the main driver. We also detected persistent effects of the invasive grass species on the ecosystem after 2 years of removal.We conclude that invasive species of the same functional group have similar types of effects in native ecosystems, but the magnitude of impact was largely dependent on invasive species biomass and cover. Where multiple invasive species are present, research and management of invaded ecosystems should tackle the interacting effects of co‐occurring invaders.

Rational Approaches to Correcting for Multiple Tests

The logic of the Bonferroni correction for multiple tests, or family-wise error, is to set the criterion to reduce the expected number of erroneous false positives, or Type I errors, below 1. This is a very stringent criterion for false positives in cases where the test may be applied millions of times, and will necessarily introduce a large proportion of false negatives (missed positives, or Type II errors). A proposed solution to this problem is to adjust the criterion for False Discovery Rate (Benjamini & Hochberg, 1995), which allows the number of false positives to increase proportionally to the number of true positives, though remaining at a small proportion, dramatically reducing the number of false negatives. This approach may be conceptualized as working with a relaxed confidence level that any one test is a true rather than a false positive, bringing the criterion more into line with our societal assessment of the validity of statements in general, and even in science, as having less than 100% certainty. The analytic strategy to the assessment of statistical significance provides a more intuitive approach to the identification of sparse signals in large datasets than the standard Bonferroni approach to correction for multiple tests.

Weighting sequence variants based on their annotation increases power of whole-genome association studies.

The consensus approach to genome-wide association studies (GWAS) has been to assign equal prior probability of association to all sequence variants tested. However, some sequence variants, such as loss-of-function and missense variants, are more likely than others to affect protein function and are therefore more likely to be causative. Using data from whole-genome sequencing of 2,636 Icelanders and the association results for 96 quantitative and 123 binary phenotypes, we estimated the enrichment of association signals by sequence annotation. We propose a weighted Bonferroni adjustment that controls for the family-wise error rate (FWER), using as weights the enrichment of sequence annotations among association signals. We show that this weighted adjustment increases the power to detect association over the standard Bonferroni correction. We use the enrichment of associations by sequence annotation we have estimated in Iceland to derive significance thresholds for other populations with different numbers and combinations of sequence variants.

A graphical weighted power improving multiplicity correction approach for SNP selections.

Controlling for the multiplicity effect is an essential part of determining statistical significance in large-scale single-locus association genome scans on Single Nucleotide Polymorphisms (SNPs). Bonferroni adjustment is a commonly used approach due to its simplicity, but is conservative and has low power for large-scale tests. The permutation test, which is a powerful and popular tool, is computationally expensive and may mislead in the presence of family structure. We propose a computationally efficient and powerful multiple testing correction approach for Linkage Disequilibrium (LD) based Quantitative Trait Loci (QTL) mapping on the basis of graphical weighted-Bonferroni methods. The proposed multiplicity adjustment method synthesizes weighted Bonferroni-based closed testing procedures into a powerful and versatile graphical approach. By tailoring different priorities for the two hypothesis tests involved in LD based QTL mapping, we are able to increase power and maintain computational efficiency and conceptual simplicity. The proposed approach enables strong control of the familywise error rate (FWER). The performance of the proposed approach as compared to the standard Bonferroni correction is illustrated by simulation and real data. We observe a consistent and moderate increase in power under all simulated circumstances, among different sample sizes, heritabilities, and number of SNPs. We also applied the proposed method to a real outbred mouse HDL cholesterol QTL mapping project where we detected the significant QTLs that were highlighted in the literature, while still ensuring strong control of the FWER.

Holm multiple correction for large-scale gene-shape association mapping.

BackgroundLinkage Disequilibrium (LD) is a powerful approach for the identification and characterization of morphological shape, which usually involves multiple genetic markers. However, multiple testing corrections substantially reduce the power of the associated tests. In addition, the principle component analysis (PCA), used to quantify the shape variations into several principal phenotypes, further increases the number of tests. As a result, a powerful multiple testing correction for simultaneous large-scale gene-shape association tests is an essential part of determining statistical significance. Bonferroni adjustments and permutation tests are the most popular approaches to correcting for multiple tests within LD based Quantitative Trait Loci (QTL) models. However, permutations are extremely computationally expensive and may mislead in the presence of family structure. The Bonferroni correction, though simple and fast, is conservative and has low power for large-scale testing.ResultsWe propose a new multiple testing approach, constructed by combining an Intersection Union Test (IUT) with the Holm correction, which strongly controls the family-wise error rate (FWER) without any additional assumptions on the joint distribution of the test statistics or dependence structure of the markers. The power improvement for the Holm correction, as compared to the standard Bonferroni correction, is examined through a simulation study. A consistent and moderate increase in power is found under the majority of simulated circumstances, including various sample sizes, Heritabilities, and numbers of markers. The power gains are further demonstrated on real leaf shape data from a natural population of poplar, Populus szechuanica var tietica, where more significant QTL associated with morphological shape are detected than under the previously applied Bonferroni adjustment.ConclusionThe Holm correction is a valid and powerful method for assessing gene-shape association involving multiple markers, which not only controls the FWER in the strong sense but also improves statistical power.

Bonferroni‐based correction factor for multiple, correlated endpoints

Multiple testing and its impact on the type I and type II error rates are frequently discussed in the statistical and biomedical literature. The Bonferroni adjustment is one of the most widely used approaches, yet it suffers from poor statistical performance when there are correlated test statistics. For example, it is criticized to be too conservative. Nonetheless, part of the strong appeal of the Bonferroni approach is the straightforward implementation and relatively intuitive explanation. In this manuscript, a novel adaptation to the traditional Bonferroni approach that accounts for correlated data is proposed. A simple correction factor based on intraclass correlation is applied to the standard Bonferroni method to overcome the shortcomings of the standard Bonferroni adjustment yet maintains its advantages. The method is motivated by an early phase clinical trial examining the effect of a study medication on marijuana craving, which is commonly quantified into four correlated constructs. A detailed simulation study demonstrated that the proposed approach is statistically sound and appropriate for a wide range of common settings.

An Assessment of the Effectiveness of Multiple Hypothesis Testing for Geographical Anomaly Detection

The practice of multiple significance testing is reviewed, and an alternative to the frequently used Bonferroni correction is considered. Rather than controlling the family-wise error rate (FWER)—the probability of a false positive in any of the significance tests—this alternative due to Benjamini and Hochberg controls the false discovery rate (FDR). This is the proportion of tests reporting a significant result that are actually ‘false alarms’. The methods (and some variants) are demonstrated on a procedure to detect clusters of full-time unpaid carers based on UK census data, and are also assessed using simulation. Simulation results show that the FDR-based corrections are typically more powerful than FWER-based ones, and also that the degree of conservatism in FWER-based procedures is quite extreme, to the extent that the standard Bonferroni procedure intended to constrain the FWER to be below 0.05 actually has a FWER of around 6 × 10−5. We conclude that in situations where one is scanning for anomalies, the extreme conservatism of FWER-based approaches results in a lack of power, and that FDR-based approaches are more appropriate.

An open-source software program for performing Bonferroni and related corrections for multiple comparisons

Increased type I error resulting from multiple statistical comparisons remains a common problem in the scientific literature. This may result in the reporting and promulgation of spurious findings. One approach to this problem is to correct groups of P-values for “family-wide significance” using a Bonferroni correction or the less conservative Bonferroni-Holm correction or to correct for the “false discovery rate” with a Benjamini-Hochberg correction. Although several solutions are available for performing this correction through commercially available software there are no widely available easy to use open source programs to perform these calculations. In this paper we present an open source program written in Python 3.2 that performs calculations for standard Bonferroni, Bonferroni-Holm and Benjamini-Hochberg corrections.

Making Informed Choices about Microarray Data Analysis

Making Informed Choices about Microarray Data Analysis

Evaluating Statistical Significance in Two-Stage Genomewide Association Studies

Genomewide association studies are being conducted to unravel the genetic etiology of complex human diseases. Because of cost constraints, these studies typically employ a two-stage design, under which a large panel of markers is examined in a subsample of subjects, and the most-promising markers are then examined in all subjects. This report describes a simple and efficient method to evaluate statistical significance for such genome studies. The proposed method, which properly accounts for the correlated nature of polymorphism data, provides accurate control of the overall false-positive rate and is substantially more powerful than the standard Bonferroni correction, especially when the markers are in strong linkage disequilibrium.

The Fallback Procedure for Evaluating a Single Family of Hypotheses

ABSTRACT In testing multiple hypotheses, control of the familywise error rate is often considered. We develop a procedure called the “fallback procedure” to control the familywise error rate when multiple primary hypotheses are tested. With the fallback procedure, the Type I error rate (α) is partitioned among the various hypotheses of interest. Unlike the standard Bonferroni adjustment, however, testing hypotheses proceeds in an order determined a priori. As long as hypotheses are rejected, the Type I error rate can be accumulated, making tests of later hypotheses more powerful than under the Bonferroni procedure. Unlike the fixed sequence test, the fallback test allows consideration of all hypotheses even if one or more hypotheses are not rejected early in the process, thereby avoiding a common concern about the fixed sequence procedure. We develop properties of the fallback procedure, including control of the familywise error rate for an arbitrary number of hypotheses via illustrating the procedure as a closed testing procedure, as well as making the test more powerful via alpha exhaustion. We compare it to other procedures for controlling familywise error rates, finding that the fallback procedure is a viable alternative to the fixed sequence procedure when there is some doubt about the power for the first hypothesis. These results expand on the previously developed properties of the fallback procedure (Wiens, 2003). Several examples are discussed to illustrate the relative advantages of the fallback procedure.

Sample Size Calculation for Simulation-Based Multiple-Testing Procedures

ABSTRACT In this article, we present a simple method to calculate sample size and power for a simulation-based multiple testing procedure which gives a sharper critical value than the standard Bonferroni method. The method is especially useful when several highly correlated test statistics are involved in a multiple-testing procedure. The formula for sample size calculation will be useful in designing clinical trials with multiple endpoints or correlated outcomes. We illustrate our method with a quality-of-life study for patients with early stage prostate cancer. Our method can also be used for comparing multiple independent groups.

Judicious Use of Multiple Hypothesis Tests

Abstract: When analyzing a table of statistical results, one must first decide whether adjustment of significance levels is appropriate. If the main goal is hypothesis generation or initial screening for potential conservation problems, then it may be appropriate to use the standard comparisonwise significance level to avoid Type II errors (not detecting real differences or trends). If the main goal is rigorous testing of a hypothesis, however, then an adjustment for multiple tests is needed. To control the familywise Type I error rate (the probability of rejecting at least one true null hypothesis), sequential modifications of the standard Bonferroni method, such as Holm's method, will provide more statistical power than the standard Bonferroni method. Additional power may be achieved through procedures that control the false discovery rate (FDR) (the expected proportion of false positives among tests found to be significant). Holm's sequential Bonferroni method and two FDR‐controlling procedures were applied to the results of multiple‐regression analyses of the relationship between habitat variables and the abundance of 25 species of forest birds in Japan, and the FDR‐controlling procedures provided considerably greater statistical power.

Sample size calculation for multiple testing in microarray data analysis

Microarray technology is rapidly emerging for genome-wide screening of differentially expressed genes between clinical subtypes or different conditions of human diseases. Traditional statistical testing approaches, such as the two-sample t-test or Wilcoxon test, are frequently used for evaluating statistical significance of informative expressions but require adjustment for large-scale multiplicity. Due to its simplicity, Bonferroni adjustment has been widely used to circumvent this problem. It is well known, however, that the standard Bonferroni test is often very conservative. In the present paper, we compare three multiple testing procedures in the microarray context: the original Bonferroni method, a Bonferroni-type improved single-step method and a step-down method. The latter two methods are based on nonparametric resampling, by which the null distribution can be derived with the dependency structure among gene expressions preserved and the family-wise error rate accurately controlled at the desired level. We also present a sample size calculation method for designing microarray studies. Through simulations and data analyses, we find that the proposed methods for testing and sample size calculation are computationally fast and control error and power precisely.

A farewell to Bonferroni: the problems of low statistical power and publication bias

A farewell to Bonferroni: the problems of low statistical power and publication bias

Exact tests for contemporaneous correlation of disturbances in seemingly unrelated regressions

This paper proposes finite-sample procedures for testing the SURE specification in multi-equation regression models, i.e. whether the disturbances in different equations are contemporaneously uncorrelated or not. We apply the technique of Monte Carlo (MC) tests, see Dwass and Barnard, respectively (Ann. Math. Statist. 28 (1957) 181; J.R. Statist. Soc. Ser. B 25 (1963) 294) to obtain exact tests based on standard LR and LM zero correlation tests. We also suggest a MC quasi-LR (QLR) test based on feasible generalized least squares (FGLS). We show that the latter statistics are pivotal under the null, which provides the justification for applying MC tests. Furthermore, we extend the exact independence test proposed by Harvey and Phillips (Bull. Econ. Res. 34 (2) (1982) 79) to the multi-equation framework. Specifically, we introduce several induced tests based on a set of simultaneous Harvey/Phillips-type tests and suggest a simulation-based solution to the associated combination problem. The properties of the proposed tests are studied in a Monte Carlo experiment which shows that standard asymptotic tests exhibit important size distortions, while MC tests achieve complete size control and display good power. Moreover, MC-QLR tests performed best in terms of power, a result of interest from the point of view of simulation-based tests. The power of the MC induced tests improves appreciably in comparison to standard Bonferroni tests and in certain cases outperform the likelihood-based MC tests. The tests are applied to data used by Fischer (J. Monetary Econ. 32, 485) (1993) to analyze the macroeconomic determinants of growth.

Simple approximations for the maximal transmission/disequilibrium test with a multi-allelic marker.

Spielman et al. (1993) popularized the transmission/disequilibrium test (TDT) to test for linkage between disease and marker loci that show a population association. Several authors have proposed extensions to the TDT for multi-allelic markers. Many of these approaches exhibit a 'swamping' effect in which a marker with a strong effect is not detected by a global test that includes many markers with no effect. To avoid this effect, Schaid (1996) proposed using the maximum of the bi-allelic TDT statistics computed for each allele versus all others combined. The maximal TDT statistic, however, no longer follows a chi-square distribution. Here, a refinement to Bonferroni's correction for multiple testing provided by Worsley (1982) based on maximal spanning trees is applied to calculate accurate upper bounds for the type I error and p-values for the maximal TDT. In addition, an accurate lower Bonferroni bound is applied to calculate power. This approach does not require any simulation-based analysis and is less conservative than the standard Bonferroni correction. The bounds are given for both the exact probability calculations and for those based on the normal approximation. The results are assessed through simulations.

Per-Experiment Error Rates: The Hidden Costs of Several Multiple Comparison Procedures

For a number of common multiple comparison procedures controlling Type I error at .05 refers to what Ryan called the experimentwise error rate. This expression represents the chance of making at least one Type I error within a given experiment. This approach ignores, however, the potential detriments of multiple errors within a single experiment, that is, it fails to acknowledge what Ryan referred to as the error rate per experiment. The current study uses a computer simulation to evaluate the differences between experimentwise error rates and error rates per experiment for a variety of multiple comparison procedures. For pairwise comparisons, Newman-Keuls's and Tukey's tests are examined, as is Dunnett's test for comparisons with a control group and Scheffe's test for all possible post hoc comparisons. For planned contrasts, a standard Bonferroni and Shaffer's sequentially rejective Bonferroni are simulated using both a nonorthogonal and an orthogonal set.

Mild pediatric traumatic brain injury: adjusting significance levels for multiple comparisons.

The outcome following mild traumatic brain injury (TBI) is controversial. We addressed this topic in a study of neurobehavioural and 'real-world' functioning among 53 children with mild TBI and their matched controls, using statistical methods recently developed for multiple comparisons. Because the study involved calculation of 414 p-values, four methods of adjustment for multiple comparisons, including the Bonferroni method, were used to avoid 'false-positive' statistical significance. The additional three methods allowed greater insight into the data than provided by the standard Bonferroni adjustment. Results showed that at initial testing, three areas of 'real-world' functioning (eating, domestic and home/community living skills) had weak but statistically significant associations with mild injury. No other specific areas of neuro-behavioural or 'real-world' functioning had plausible associations with the injury either initially, at 1 year, or when changes over the year were considered. However, the adjustment for multiple comparisons provided additional results. There were statistical significant associations of the injury with the entire domain of neurobehavioural variables both initially and at 1 year. These results suggest that the injury affects a spectrum of neurobehavioural skills weakly, rather than a single area substantially. The gain from using multiple comparison methods is discussed.