Bonferroni Procedure Research Articles

A Closer Look at Correction for False Discovery Bias When Making Multiple Comparisons

A recent article in this journal by Tabak (2006) highlighted a potentially serious source of bias that can arise when multiple statistical tests of a damages theory are performed and even one test rejecting the null hypothesis is regarded as supporting the damages theory. Repeated testing will eventually produce a “false discovery,” that is, rejection of the null hypothesis in favor of the alternative hypothesis when the null hypothesis is true, which statisticians refer to as type I error. Consequently, performing multiple tests without adjusting the critical value can be problematical because it can lead to improperly accepting statistical evidence that apparently supports rejection of the null hypothesis as reliable when it is not. Tabak (2006) recommends making the Sidak (1968, 1971) multiple-comparison adjustment to the standard statistical t-test to correct for the false-discovery bias inherent in multiple-comparison testing. In particular, he recommends making this adjustment when performing 10b-5 securities fraud event studies when more than one corrective disclosure date is involved. This article clarifies the circumstances in which a multiple-comparison adjustment is appropriate and explains why the correction is normally not needed in securities fraud event-study testing. More generally, I explain why it is not required when each of several tests is performed and its results are reported separately, as for example, where the objective is simply to test the statistical significance of the abnormal stock return on each day on which a new and distinct curative disclosure occurred. I show that the Sidak multiple-comparison adjustment is nearly as stringent as the classical Bonferroni procedure (Simes, 1986), which can increase the risk of type II error. This article discusses a more powerful alternative to the Sidak adjustment due to Benjamini and Hochberg (1993), which directly corrects for the false-discovery bias in multiple-comparison testing and reduces the risk of type II error. II. Application of Multiple-Comparison Adjustments to Securities Fraud Event Studies Multiple-comparison-false-discovery bias can arise when (a) several statistical tests are performed on subsets of the same larger data set in an effort to

Genome-wide association studies of rheumatoid arthritis data via multiple hypothesis testing methods for correlated tests

Genome-wide association studies often involve testing hundreds of thousands of single-nucleotide polymorphisms (SNPs). These tests may be highly correlated because of linkage disequilibrium among SNPs. Multiple testing correction ignoring the correlation among markers, as is done in the Bonferroni procedure, can cause loss of power. Several multiple testing adjustment methods accounting for correlations among tests have been developed and have shown improved power compared to the Bonferroni procedure. These methods include a Monte Carlo (MC) method and a method of computing p-values adjusted for correlated tests. The objective of this study is to apply these two multiple testing methods to genome-wide association study of the Genetic Analysis Workshop 16 rheumatoid arthritis data from the North American Rheumatoid Arthritis Consortium, to compare the performance of these two methods to the Bonferroni procedure in identifying susceptibility loci underlying rheumatoid arthritis, and to discuss the strengths and weaknesses of these methods. The results show that both the MC method and p-values adjusted for correlated tests method identified more significant SNPs, thus potentially have higher power than the corresponding Bonferroni methods using the same test statistics as in the MC method and p-values adjusted for correlated tests, respectively. Simulation studies demonstrate that the MC method may have slightly higher power than the p-values adjusted for correlated tests method.

MultiTest V.1.2, a program to binomially combine independent tests and performance comparison with other related methods on proportional data

BackgroundCombining multiple independent tests, when all test the same hypothesis and in the same direction, has been the subject of several approaches. Besides the inappropriate (in this case) Bonferroni procedure, the Fisher's method has been widely used, in particular in population genetics. This last method has nevertheless been challenged by the SGM (symmetry around the geometric mean) and Stouffer's Z-transformed methods that are less sensitive to asymmetry and deviations from uniformity of the distribution of the partial P-values. Performances of these different procedures were never compared on proportional data such as those currently used in population genetics.ResultsWe present new software that implements a more recent method, the generalised binomial procedure, which tests for the deviation of the observed proportion of P-values lying under a chosen threshold from the expected proportion of such P-values under the null hypothesis. The respective performances of all available procedures were evaluated using simulated data under the null hypothesis with standard P-values distribution (differentiation tests). All procedures more or less behaved consistently with ~5% significant tests at α = 0.05. Then, linkage disequilibrium tests with increasing signal strength (rate of clonal reproduction), known to generate highly non-standard P-value distributions are undertaken and finally real population genetics data are analysed. In these cases, all procedures appear, more or less equally, very conservative, though SGM seems slightly more conservative.ConclusionBased on our results and those discussed in the literature we conclude that the generalised binomial and Stouffer's Z procedures should be preferred and Z when the number of tests is very small. The more conservative SGM might still be appropriate for meta-analyses when a strong publication bias in favour of significant results is expected to inflate type 2 error.

A note on adaptive Bonferroni and Holm procedures under dependence

SUMMARY Hochberg & Benjamini (1990) first presented adaptive procedures for controlling familywise error rate. However, until now, it has not been proved that these procedures control the familywise error rate. We introduce a simplified version of Hochberg & Benjamini’s adaptive Bonferroni and Holm procedures. Assuming a conditional dependence model, we prove that the former procedure controls the familywise error rate in finite samples while the latter controls it approximately.

TESTS FOR NONLINEAR COINTEGRATION

This paper develops tests for the null hypothesis of cointegration in the nonlinear regression model with I(1) variables. The test statistics we use in this paper are Kwiatkowski, Phillips, Schmidt, and Shin’s (1992; KPSS hereafter) tests for the null of stationarity, though using other kinds of tests is also possible. The tests are shown to depend on the limiting distributions of the estimators and parameters of the nonlinear model when they use full-sample residuals from the nonlinear least squares and nonlinear leads-and-lags regressions. This feature makes it difficult to use them in practice. As a remedy, this paper develops tests using subsamples of the regression residuals. For these tests, first, the nonlinear least squares and nonlinear leads-and-lags regressions are run and residuals are calculated. Second, the KPSS tests are applied using subresiduals of size b. As long as b/T → 0 as T → ∞, where T is the sample size, the tests using the subresiduals have limiting distributions that are not affected by the limiting distributions of the full-sample estimators and the parameters of the model. Third, the Bonferroni procedure is used for a selected number of the subresidual-based tests. Monte Carlo simulation shows that the tests work reasonably well in finite samples for polynomial and smooth transition regression models when the block size is chosen by the minimum volatility rule. In particular, the subresidual-based tests using the leads-and-lags regression residuals appear to be promising for empirical work. An empirical example studying the U.S. money demand equation illustrates the use of the tests.

Unconditional Exact Tests for Dichotomous Data in the Comparison of Two Treatments with One Control Group

In this article I discuss four procedures that compare each of two treatments against the control for dichotomous data while maintaining the familywise type I error rate at a level a. The four procedures are two unconditional exact tests, the Hochberg procedure, and the Bonferroni procedure, and they are compared in terms of exact power. After that, the numerical results of this article are also compared with those of Kang and Choi (Drug Information Journal 2008; 42:337–347). The best procedures are recommended depending on the values of the parameters under the alternative hypothesis.

Correction to Wu, L., Chuang, A., & Chen, P. (2008). Motivation for using search engines: A two factor model. Journal of American Society for Information Science and Technology, 59(11), 1829-1840.

Correction to Wu, L., Chuang, A., & Chen, P. (2008). Motivation for using search engines: A two factor model. Journal of American Society for Information Science and Technology, 59(11), 1829-1840.

CT Assessment of Woodworkers’ Nasal Adenocarcinomas Confirms the Origin in the Olfactory Cleft

Endoscopic endonasal surgery let us observe that woodworkers' nasal adenocarcinomas originate in the olfactory cleft. Our aim was the identification of CT imaging features that corroborate the olfactory cleft as the site of origin for woodworkers' adenocarcinoma. We designed a retrospective study to compare CT scans of 27 unilateral olfactory cleft adenocarcinomas with 30 cases of nasosinusal polyposis (NSP) and 33 healthy sinus controls. Enlargement of the olfactory cleft, lateralization of the ethmoidal turbinate wall, and contralateral bulging of the nasal septum were measured on coronal scans passing through crista galli and posterior half of both ocular globes. Comparisons have been performed by using analysis of variance and the Bonferroni procedure. The nasal septum was significantly bulging across the midline in adenocarcinoma (4.6 +/- 3 mm; range, -0.1-13.7 mm) compared with NSP (0.7 +/- 1 mm; range, -2.1-2.3 mm) or healthy sinus controls (0.5 +/- 1 mm; range, -1.2-2 mm) (P < .001). The olfactory cleft was significantly wider in adenocarcinoma (15.1 +/- 4.5 mm; range, 8.6-25.7 mm) than in NSP (3.6 +/- 0.4 mm; range, 2.8-4.6 mm) or healthy sinus controls (3.3 +/- 0.7 mm; range, 1.4-4.6 mm). The ethmoidal labyrinth width was significantly smaller on the pathologic side in adenocarcinoma (7.2 +/- 2.7 mm; range, 3.2-14.2 mm) than in the control groups (P < .001). Whereas the angle between the conchal lamina and vertical midline was close to zero degrees in NSP (0.03 +/- 2.25 degrees ; range, -5 degrees -3 degrees ) and healthy sinus controls (0.45 +/- 2.13 degrees , range, -5 degrees -5 degrees ), it reached 39.76 +/- 13.83 degrees (P < .001) in adenocarcinoma. Radiologists should suspect nasal adenocarcinoma on sinus CT scans showing a unilateral expanding opacity of the olfactory cavity.

Pilot evaluation of a walking school bus program in a low-income, urban community

BackgroundTo evaluate the impact of a walking school bus (WSB) program on student transport in a low-income, urban neighborhood.MethodsThe design was a controlled, quasi-experimental trial with consecutive cross-sectional assessments. The setting was three urban, socioeconomically disadvantaged, public elementary schools (1 intervention vs. 2 controls) in Seattle, Washington, USA. Participants were ethnically diverse students in kindergarten-5th grade (aged 5–11 years). The intervention was a WSB program consisting of a part-time WSB coordinator and parent volunteers. Students' method of transportation to school was assessed by a classroom survey at baseline and one-year follow-up. The Pearson Chi-squared test compared students transported to school at the intervention versus control schools at each time point. Due to multiple testing, we calculated adjusted p-values using the Ryan-Holm stepdown Bonferroni procedure. McNemar's test was used to examine the change from baseline to 12-month follow-up for walking versus all other forms of school transport at the intervention or control schools.ResultsAt baseline, the proportions of students (n = 653) walking to the intervention (20% +/- 2%) or control schools (15% +/- 2%) did not differ (p = 0.39). At 12-month follow up, higher proportions of students (n = 643, p = 0.001)) walked to the intervention (25% +/- 2%) versus the control schools (7% +/- 1%). No significant changes were noted in the proportion of students riding in a car or taking the school bus at baseline or 12-month follow up (all p > 0.05). Comparing baseline to 12-month follow up, the numbers of students who walked to the intervention school increased while the numbers of students who used the other forms of transport did not change (p < 0.0001). In contrast, the numbers of students who walked to the control schools decreased while the numbers of students who used the other forms of transport did not change (p < 0.0001).ConclusionA WSB program is a promising intervention among urban, low-income elementary school students that may promote favorable changes toward active transport to school.Trial RegistrationClinicalTrials.gov NCT00402701

A modified forward multiple regression in high‐density genome‐wide association studies for complex traits

Genome-wide association studies (GWAS) have been widely used to identify genetic effects on complex diseases or traits. Most currently used methods are based on separate single-nucleotide polymorphism (SNP) analyses. Because this approach requires correction for multiple testing to avoid excessive false-positive results, it suffers from reduced power to detect weak genetic effects under limited sample size. To increase the power to detect multiple weak genetic factors and reduce false-positive results caused by multiple tests and dependence among test statistics, a modified forward multiple regression (MFMR) approach is proposed. Simulation studies show that MFMR has higher power than the Bonferroni and false discovery rate procedures for detecting moderate and weak genetic effects, and MFMR retains an acceptable-false positive rate even if causal SNPs are correlated with many SNPs due to population stratification or other unknown reasons.

Weighted multiple hypothesis testing procedures.

Multiple hypothesis testing is commonly used in genome research such as genome-wide studies and gene expression data analysis (Lin, 2005). The widely used Bonferroni procedure controls the family-wise error rate (FWER) for multiple hypothesis testing, but has limited statistical power as the number of hypotheses tested increases. The power of multiple testing procedures can be increased by using weighted p-values (Genovese et al., 2006). The weights for the p-values can be estimated by using certain prior information. Wasserman and Roeder (2006) described a weighted Bonferroni procedure, which incorporates weighted p-values into the Bonferroni procedure, and Rubin et al. (2006) and Wasserman and Roeder (2006) estimated the optimal weights that maximize the power of the weighted Bonferroni procedure under the assumption that the means of the test statistics in the multiple testing are known (these weights are called optimal Bonferroni weights). This weighted Bonferroni procedure controls FWER and can have higher power than the Bonferroni procedure, especially when the optimal Bonferroni weights are used. To further improve the power of the weighted Bonferroni procedure, first we propose a weighted Sidák procedure that incorporates weighted p-values into the Sidák procedure, and then we estimate the optimal weights that maximize the average power of the weighted Sidák procedure under the assumption that the means of the test statistics in the multiple testing are known (these weights are called optimal Sidák weights). This weighted Sidák procedure can have higher power than the weighted Bonferroni procedure. Second, we develop a generalized sequential (GS) Sidák procedure that incorporates weighted p-values into the sequential Sidák procedure (Scherrer, 1984). This GS idák procedure is an extension of and has higher power than the GS Bonferroni procedure of Holm (1979). Finally, under the assumption that the means of the test statistics in the multiple testing are known, we incorporate the optimal Sidák weights and the optimal Bonferroni weights into the GS Sidák procedure and the GS Bonferroni procedure, respectively. Theoretical proof and/or simulation studies show that the GS Sidák procedure can have higher power than the GS Bonferroni procedure when their corresponding optimal weights are used, and that both of these GS procedures can have much higher power than the weighted Sidák and the weighted Bonferroni procedures. All proposed procedures control the FWER well and are useful when prior information is available to estimate the weights.

Association Between Perceived Stress and Plasma B-Type Natriuretic Peptide Concentrations

Many patients with heart disease continue to have cardiac events despite receiving optimal treatments for traditional risk factors. Consequently, non-traditional risk factors for heart disease, such as perceived stress, have attracted attention. Associations between perceived stress and plasma B-type natriuretic peptide (BNP) were explored, while controlling for traditional heart disease risk factors. This cross-sectional study examined 360 male and 446 female (age, >40 years) residents of a rural Japanese community who received annual health checkups in 2006. A lifestyle questionnaire was used to obtain information regarding perceived stress and medical history, and routine anthropometric and blood pressure measurements and a laboratory assessment of cardiovascular risk factors, including plasma BNP concentrations and an electrocardiogram, were done. After adjusting for traditional heart disease risk factors, multiple regression analysis showed that perceived stress was associated with BNP concentrations, particularly in women (F=6.12, P=0.026). In addition, multiple tests using Bonferroni's procedure showed that BNP concentrations decreased with perceived stress level in men and women. Similar trends were observed in the sub-analyses of subjects with and without known heart disease. Perceived stress in our study was negatively associated with plasma BNP concentrations, independently of traditional heart disease risk factors.

Screening and Replication using the Same Data Set: Testing Strategies for Family-Based Studies in which All Probands Are Affected

For genome-wide association studies in family-based designs, we propose a powerful two-stage testing strategy that can be applied in situations in which parent-offspring trio data are available and all offspring are affected with the trait or disease under study. In the first step of the testing strategy, we construct estimators of genetic effect size in the completely ascertained sample of affected offspring and their parents that are statistically independent of the family-based association/transmission disequilibrium tests (FBATs/TDTs) that are calculated in the second step of the testing strategy. For each marker, the genetic effect is estimated (without requiring an estimate of the SNP allele frequency) and the conditional power of the corresponding FBAT/TDT is computed. Based on the power estimates, a weighted Bonferroni procedure assigns an individually adjusted significance level to each SNP. In the second stage, the SNPs are tested with the FBAT/TDT statistic at the individually adjusted significance levels. Using simulation studies for scenarios with up to 1,000,000 SNPs, varying allele frequencies and genetic effect sizes, the power of the strategy is compared with standard methodology (e.g., FBATs/TDTs with Bonferroni correction). In all considered situations, the proposed testing strategy demonstrates substantial power increases over the standard approach, even when the true genetic model is unknown and must be selected based on the conditional power estimates. The practical relevance of our methodology is illustrated by an application to a genome-wide association study for childhood asthma, in which we detect two markers meeting genome-wide significance that would not have been detected using standard methodology.

Compatible simultaneous lower confidence bounds for the Holm procedure and other Bonferroni‐based closed tests

We consider the problem of simultaneously testing multiple one-sided null hypotheses. Single-step procedures, such as the Bonferroni test, are characterized by the fact that the rejection or non-rejection of a null hypothesis does not take the decision for any other hypothesis into account. For stepwise test procedures, such as the Holm procedure, the rejection or non-rejection of a null hypothesis may depend on the decision of other hypotheses. It is well known that stepwise test procedures are by construction more powerful than their single-step counterparts. This power advantage, however, comes only at the cost of increased difficulties in constructing compatible simultaneous confidence intervals for the parameters of interest. For example, such simultaneous confidence intervals are easily obtained for the Bonferroni method, but surprisingly hard to derive for the Holm procedure. In this paper, we discuss the inherent problems and show that ad hoc solutions used in practice typically do not control the pre-specified simultaneous confidence level. Instead, we derive simultaneous confidence intervals that are compatible with a certain class of closed test procedures using weighted Bonferroni tests for each intersection hypothesis. The class of multiple test procedures covered in this paper includes gatekeeping procedures based on Bonferroni adjustments, fixed sequence procedures, the simple weighted or unweighted Bonferroni procedure by Holm and the fallback procedure. We illustrate the results with a numerical example.

Short‐ and Long‐Term Effects of Training on Dental Hygiene Faculty Members’ Capacity to Write SOAP Notes

Calibration among faculty is challenging to achieve and maintain. In this study, calibration refers to the training process by which standardization of chart documentation in a SOAP note format was achieved. In the SOAP format, chart entries by health care providers are written in the following categories: Subjective data, Objective data, Assessment, and Plans. The primary training "effect" or outcome that was measured in this study was the capacity of faculty members to write a SOAP note that adhered to prescribed standards for chart documentation. This study was conducted to assess the short-term effects of training and determine whether faculty members' capacity to write appropriately constructed SOAP notes could be sustained for one year. Eight dental hygiene faculty members at the University of Minnesota participated in a pre-training assessment in which they prepared a SOAP note based on a patient case, completed a training session on writing SOAP notes, and completed a post-training test shortly after training that also consisted of writing a SOAP note based on a patient's case. One year later, a follow-up test, similar to the pre- and post-tests, was conducted. Each component of the SOAP note was compared and scored against a gold standard benchmark score of 29 that represented the number of items that should have been included in an ideal SOAP note in the estimation of the investigators, based on chart documentation guidelines of the University of Minnesota Dental Hygiene Division. The mean score for the pre-test was 18.25 (SD=2.82), which represented 63 percent of the benchmark gold standard score of 29. The post-test mean score immediately after training was 24.63 (SD=2.13; 84.9 percent of the benchmark score), and the one-year follow-up mean score was 22.75 (SD=1.83; 78.4 percent of the gold standard benchmark). From the pre-test to the post-test administered in close approximation to the SOAP note training, faculty members' approximation of the gold standard benchmark increased by 35 percent, or 6.28 points, and from the post-test to the follow-up test one year subsequently, approximation of the benchmark score decreased by approximately 1 percent or 1.88 points. Friedman's test indicated that the differences in mean scores for the pre-test, post-test, and follow-up test were significant. The Sign test was used for post hoc tests; alpha was adjusted using Bonferroni's procedure. Conclusions support a hypothesis that faculty capacity to write a SOAP note that adheres to standards can be increased through training and that the effects can be maintained over a period of approximately one year.

On the Applications of Fisher’s Least Significant Difference (LSD) Procedure in Three-Arm Clinical Trials with Survival Endpoints

Fisher’s (protected) least significant difference (LSD) procedure has been suggested in the design and analysis of multiarm clinical trials with survival endpoints. In this article, the power of this procedure is evaluated and compared with that of Bonferroni and Hochberg procedures by Monte Carlo simulations under three scenarios of alternative hypothesis encountered in three arm clinical trials with survival endpoint. The approach of sample size calculation based on the power of a global test in the first step of Fisher’s LSD procedure is also evaluated and contrasted with that based on Bonferroni adjustment. It is shown that Bonferroni and Hochberg procedures are more powerful than Fisher’s LSD procedure when two specific pair-wise comparisons are of interest, while Fisher’s LSD procedure is the most powerful if all three pairwise comparisons are performed. It is recommended that the formula of Makuch and Simon be used in the sample size calculations for all three types of alternative hypothesis considered in this article. However, the Hochberg procedure should be used in the analysis if only two specific pairwise comparisons are of interest and the Fisher’s LSD procedure should be used in the analysis when objective of the trial includes all of three pairwise comparisons.

Statistical Approaches to Multiplicity Issues in Clinical Trials

This paper discusses various multiplicity issues arisen in clinical trials and possible statistical approaches to these issues. We especially stress the importance of the closed testing procedures (CTPs) in the setting of clinical trials: They include various modified Bonferroni procedures, e.g., step-down Dunnett procedure, hierarchical procedure, and Williams test. Moreover they can be even applied to adaptive designs in clinical trials. We illustrate the basic CTP procedures in detail.

Multiple comparisons of several heteroscedastic multivariate populations

This paper presents several statistics appearing in multiple comparisons of heteroscedastic multivariate populations. Due to the very slow convergence of these statistics to their limiting distributions, the large sample Bonferroni or DL-based procedures reveal poor coverage probabilities even in the normal case. Thus, the second-order asymptotic expansions with estimated cumulants are applied to improve their coverage probabilities. A large simulation study illustrates the performance of the second-order corrected procedures.

So Many Correlated Tests, So Little Time! Rapid Adjustment of P Values for Multiple Correlated Tests

Contemporary genetic association studies may test hundreds of thousands of genetic variants for association, often with multiple binary and continuous traits or under more than one model of inheritance. Many of these association tests may be correlated with one another because of linkage disequilibrium between nearby markers and correlation between traits and models. Permutation tests and simulation-based methods are often employed to adjust groups of correlated tests for multiple testing, since conventional methods such as Bonferroni correction are overly conservative when tests are correlated. We present here a method of computing P values adjusted for correlated tests (P(ACT)) that attains the accuracy of permutation or simulation-based tests in much less computation time, and we show that our method applies to many common association tests that are based on multiple traits, markers, and genetic models. Simulation demonstrates that P(ACT) attains the power of permutation testing and provides a valid adjustment for hundreds of correlated association tests. In data analyzed as part of the Finland-United States Investigation of NIDDM Genetics (FUSION) study, we observe a near one-to-one relationship (r(2)>.999) between P(ACT) and the corresponding permutation-based P values, achieving the same precision as permutation testing but thousands of times faster.

A New Procedure to Monitor the Mean of a Quality Characteristic

The Shewhart, Bonferroni-adjustment, and analysis of means (ANOM) control charts are typically applied to monitor the mean of a quality characteristic. The Shewhart and Bonferroni procedure are utilized to recognize special causes in production process, where the control limits are constructed by assuming normal distribution for known parameters (mean and standard deviation), and approximately normal distribution regarding to unknown parameters. The ANOM method is an alternative to the analysis of variance method. It can be used to establish the mean control charts by applying equicorrelated multivariate non central t distribution. In this article, we establish new control charts, in phases I and II monitoring, based on normal and t distributions having as a cause a known (or unknown) parameter (standard deviation). Our proposed methods are at least as effective as the classical Shewhart methods and have some advantages.