False Null Hypotheses Research Articles

Tests of Independence for Ordinal Data Using Bootstrap

In a two-way cross table, the association between two ordinal variables can be assessed by different measures such as Goodman and Kruskal's gamma (y) and Kendall's tau-b (tb). When sample size is large, the independence hypothesis between the row and the column variables can be tested by the traditional asymptotic test (TAT). TAT, however, fails to work satisfactorily when sample size is small because the theoretical distribution of the test statistic may only hold asymptotically. In this study, two bootstrap-based tests (BBTs) are proposed for testing the independence hypothesis. Monte Carlo studies are used to compare the TAT with the BBTs at small to moderate sample sizes. The BBTs are superior to the TAT in two aspects: The control of Type I error rate by the BBTs is more accurate, and the BBTs are more powerful because they are more likely than the TAT to reject false null hypotheses.

Methods of correcting for multiple testing: operating characteristics.

We examine the operating characteristics of 17 methods for correcting p-values for multiple testing on synthetic data with known statistical properties. These methods are derived p-values only and not the raw data. With the test cases, we systematically varied the number of p-values, the proportion of false null hypotheses, the probability that a false null hypothesis would result in a p-value less than 5 per cent and the degree of correlation between p-values. We examined the effect of each of these factors on family-wise and false negative error rates and compared the false negative error rates of methods with an acceptable family-wise error. Only four methods were not bettered in this comparison. Unfortunately, however, a uniformly best method of those examined does not exist. A suggested strategy for examining corrections uses a succession of methods that are increasingly lax in family-wise error. A computer program for these corrections is available.

Empowering research: statistical power in general practice research.

Statistical power is a measure of the extent to which a study is capable of discerning differences or associations which exist within the population under investigation, and is of critical importance whenever a hypothesis is tested by statistics. Conventionally, studies should reach a power level of 0.8, such that four times out of five a false null hypothesis will be rejected by a study. Statistical power may most easily be increased by increasing sample size. We aimed to assess the level of statistical power of general practice research. A total of 1422 statistical tests in 85 quantitative original papers in the British Journal of General Practice were analysed for statistical power. The median power of tests analysed was 0.71, representing a slightly greater than two-thirds likelihood of rejecting false null hypotheses. Of 85 studies, 37 (44%) attained power of 0.8 or more. Ten studies had power of more than 0.99 suggesting 'over-powering'. Twenty-one of the papers surveyed (25%) had a likelihood of gaining significant results poorer than that obtained by tossing a coin when a null hypothesis is false. While achieving higher power than studies in similar surveys of other disciplines, the power of general practice research falls short of the 0.8 convention. Adequate power is essential so that effects which exist are not missed. Recommendations are made concerning power calculations prior to the start of research and reporting of results in journal articles.

Power Analysis in Covariance Structure Modeling Using GFI and AGFI

Extending earlier work by MacCallum, Browne, and Sugawara (1996), procedures are shown for conducting power analysis of tests of overall fit of covariance structure models when null and alternative levels of model fit are specified in terms of values of the GFI or AGFl fit indexes. Results show that for GFI-based power analyses, holding null and alternative values of GFI fixed, power decreases as degrees of freedom increase, which is a counter-intuitive and undesirable phenomenon indicating lower power for detecting false null hypotheses about simpler models. For AGFI-based analyses, power increases as degrees of freedom increase. However, for both indexes it is shown that it is problematic to establish consistently appropriate values for null and alternative hypotheses about model fit. Because of these problems, it is recommended that the RMSEA index is preferable as a basis for power analysis and model evaluation.

On the Assessment of the Performance of Multiple Test Procedures

AbstractBased on the concept of the multiple level of significance two criteria for assessing the performance of multiple tests are proposed: The simultaneous power is defined as the probability of rejecting all false null hypotheses. The probability of a correct decision is defined as the probability of correctly rejecting all false null hypotheses and accepting all true ones. Both criteria are discussed for nonstagewise and stagewise procedures in case of independent test statistics. For the example of 5 independently and normally distributed test statistics the values of the two criteria are calcutated under reasonably simple alternatives.

Statistical Power Lost and Statistical Power Regained: The Bonferroni Procedure in Exploratory Research

The Bonferroni procedure controls the risk of rejecting one or more true null hypotheses no matter how many significance tests are performed, but permits the risk of failing to reject false null hypotheses to grow with the number of tests. The loss of statistical power associated with the use of this procedure is demonstrated, and two options for alleviating the problem are explored. Setting a less stringent significance level for the set of tests is shown to be less effective than increasing the sample size.

Plots of P-values to evaluate many tests simultaneously

SUMMARY When a large number of tests are made, possibly on the same data, it is proposed to base a simultaneous evaluation of all the tests on a plot of cumulative P-values using the observed significance probabilities. The points corresponding to true null hypotheses should form a straight line, while those for false null hypotheses should deviate from this line. The line may be used to estimate the number of true hypotheses. The properties of the method are studied in some detail for the problems of comparing all pairs of means in a one-way layout, testing all correlation coefficients in a large correlation matrix, and the evaluation of all 2 x 2 subtables in a contingency table. The plot is illustrated on real data.

A REASSESSMENT OF STATISTICAL POWER ANALYSIS IN HUMAN COMMUNICATION RESEARCH

This study conducted a statistical power analysis of 64 articles appearing in the first four volumes of Human Communication Research, 1974–1978. Each article was examined, using Cohen's revised handbook, assuming nondirectional null hypotheses and an alpha level of .05. Statistical power, the probability of rejecting a false null hypothesis, was calculated for small, medium, and large experimental effect sizes and averaged by article and volume. Results indicated that the average probability of beta errors appears to have decreased over time, providing a greater chance of rejecting false null hypotheses, but this also raised several power-related issues relevant to communication research in general.