False Null Hypotheses Research Articles

Cost-Aware Generalized α-Investing for Multiple Hypothesis Testing

We consider the problem of sequential multiple hypothesis testing with nontrivial data collection costs. This problem appears, for example, when conducting biological experiments to identify differentially expressed genes of a disease process. This work builds on the generalized α-investing framework which enables control of the marginal false discovery rate in a sequential testing setting. We make a theoretical analysis of the long term asymptotic behavior of α-wealth which motivates a consideration of sample size in the α-investing decision rule. Posing the testing process as a game with nature, we construct a decision rule that optimizes the expected α-wealth reward (ERO) and provides an optimal sample size for each test. Empirical results show that a cost-aware ERO decision rule correctly rejects more false null hypotheses than other methods for $n=1$ where n is the sample size. When the sample size is not fixed cost-aware ERO uses a prior on the null hypothesis to adaptively allocate of the sample budget to each test. We extend cost-aware ERO investing to finite-horizon testing which enables the decision rule to allocate samples in a non-myopic manner. Finally, empirical tests on real data sets from biological experiments show that cost-aware ERO balances the allocation of samples to an individual test against the allocation of samples across multiple tests.

The Effects of Weight Choices on the Power of the Getis–Ord Statistic

When local spatial clustering exists, local statistics are most likely to be significant when their associated weights match the spatial form and extent of the actual clustering. This paper focuses upon the cost of misspecifying the weights of the Getis–Ord statistic. In particular, it is more difficult to reject false null hypotheses when the weights are poorly chosen. I also examine the likelihood of finding spatial clusters when a range of spatial scales is examined, and when the multiple testing that this entails is accounted for. If there is uncertainty regarding the scale of the process, there is little cost in examining a range that includes spatial scales that are larger than the true cluster. Gains in the power to detect significant clustering may be had if the examination of cluster sizes that are clearly too small may be omitted. A small number of Bonferroni‐adjusted tests will often provide a slight decline in statistical power, relative to tests that search comprehensively over a range, but may have the benefit of providing a relatively better estimate of the location of change.

Optimal and Maximin Procedures for Multiple Testing Problems

AbstractMultiple testing problems (MTPs) are a staple of modern statistical analysis. The fundamental objective of MTPs is to reject as many false null hypotheses as possible (that is, maximize some notion of power), subject to controlling an overall measure of false discovery, like family-wise error rate (FWER) or false discovery rate (FDR). In this paper we provide generalizations to MTPs of the optimal Neyman-Pearson test for a single hypothesis. We show that for simple hypotheses, for both FWER and FDR and relevant notions of power, finding the optimal multiple testing procedure can be formulated as infinite dimensional binary programs and can in principle be solved for any number of hypotheses. We also characterize maximin rules for complex alternatives, and demonstrate that such rules can be found in practice, leading to improved practical procedures compared to existing alternatives that guarantee strong error control on the entire parameter space. We demonstrate the usefulness of these novel rules for identifying which studies contain signal in numerical experiments as well as in application to clinical trials with multiple studies. In various settings, the increase in power from using optimal and maximin procedures can range from 15% to more than 100%.

Scan Statistics Adjusted for Global Spatial Autocorrelation

Failure to account for global spatial autocorrelation when using scan statistics to find clusters generated by local processes will result in P‐values that are too low, and consequently, spurious findings of statistical significance are not uncommon. The presence of global spatial autocorrelation also decreases the ability to reject false null hypotheses and it is therefore more difficult to find local clusters when they exist. By estimating the degree of global autocorrelation and using that estimate to transform the data, it is then possible to apply scan statistics to the transformed data. This results in a reduction in the likelihood of spurious finding of statistical significance when local clusters do not exist.

Using Copula to Model Dependence When Testing Multiple Hypotheses in DNA Microarray Experiments: A Bayesian Approximation

Many experiments require simultaneously testing many hypotheses. This is particularly relevant in the context of DNA microarray experiments, where it is common to analyze many genes to determine which of them are differentially expressed under two conditions. Another important problem in this context is how to model the dependence at the level of gene expression. In this paper, we propose a Bayesian procedure for simultaneously testing multiple hypotheses, modeling the dependence through copula functions, where all available information, both objective and subjective, can be used. The approach has the advantage that it can be used with different dependency structures. Simulated data analysis was performed to examine the performance of the proposed approach. The results show that our procedure captures the dependence appropriately classifying adequately a high percentage of true and false null hypotheses when choosing a prior distribution beta skewed to the right for the initial probability of each null hypothesis, resulting in a very powerful procedure. The procedure is also illustrated with real data.

Estimating the proportion of true null hypotheses with application in microarray data

A new formulation for the proportion of true null hypotheses based on the sum of all p-values and the average of expected p-values under the false null hypotheses has been proposed in the current work. This formulation of the parameter of interest π 0 has also been used to construct a new estimator for the same. The proposed estimator removes the problem of choosing tuning parameters in the existing estimators. Though the formulation is quite general, computation of the new estimator requires use of an initial estimate of π 0. The issue of choosing an appropriate initial estimator is also discussed in this work. The current work assumes normality for the expression level of each gene and also assumes similar tests for all the hypotheses. Simulation study shows that, the proposed estimator performs better than its closest competitor over a substantial continuous subinterval of the parameter space, under independence and weak dependence among the gene expression levels. The proposed method of estimation is applied to two real gene expression level data-sets and the results are in line with what is obtained by the competing method.

Asymptotically optimal sequential FDR and pFDR control with (or without) prior information on the number of signals

We investigate asymptotically optimal multiple testing procedures for streams of sequential data in the context of prior information on the number of false null hypotheses (“signals”). We show that the “gap” and “gap-intersection” procedures, recently proposed and shown by Song and Fellouris (2017) to be asymptotically optimal for controlling type 1 and 2 familywise error rates (FWEs), are also asymptotically optimal for controlling FDR/FNR when their critical values are appropriately adjusted. Generalizing this result, we show that these procedures, again with appropriately adjusted critical values, are asymptotically optimal for controlling any multiple testing error metric that is bounded between multiples of FWE in a certain sense. This class of metrics includes FDR/FNR but also pFDR/pFNR, the per-comparison and per-family error rates, and the false positive rate. Our analysis includes asymptotic regimes in which the number of null hypotheses approaches ∞ as the type 1 and 2 error metrics approach 0.

Uniformly consistently estimating the proportion of false null hypotheses via Lebesgue–Stieltjes integral equations

The proportion of false null hypotheses is a very important quantity in statistical modelling and inference based on the two-component mixture model and its extensions, and in control and estimation of the false discovery rate and false non-discovery rate. Most existing estimators of this proportion threshold p-values, deconvolve the mixture model under constraints on its components, or depend heavily on the location-shift property of distributions. Hence, they usually are not consistent, applicable to non-location-shift distributions, or applicable to discrete statistics or p-values. To eliminate these shortcomings, we construct uniformly consistent estimators of the proportion as solutions to Lebesgue–Stieltjes integral equations. In particular, we provide such estimators respectively for random variables whose distributions have Riemann–Lebesgue type characteristic functions, whose distributions form discrete natural exponential families with infinite supports, and whose distributions form natural exponential families with separable moment sequences. We provide the speed of convergence and uniform consistency class for each such estimator under independence. In addition, we provide two examples for which a consistent estimator of the proportion cannot be constructed using our techniques.

Some clarifications regarding power and Type I error control for pairwise comparisons of three groups

A previous study in this journal used Monte Carlo simulations to compare the power and familywise Type I error rates of ten multiple-testing procedures in the context of pairwise comparisons in balanced three-group designs. The authors concluded that the Benjamini–Hochberg procedure was the best.' However, they did not compare the Benjamini–Hochberg procedure to commonly used multiple-testing procedures that were developed specifically for pairwise comparisons, such as Fisher's protected least significant difference and Tukey's honest significant difference. Simulations in the present study show that in the three-group case, Fisher's method is more powerful than both Tukey's method and the Benjamini–Hochberg procedure. Compared to the Benjamini–Hochberg procedure, Tukey's method is shown to be less powerful in terms of per-pair power (average probability of significance across the tests of false null hypotheses), but more powerful in terms of any-pair power (probability of significance in at least one test of a false null hypothesis). Additionally, the present study shows that small deviations from normality in the population distributions have little effect on the power of pairwise comparisons, and that the previous study's finding to the contrary was based on a methodological inconsistency.

Multiple hypothesis testing in experimental economics

The analysis of data from experiments in economics routinely involves testing multiple null hypotheses simultaneously. These different null hypotheses arise naturally in this setting for at least three different reasons: when there are multiple outcomes of interest and it is desired to determine on which of these outcomes a treatment has an effect; when the effect of a treatment may be heterogeneous in that it varies across subgroups defined by observed characteristics and it is desired to determine for which of these subgroups a treatment has an effect; and finally when there are multiple treatments of interest and it is desired to determine which treatments have an effect relative to either the control or relative to each of the other treatments. In this paper, we provide a bootstrap-based procedure for testing these null hypotheses simultaneously using experimental data in which simple random sampling is used to assign treatment status to units. Using the general results in Romano and Wolf (Ann Stat 38:598–633, 2010), we show under weak assumptions that our procedure (1) asymptotically controls the familywise error rate—the probability of one or more false rejections—and (2) is asymptotically balanced in that the marginal probability of rejecting any true null hypothesis is approximately equal in large samples. Importantly, by incorporating information about dependence ignored in classical multiple testing procedures, such as the Bonferroni and Holm corrections, our procedure has much greater ability to detect truly false null hypotheses. In the presence of multiple treatments, we additionally show how to exploit logical restrictions across null hypotheses to further improve power. We illustrate our methodology by revisiting the study by Karlan and List (Am Econ Rev 97(5):1774–1793, 2007) of why people give to charitable causes.

An omnibus test for the global null hypothesis.

Global hypothesis tests are a useful tool in the context of clinical trials, genetic studies, or meta-analyses, when researchers are not interested in testing individual hypotheses, but in testing whether none of the hypotheses is false. There are several possibilities how to test the global null hypothesis when the individual null hypotheses are independent. If it is assumed that many of the individual null hypotheses are false, combination tests have been recommended to maximize power. If, however, it is assumed that only one or a few null hypotheses are false, global tests based on individual test statistics are more powerful (e.g. Bonferroni or Simes test). However, usually there is no a priori knowledge on the number of false individual null hypotheses. We therefore propose an omnibus test based on cumulative sums of the transformed p-values. We show that this test yields an impressive overall performance. The proposed method is implemented in an R-package called omnibus.

Combination of “combinations of p values”

We investigate the impact of an uncertain number of false individual null hypotheses on commonly used p value combination methods. Under such uncertainty, these methods perform quite differently and often yield conflicting results. Consequently, we develop a combination of “combinations of p values” (CCP) test aimed at maintaining good power properties across such uncertainty. The CCP test is based on a simple union–intersection principle that exploits the weak correspondence between two underlying p value combination methods. Monte Carlo simulations show that the CCP test controls size and closely tracks the power of the best individual methods. We empirically apply the CCP test to explore the stationarity in real exchange rates and the information rigidity in inflation and output growth forecasts.

Comments and Suggestions on Defining and Using P-Values and Null Hypothesis Tests

Comments and Suggestions on Defining and Using P-Values and Null Hypothesis Tests David Marshall, Keston Lindsay Abstract Recent corrections of fallacious language about P-values preserved the debatable definition of the P-value as an element of evidence for or against a null hypothesis. Hypothesis tests are mathematical simulations. Simulations provide expectations about modeled outcomes and cannot provide empirical evidence concerning the object of the simulation. No element of the H0 test simulation, including the P-value, provides �evidence� for or against a hypothesis. Adequately powered hypothesis test simulations provide very encouraging predictions that correct decisions will be made about true and false null hypotheses, giving the researcher confidence that a correct decision about the hypothesis will be made. Computed test statistics and their related P-values remain useful as after-the-fact decision tools, while sample data and descriptive statistics make the sole empirical evidentiary base of support for a hypothesis. Full Text: PDF DOI: 10.15640/arms.v5n1a1

Type-II generalized family-wise error rate formulas with application to sample size determination.

Multiple endpoints are increasingly used in clinical trials. The significance of some of these clinical trials is established if at least r null hypotheses are rejected among m that are simultaneously tested. The usual approach in multiple hypothesis testing is to control the family-wise error rate, which is defined as the probability that at least one type-I error is made. More recently, the q-generalized family-wise error rate has been introduced to control the probability of making at least q false rejections. For procedures controlling this global type-I error rate, we define a type-II r-generalized family-wise error rate, which is directly related to the r-power defined as the probability of rejecting at least r false null hypotheses. We obtain very general power formulas that can be used to compute the sample size for single-step and step-wise procedures. These are implemented in our R package rPowerSampleSize available on the CRAN, making them directly available to end users. Complexities of the formulas are presented to gain insight into computation time issues. Comparison with Monte Carlo strategy is also presented. We compute sample sizes for two clinical trials involving multiple endpoints: one designed to investigate the effectiveness of a drug against acute heart failure and the other for the immunogenicity of a vaccine strategy against pneumococcus. Copyright © 2016 John Wiley & Sons, Ltd.

A powerful FDR control procedure for multiple hypotheses

A powerful test procedure is proposed for multiple hypotheses for the false discovery rate (FDR) control. The proposed procedure is a weighted p-value procedure which explores false null hypotheses information. It is theoretically shown to control the FDR and be more powerful than the widely used plug-in BH procedure. When there are unknown parameters estimated from the data, the asymptotic properties of the proposed procedure are discussed. The extensive simulation studies further verify the theoretical results. A real data is analyzed to illustrate the proposed method.

Optimal False Discovery Rate Control with Kernel Density Estimation in a Microarray Experiment

Most of current false discovery rate (FDR) procedures in a microarray experiment assume restrictive dependence structures, resulting in being less reliable. FDR controlling procedure under suitable dependence structures based on Poisson distributional approximation is shown. Unlike other procedures, the distribution of false null hypotheses is estimated by using kernel density estimation allowing for dependent structures among the genes. Furthermore, we develop an FDR framework that minimizes the false nondiscovery rate (FNR) with a constraint on the controlled level of the FDR. The performance of the proposed FDR procedure is compared with that of other existing FDR controlling procedures, with an application to the microarray study of simulated data.

Dynamic adaptive multiple tests with finite sample FDR control

The present paper introduces new adaptive multiple tests which rely on the estimation of the number of true null hypotheses and which control the false discovery rate (FDR) at level α for finite sample size. We derive exact formulas for the FDR for a large class of adaptive multiple tests which apply to a new class of testing procedures. Generalized Storey estimators and weighted versions are introduced and it turns out that the corresponding adaptive step up and step down tests control the FDR. The present results also include particular dynamic adaptive step wise tests which use a data dependent weighting of the new generalized Storey estimators. In addition, a converse of the Benjamini and Hochberg (1995) theorem is given. The Benjamini and Hochberg (1995) test is the only “distribution free” step up test with FDR being independent of the distribution of the p-values of false null hypotheses.

Minimal detectable outliers as measures of reliability

The concept of reliability was introduced into geodesy by Baarda (A testing procedure for use in geodetic networks. Publications on Geodesy, vol. 2. Netherlands Geodetic Commission, Delft, 1968). It gives a measure for the ability of a parameter estimation to detect outliers and leads in case of one outlier to the MDB, the minimal detectable bias or outlier. The MDB depends on the non-centrality parameter of the \(\chi ^2\)-distribution, as the variance factor of the linear model is assumed to be known, on the size of the outlier test of an individual observation which is set to 0.001 and on the power of the test which is generally chosen to be 0.80. Starting from an estimated variance factor, the \(F\)-distribution is applied here. Furthermore, the size of the test of the individual observation is a function of the number of outliers to keep the size of the test of all observations constant, say 0.05. The power of the test is set to 0.80. The MDBs for multiple outliers are derived here under these assumptions. The method is applied to the reconstruction of a bell-shaped surface measured by a laser scanner. The MDBs are introduced as outliers for the alternative hypotheses of the outlier tests. A Monte Carlo method reveals that due to the way of introducing the outliers, the false null hypotheses cannot be rejected on the average with a power of 0.80 if the MDBs are not enlarged by a factor.

Compound p-value statistics for multiple testing procedures.

Many multiple testing procedures make use of the p-values from the individual pairs of hypothesis tests, and are valid if the p-value statistics are independent and uniformly distributed under the null hypotheses. However, it has recently been shown that these types of multiple testing procedures are inefficient since such p-values do not depend upon all of the available data. This paper provides tools for constructing compound p-value statistics, which are those that depend upon all of the available data, but still satisfy the conditions of independence and uniformity under the null hypotheses. Several examples are provided, including a class of compound p-value statistics for testing location shifts. It is demonstrated, both analytically and through simulations, that multiple testing procedures tend to reject more false null hypotheses when applied to these compound p-values rather than the usual p-values, and at the same time still guarantee the desired type I error rate control. The compound p-values are used to analyze a real microarray data set and allow for more rejected null hypotheses.

A Bayesian decision procedure for testing multiple hypotheses in DNA microarray experiments

DNA microarray experiments require the use of multiple hypothesis testing procedures because thousands of hypotheses are simultaneously tested. We deal with this problem from a Bayesian decision theory perspective. We propose a decision criterion based on an estimation of the number of false null hypotheses (FNH), taking as an error measure the proportion of the posterior expected number of false positives with respect to the estimated number of true null hypotheses. The methodology is applied to a Gaussian model when testing bilateral hypotheses. The procedure is illustrated with both simulated and real data examples and the results are compared to those obtained by the Bayes rule when an additive loss function is considered for each joint action and the generalized loss 0-1 function for each individual action. Our procedure significantly reduced the percentage of false negatives whereas the percentage of false positives remains at an acceptable level.