Number Of True Null Hypotheses Research Articles

Adaptive procedure for generalized familywise error rate control

ABSTRACTThis article considers multiple hypotheses testing with the generalized familywise error rate k-FWER control, which is the probability of at least k false rejections. We first assume the p-values corresponding to the true null hypotheses are independent, and propose adaptive generalized Bonferroni procedure with k-FWER control based on the estimation of the number of true null hypotheses. Then, we assume the p-values are dependent, satisfying block dependence, and propose adaptive procedure with k-FWER control. Extensive simulations compare the performance of the adaptive procedures with different estimators.

Dynamic adaptive multiple tests with finite sample FDR control

The present paper introduces new adaptive multiple tests which rely on the estimation of the number of true null hypotheses and which control the false discovery rate (FDR) at level α for finite sample size. We derive exact formulas for the FDR for a large class of adaptive multiple tests which apply to a new class of testing procedures. Generalized Storey estimators and weighted versions are introduced and it turns out that the corresponding adaptive step up and step down tests control the FDR. The present results also include particular dynamic adaptive step wise tests which use a data dependent weighting of the new generalized Storey estimators. In addition, a converse of the Benjamini and Hochberg (1995) theorem is given. The Benjamini and Hochberg (1995) test is the only “distribution free” step up test with FDR being independent of the distribution of the p-values of false null hypotheses.

New procedures controlling the false discovery proportion via Romano–Wolf’s heuristic

The false discovery proportion (FDP) is a convenient way to account for false positives when a large number $m$ of tests are performed simultaneously. Romano and Wolf [ Ann. Statist. 35 (2007) 1378–1408] have proposed a general principle that builds FDP controlling procedures from $k$-family-wise error rate controlling procedures while incorporating dependencies in an appropriate manner; see Korn et al. [ J. Statist. Plann. Inference 124 (2004) 379–398]; Romano and Wolf (2007). However, the theoretical validity of the latter is still largely unknown. This paper provides a careful study of this heuristic: first, we extend this approach by using a notion of “bounding device” that allows us to cover a wide range of critical values, including those that adapt to $m_{0}$, the number of true null hypotheses. Second, the theoretical validity of the latter is investigated both nonasymptotically and asymptotically. Third, we introduce suitable modifications of this heuristic that provide new methods, overcoming the existing procedures with a proven FDP control.

Bonferroni-type Plug-in Procedure Controlling Generalized Familywise Error Rate

Consider the multiple hypotheses testing problem controlling the generalized familywise error rate k-FWER, the probability of at least k false rejections. We propose a plug-in procedure based on the estimation of the number of true null hypotheses. Under the independence assumption of the p-values corresponding to the true null hypotheses, we first introduce the least favorable configuration (LFC) of k-FWER for Bonferroni-type plug-in procedure, then we construct a plug-in k-FWER-controlled procedure based on LFC. For dependent p-values, we establish the asymptotic k-FWER control under some mild conditions. Simulation studies suggest great improvement over generalized Bonferroni test and generalized Holm test.

False discovery rate control under Archimedean copula

We are concerned with the false discovery rate (FDR) of the linear step-up test $\varphi^{LSU}$ considered by Benjamini and Hochberg (1995). It is well known that $\varphi^{LSU}$ controls the FDR at level $m_{0}q/m$ if the joint distribution of $p$-values is multivariate totally positive of order $2$. In this, $m$ denotes the total number of hypotheses, $m_{0}$ the number of true null hypotheses, and $q$ the nominal FDR level. Under the assumption of an Archimedean $p$-value copula with completely monotone generator, we derive a sharper upper bound for the FDR of $\varphi^{LSU}$ as well as a non-trivial lower bound. Application of the sharper upper bound to parametric subclasses of Archimedean $p$-value copulae allows us to increase the power of $\varphi^{LSU}$ by pre-estimating the copula parameter and adjusting $q$. Based on the lower bound, a sufficient condition is obtained under which the FDR of $\varphi^{LSU}$ is exactly equal to $m_{0}q/m$, as in the case of stochastically independent $p$-values. Finally, we deal with high-dimensional multiple test problems with exchangeable test statistics by drawing a connection between infinite sequences of exchangeable $p$-values and Archimedean copulae with completely monotone generators. Our theoretical results are applied to important copula families, including Clayton copulae and Gumbel-Hougaard copulae.

A Bayesian decision procedure for testing multiple hypotheses in DNA microarray experiments

DNA microarray experiments require the use of multiple hypothesis testing procedures because thousands of hypotheses are simultaneously tested. We deal with this problem from a Bayesian decision theory perspective. We propose a decision criterion based on an estimation of the number of false null hypotheses (FNH), taking as an error measure the proportion of the posterior expected number of false positives with respect to the estimated number of true null hypotheses. The methodology is applied to a Gaussian model when testing bilateral hypotheses. The procedure is illustrated with both simulated and real data examples and the results are compared to those obtained by the Bayes rule when an additive loss function is considered for each joint action and the generalized loss 0-1 function for each individual action. Our procedure significantly reduced the percentage of false negatives whereas the percentage of false positives remains at an acceptable level.

Estimating the number of true null hypotheses in multiple hypothesis testing

The overall Type I error computed based on the traditional means may be inflated if many hypotheses are compared simultaneously. The family-wise error rate (FWER) and false discovery rate (FDR) are some of commonly used error rates to measure Type I error under the multiple hypothesis setting. Many controlling FWER and FDR procedures have been proposed and have the ability to control the desired FWER/FDR under certain scenarios. Nevertheless, these controlling procedures become too conservative when only some hypotheses are from the null. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) proposed an adaptive FDR-controlling procedure that adapts the information of the number of true null hypotheses (m 0) to overcome this problem. Since m 0 is unknown, estimators of m 0 are needed. Benjamini and Hochberg (J. Educ. Behav. Stat. 25:60–83, 2000) suggested a graphical approach to construct an estimator of m 0, which is shown to overestimate m 0 (see Hwang in J. Stat. Comput. Simul. 81:207–220, 2011). Following a similar construction, this paper proposes new estimators of m 0. Monte Carlo simulations are used to evaluate accuracy and precision of new estimators and the feasibility of these new adaptive procedures is evaluated under various simulation settings.

Denoising with advanced stepwise false discovery rate control and its application to fault diagnosis

To extract defect features from the signal with background noise for fault diagnosis, a novel approach is proposed by using advanced false discovery rate procedure (AFDR). The main idea is based on controlling false discovery rate (FDR) through combination of all three stepwise procedures (step-up, step-down, step-up-down) and estimation of the number of true null hypotheses. The AFDR procedure differs from the standard FDR procedure in two respects, i.e., enhancing the efficiency by reducing the number of tested hypotheses and improving the power. The proposed procedure consists of two main steps: firstly, the signal is represented more parsimoniously in wavelet domain; secondly, a most appropriate stepwise FDR procedure is selected according to the character of wavelet coefficients. Both the numerical simulation results and the experimental results for bearing defect diagnosis show that the proposed approach is a competitive shrinkage method compared with other popular techniques.

Sequential Monte Carlo multiple testing

Motivation: In molecular biology, as in many other scientific fields, the scale of analyses is ever increasing. Often, complex Monte Carlo simulation is required, sometimes within a large-scale multiple testing setting. The resulting computational costs may be prohibitively high.Results: We here present MCFDR, a simple, novel algorithm for false discovery rate (FDR) modulated sequential Monte Carlo (MC) multiple hypothesis testing. The algorithm iterates between adding MC samples across tests and calculating intermediate FDR values for the collection of tests. MC sampling is stopped either by sequential MC or based on a threshold on FDR. An essential property of the algorithm is that it limits the total number of MC samples whatever the number of true null hypotheses. We show on both real and simulated data that the proposed algorithm provides large gains in computational efficiency.Availability: MCFDR is implemented in the Genomic HyperBrowser (http://hyperbrowser.uio.no/mcfdr), a web-based system for genome analysis. All input data and results are available and can be reproduced through a Galaxy Pages document at: http://hyperbrowser.uio.no/mcfdr/u/sandve/p/mcfdr.Contact: geirksa@ifi.uio.no

On adaptive procedures controlling the familywise error rate

The idea of modifying, and potentially improving, classical multiple testing methods controlling the familywise error rate (FWER) via an estimate of the unknown number of true null hypotheses has been around for a long time without a formal answer to the question whether or not such adaptive methods ultimately maintain the strong control of FWER, until Finner and Gontscharuk (2009) and Guo (2009) have offered some answers. A class of adaptive Bonferroni and S˘idàk methods larger than considered in those papers is introduced, with the FWER control now proved under a weaker distributional setup. Numerical results show that there are versions of adaptive Bonferroni and S˘idàk methods that can perform better under certain positive dependence situations than those previously considered. A different adaptive Holm method and its stepup analog, referred to as an adaptive Hochberg method, are also introduced, and their FWER control is proved asymptotically, as in those papers. These adaptive Holm and Hochberg methods are numerically seen to often outperform the previously considered adaptive Holm method.

Comparisons of estimators of the number of true null hypotheses and adaptive FDR procedures in multiplicity testing

Many exploratory studies such as microarray experiments require the simultaneous comparison of hundreds or thousands of genes. It is common to see that most genes in many microarray experiments are not expected to be differentially expressed. Under such a setting, a procedure that is designed to control the false discovery rate (FDR) is aimed at identifying as many potential differentially expressed genes as possible. The usual FDR controlling procedure is constructed based on the number of hypotheses. However, it can become very conservative when some of the alternative hypotheses are expected to be true. The power of a controlling procedure can be improved if the number of true null hypotheses (m 0) instead of the number of hypotheses is incorporated in the procedure [Y. Benjamini and Y. Hochberg, On the adaptive control of the false discovery rate in multiple testing with independent statistics, J. Edu. Behav. Statist. 25(2000), pp. 60–83]. Nevertheless, m 0 is unknown, and has to be estimated. The objective of this article is to evaluate some existing estimators of m 0 and discuss the feasibility of these estimators in incorporating into FDR controlling procedures under various experimental settings. The results of simulations can help the investigator to choose an appropriate procedure to meet the requirement of the study.

Evaluations of FWER-controlling methods in multiple hypothesis testing

Simultaneously testing a family of n null hypotheses can arise in many applications. A common problem in multiple hypothesis testing is to control Type-I error. The probability of at least one false rejection referred to as the familywise error rate (FWER) is one of the earliest error rate measures. Many FWER-controlling procedures have been proposed. The ability to control the FWER and achieve higher power is often used to evaluate the performance of a controlling procedure. However, when testing multiple hypotheses, FWER and power are not sufficient for evaluating controlling procedure’s performance. Furthermore, the performance of a controlling procedure is also governed by experimental parameters such as the number of hypotheses, sample size, the number of true null hypotheses and data structure. This paper evaluates, under various experimental settings, the performance of some FWER-controlling procedures in terms of five indices, the FWER, the false discovery rate, the false non-discovery rate, the sensitivity and the specificity. The results can provide guidance on how to select an appropriate FWER-controlling procedure to meet a study’s objective.

Estimation of the number of true null hypotheses when conducting a multiple testing

Application of the standard hypothesis test with no adjustment for the multiple testing leads to a large number of false discoveries. The most convenient error measure in multiple testing is False Discovery Rate, FDR. However, calculation of FDR requires good estimation of the number of true null hypotheses, n 0, (or equivalently, of π 0 = n 0/ n, where n denotes the number of all tested hypotheses). Estimation of π 0 is a non-trivial problem and it can be done under several assumptions about input data. In our study, several approaches to the estimation of π 0 values are compared for the different data models (independent features, block-correlated data and mixture models). The presented results give evidence that general dependence of the features leads to a very doubtful estimation of their significance.

Axonal Transport Proteomics Reveals Mobilization of Translation Machinery to the Lesion Site in Injured Sciatic Nerve

Investigations of the molecular mechanisms underlying responses to nerve injury have highlighted the importance of axonal transport systems. To obtain a comprehensive view of the protein ensembles associated with axonal transport in injured axons, we analyzed the protein compositions of axoplasm concentrated at ligatures following crush injury of rat sciatic nerve. LC-MS/MS analyses of iTRAQ-labeled peptides from axoplasm distal and proximal to the ligation sites revealed protein ensembles transported in both anterograde and retrograde directions. Variability of replicates did not allow straightforward assignment of proteins to functional transport categories; hence, we performed principal component analysis and factor analysis with subsequent clustering to determine the most prominent injury-related transported proteins. This strategy circumvented experimental variability and allowed the extraction of biologically meaningful information from the quantitative neuroproteomics experiments. 299 proteins were highlighted by principal component analysis and factor analysis, 145 of which correlate with retrograde and 154 of which correlate with anterograde transport after injury. The analyses reveal extensive changes in both anterograde and retrograde transport proteomes in injured peripheral axons and emphasize the importance of RNA binding and translational machineries in the axonal response to injury.

Effect of Quantitative Trait Loci for Milk Protein Percentage on Milk Protein Yield and Milk Yield in Israeli Holstein Dairy Cattle

Although numerous quantitative trait loci (QTL) mapping studies involving milk protein percent (PP), milk yield (MY), and protein yield (PY) have been carried out, there has not been any systematic evaluation of the effects of individual QTL on these 3 interrelated traits. Consequently, the aim of the present study was to investigate the effects on MY and PY of QTL for PP previously mapped in various laboratories. The study, based on selective DNA pooling of milk samples, included 10 Israeli Holstein artificial insemination bulls, each the sire of 1,800 or more milk-recorded daughters. For each sire-trait combination across the 10 sires, milk samples of the highest and lowest daughters with respect to estimated breeding values for PP, PY, and MY were collected for pooling. A total of 134 dinucleotide microsatellites distributed over 25 bovine autosomes were used. An empirical standard error for marker-QTL linkage testing was calculated based on the variation among split samples within the same tail. Threshold comparison-wise error rate P-values were set to control proportion of false positives at P=0.10 level for declaring significant effects at the marker-trait level. Estimates of the number of true null hypotheses for each trait were obtained from the histogram of marker comparison-wise error rate P-values. Based on these estimates, effective power of the experiment at the marker-trait level was estimated as 0.75, 0.41, and 0.73 for PP, PY, and MY. The proportion of heterozygosity at the QTL was estimated as 0.46, 0.39, and 0.40, respectively. After correcting for incomplete power and proportion of false positives, it was estimated that 38.7 and 37.5% of the markers affecting PP and MY, respectively, also affected PY. Of the markers affecting PY, 68.9 and 76.5%, respectively, also affected PP and MY. Apparently, none of the significant markers affected PY exclusively, and only 6.5 and 16.0%, respectively, affected PP or MY exclusively. Thus, almost all significant markers, and by inference almost all QTL, had effects on at least 2 of the 3 traits.

Effects of dependence in high-dimensional multiple testing problems

BackgroundWe consider effects of dependence among variables of high-dimensional data in multiple hypothesis testing problems, in particular the False Discovery Rate (FDR) control procedures. Recent simulation studies consider only simple correlation structures among variables, which is hardly inspired by real data features. Our aim is to systematically study effects of several network features like sparsity and correlation strength by imposing dependence structures among variables using random correlation matrices.ResultsWe study the robustness against dependence of several FDR procedures that are popular in microarray studies, such as Benjamin-Hochberg FDR, Storey's q-value, SAM and resampling based FDR procedures. False Non-discovery Rates and estimates of the number of null hypotheses are computed from those methods and compared. Our simulation study shows that methods such as SAM and the q-value do not adequately control the FDR to the level claimed under dependence conditions. On the other hand, the adaptive Benjamini-Hochberg procedure seems to be most robust while remaining conservative. Finally, the estimates of the number of true null hypotheses under various dependence conditions are variable.ConclusionWe discuss a new method for efficient guided simulation of dependent data, which satisfy imposed network constraints as conditional independence structures. Our simulation set-up allows for a structural study of the effect of dependencies on multiple testing criterions and is useful for testing a potentially new method on π0 or FDR estimation in a dependency context.

On control of the false discovery rate under no assumption of dependency

Most false discovery rate (FDR) controlling procedures require certain assumptions on the joint distribution of p-values. Benjamini and Hochberg [1995. Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B 57, 289–300] proposed a step-up procedure with critical constants α i = ( i / m ) α , 1 ⩽ i ⩽ m , for a given level 0 < α < 1 and showed that FDR ⩽ ( m 0 / m ) α under the assumption of independence of p-values, where m is the total number of null hypotheses and m 0 the number of true null hypotheses. Benjamini and Yekutieli [2001. The control of the false discovery rate in multiple testing under dependency. Ann. Statist. 29, 1165–1188] showed that for the same procedure FDR ⩽ ( m 0 / m ) α ∑ j = 1 m 1 / j , whatever may be the joint distribution of p-values. In one of the results in this paper, we show that this upper bound for FDR cannot be improved in the sense that there exists a joint distribution of p-values for which the upper bound is attained. A major thrust of this paper is to work in the realm of step-down procedures without imposing any condition on the joint distribution of the underlying p-values. As a starting point, we give an explicit expression for FDR specially tailored for step-down procedures. Using the same critical constants as those of the Benjamini–Hochberg procedure, we present a new step-down procedure for which the upper bound for FDR is much lower than what is given by Benjamini and Yekutieli. The explicit expression given for FDR and some optimization techniques stemming from the knapsack problem are instrumental in getting the main result. We also present some general results on stepwise procedures built on non-decreasing sequences of critical constants.

Incorporating the number of true null hypotheses to improve power in multiple testing: application to gene microarray data

Testing for significance with gene expression data from DNA microarray experiments involves simultaneous comparisons of hundreds or thousands of genes. In common exploratory microarray experiments, most genes are not expected to be differentially expressed. The family-wise error (FWE) rate and false discovery rate (FDR) are two common approaches used to account for multiple hypothesis tests to identify differentially expressed genes. When the number of hypotheses is very large and some null hypotheses are expected to be true, the power of an FWE or FDR procedure can be improved if the number of null hypotheses is known. The mean of differences (MD) of ranked p-values has been proposed to estimate the number of true null hypotheses under the independence model. This article proposes to incorporate the MD estimate into an FWE or FDR approach for gene identification. Simulation results show that the procedure appears to control the FWE and FDR well at the FWE=0.05 and FDR=0.05 significant levels; it exceeds the nominal level for FDR=0.01 when the null hypotheses are highly correlated, a correlation of 0.941. The proposed approach is applied to a public colon tumor data set for illustration.

Estimating the number of true null hypotheses from a histogram of p values

In an earlier article, an intuitively appealing method for estimating the number of true null hypotheses in a multiple test situation was proposed. That article presented an iterative algorithm that relies on a histogram of observed p values to obtain the estimator. We characterize the limit of that iterative algorithm and show that the estimator can be computed directly without iteration. We compare the performance of the histogram-based estimator with other procedures for estimating the number of true null hypotheses from a collection of observed p values and find that the histogram-based estimator performs well in settings similar to those encountered in microarray data analysis. We demonstrate the approach using p values from a large microarray experiment aimed at uncovering molecular mechanisms of barley resistance to a fungal pathogen.

Multiple-testing strategy for analyzing cDNA array data on gene expression.

An objective of many functional genomics studies is to estimate treatment-induced changes in gene expression. cDNA arrays interrogate each tissue sample for the levels of mRNA for hundreds to tens of thousands of genes, and the use of this technology leads to a multitude of treatment contrasts. By-gene hypotheses tests evaluate the evidence supporting no effect, but selecting a significance level requires dealing with the multitude of comparisons. The p-values from these tests order the genes such that a p-value cutoff divides the genes into two sets. Ideally one set would contain the affected genes and the other would contain the unaffected genes. However, the set of genes selected as affected will have false positives, i.e., genes that are not affected by treatment. Likewise, the other set of genes, selected as unaffected, will contain false negatives, i.e., genes that are affected. A plot of the observed p-values (1 - p) versus their expectation under a uniform [0, 1] distribution allows one to estimate the number of true null hypotheses. With this estimate, the false positive rates and false negative rates associated with any p-value cutoff can be estimated. When computed for a range of cutoffs, these rates summarize the ability of the study to resolve effects. In our work, we are more interested in selecting most of the affected genes rather than protecting against a few false positives. An optimum cutoff, i.e., the best set given the data, depends upon the relative cost of falsely classifying a gene as affected versus the cost of falsely classifying a gene as unaffected. We select the cutoff by a decision-theoretic method analogous to methods developed for receiver operating characteristic curves. In addition, we estimate the false discovery rate and the false nondiscovery rate associated with any cutoff value. Two functional genomics studies that were designed to assess a treatment effect are used to illustrate how the methods allowed the investigators to determine a cutoff to suit their research goals.