Standard False Discovery Rate Research Articles

Multiple testing with heterogeneous multinomial distributions.

Standard false discovery rate (FDR) procedures can provide misleading inference when testing multiple null hypotheses with heterogeneous multinomial data. For example, in the motivating study the goal is to identify species of bacteria near the roots of wheat plants (rhizobacteria) that are moderately or strongly associated with productivity. However, standard procedures discover the most abundant species even when their association is weak and fail to discover many moderate and strong associations when the species are not abundant. This article provides a new FDR-controlling method based on a finite mixture of multinomial distributions and shows that it tends to discover more moderate and strong associations and fewer weak associations when the data are heterogeneous across tests. The new method is applied to the rhizobacteria data and performs favorably over competing methods.

Symmetric directional false discovery rate control

This research is motivated from the analysis of a real gene expression data that aims to identify a subset of “interesting” or “significant” genes for further studies. When we blindly applied the standard false discovery rate (FDR) methods, our biology collaborators were suspicious or confused, as the selected list of significant genes was highly unbalanced: there were ten times more under-expressed genes than the over-expressed genes. Their concerns led us to realize that the observed two-sample t-statistics were highly skewed and asymmetric, and thus the standard FDR methods might be inappropriate. To tackle this case, we propose a symmetric directional FDR control method that categorizes the genes into “over-expressed” and “under-expressed” genes, pairs “over-expressed” and “under-expressed” genes, defines the p-values for gene pairs via column permutations, and then applies the standard FDR method to select “significant” gene pairs instead of “significant” individual genes. We compare our proposed symmetric directional FDR method with the standard FDR method by applying them to simulated data and several well-known real data sets.

Denoising with advanced stepwise false discovery rate control and its application to fault diagnosis

To extract defect features from the signal with background noise for fault diagnosis, a novel approach is proposed by using advanced false discovery rate procedure (AFDR). The main idea is based on controlling false discovery rate (FDR) through combination of all three stepwise procedures (step-up, step-down, step-up-down) and estimation of the number of true null hypotheses. The AFDR procedure differs from the standard FDR procedure in two respects, i.e., enhancing the efficiency by reducing the number of tested hypotheses and improving the power. The proposed procedure consists of two main steps: firstly, the signal is represented more parsimoniously in wavelet domain; secondly, a most appropriate stepwise FDR procedure is selected according to the character of wavelet coefficients. Both the numerical simulation results and the experimental results for bearing defect diagnosis show that the proposed approach is a competitive shrinkage method compared with other popular techniques.

Detecting dyads of related individuals in large collections of DNA‐profiles by controlling the false discovery rate

The search for pairs (dyads) of related individuals in large databases of DNA-profiles has become an increasingly important inference tool in ecology. However, the many, partly dependent, pairwise comparisons introduce statistical issues. We show that the false discovery rate (FDR) procedure is well suited to control for the proportion of false positives, i.e. dyads consisting of unrelated individuals, which under normal circumstances would have been labelled as related individuals. We verify the behaviour of the standard FDR procedure by simulation, demonstrating that the FDR procedure works satisfactory in spite of the many dependent pairwise comparisons involved in an exhaustive database screening. A computer program that implements this method is available online. In addition, we propose to implement a second stage in the procedure, in which additional independent genetic markers are used to identify the false positives. We demonstrate the application of the approach in an analysis of a DNA database consisting of 3300 individual minke whales (Balaenoptera acutorostrata) each typed at ten microsatellite loci. Applying the standard procedure with an FDR of 50% led to the identification of 74 putative dyads of 1st- or 2nd-order relatives. However, introducing the second step, which involved additional genotypes at 15 microsatellite loci, revealed that only 21 of the putative dyads can be claimed with high certainty to be true dyads.

Empirical null and false discovery rate inference for exponential families

In large scale multiple testing, the use of an empirical null distribution rather than the theoretical null distribution can be critical for correct inference. This paper proposes a ``mode matching'' method for fitting an empirical null when the theoretical null belongs to any exponential family. Based on the central matching method for $z$-scores, mode matching estimates the null density by fitting an appropriate exponential family to the histogram of the test statistics by Poisson regression in a region surrounding the mode. The empirical null estimate is then used to estimate local and tail false discovery rate (FDR) for inference. Delta-method covariance formulas and approximate asymptotic bias formulas are provided, as well as simulation studies of the effect of the tuning parameters of the procedure on the bias-variance trade-off. The standard FDR estimates are found to be biased down at the far tails. Correlation between test statistics is taken into account in the covariance estimates, providing a generalization of Efron's ``wing function'' for exponential families. Applications with $\chi^2$ statistics are shown in a family-based genome-wide association study from the Framingham Heart Study and an anatomical brain imaging study of dyslexia in children.

Adaptive Choice of the Number of Bootstrap Samples in Large Scale Multiple Testing

It is a common practice to use resampling methods such as the bootstrap for calculating the p-value for each test when performing large scale multiple testing. The precision of the bootstrap p-values and that of the false discovery rate (FDR) relies on the number of bootstraps used for testing each hypothesis. Clearly, the larger the number of bootstraps the better the precision. However, the required number of bootstraps can be computationally burdensome, and it multiplies the number of tests to be performed. Further adding to the computational challenge is that in some applications the calculation of the test statistic itself may require considerable computation time. As technology improves one can expect the dimension of the problem to increase as well. For instance, during the early days of microarray technology, the number of probes on a cDNA chip was less than 10,000. Now the Affymetrix chips come with over 50,000 probes per chip. Motivated by this important need, we developed a simple adaptive bootstrap methodology for large scale multiple testing, which reduces the total number of bootstrap calculations while ensuring the control of the FDR. The proposed algorithm results in a substantial reduction in the number of bootstrap samples. Based on a simulation study we found that, relative to the number of bootstraps required for the Benjamini-Hochberg (BH) procedure, the standard FDR methodology which was the proposed methodology achieved a very substantial reduction in the number of bootstraps. In some cases the new algorithm required as little as 1/6th the number of bootstraps as the conventional BH procedure. Thus, if the conventional BH procedure used 1,000 bootstraps, then the proposed method required only 160 bootstraps. This methodology has been implemented for time-course/dose-response data in our software, ORIOGEN, which is available from the authors upon request.

Detecting differential expression in microarray data: comparison of optimal procedures.

BackgroundMany procedures for finding differentially expressed genes in microarray data are based on classical or modified t-statistics. Due to multiple testing considerations, the false discovery rate (FDR) is the key tool for assessing the significance of these test statistics. Two recent papers have generalized two aspects: Storey et al. (2005) have introduced a likelihood ratio test statistic for two-sample situations that has desirable theoretical properties (optimal discovery procedure, ODP), but uses standard FDR assessment; Ploner et al. (2006) have introduced a multivariate local FDR that allows incorporation of standard error information, but uses the standard t-statistic (fdr2d). The relationship and relative performance of these methods in two-sample comparisons is currently unknown.MethodsUsing simulated and real datasets, we compare the ODP and fdr2d procedures. We also introduce a new procedure called S2d that combines the ODP test statistic with the extended FDR assessment of fdr2d.ResultsFor both simulated and real datasets, fdr2d performs better than ODP. As expected, both methods perform better than a standard t-statistic with standard local FDR. The new procedure S2d performs as well as fdr2d on simulated data, but performs better on the real data sets.ConclusionThe ODP can be improved by including the standard error information as in fdr2d. This means that the optimality enjoyed in theory by ODP does not hold for the estimated version that has to be used in practice. The new procedure S2d has a slight advantage over fdr2d, which has to be balanced against a significantly higher computational effort and a less intuititive test statistic.

Estimation of false discovery proportion under general dependence

Wide-scale correlations between genes are commonly observed in gene expression data, due to both biological and technical reasons. These correlations increase the variability of the standard estimate of the false discovery rate (FDR). We highlight the false discovery proportion (FDP, instead of the FDR) as the suitable quantity for assessing differential expression in microarray data, demonstrate the deleterious effects of correlation on FDP estimation and propose an improved estimation method that accounts for the correlations. We analyse the variation pattern of the distribution of test statistics under permutation using the singular value decomposition. The results suggest a latent FDR model that accounts for the effects of correlation, and is statistically closer to the FDP. We develop a procedure for estimating the latent FDR (ELF) based on a Poisson regression model. For simulated data based on the correlation structure of real datasets, we find that ELF performs substantially better than the standard FDR approach in estimating the FDP. We illustrate the use of ELF in the analysis of breast cancer and lymphoma data. R code to perform ELF is available in http://www.meb.ki.se/~yudpaw.

A note on using permutation-based false discovery rate estimates to compare different analysis methods for microarray data

False discovery rate (FDR) is defined as the expected percentage of false positives among all the claimed positives. In practice, with the true FDR unknown, an estimated FDR can serve as a criterion to evaluate the performance of various statistical methods under the condition that the estimated FDR approximates the true FDR well, or at least, it does not improperly favor or disfavor any particular method. Permutation methods have become popular to estimate FDR in genomic studies. The purpose of this paper is 2-fold. First, we investigate theoretically and empirically whether the standard permutation-based FDR estimator is biased, and if so, whether the bias inappropriately favors or disfavors any method. Second, we propose a simple modification of the standard permutation to yield a better FDR estimator, which can in turn serve as a more fair criterion to evaluate various statistical methods. Both simulated and real data examples are used for illustration and comparison. Three commonly used test statistics, the sample mean, SAM statistic and Student's t-statistic, are considered. The results show that the standard permutation method overestimates FDR. The overestimation is the most severe for the sample mean statistic while the least for the t-statistic with the SAM-statistic lying between the two extremes, suggesting that one has to be cautious when using the standard permutation-based FDR estimates to evaluate various statistical methods. In addition, our proposed FDR estimation method is simple and outperforms the standard method.

Nonparametric Hypothesis Testing for a Spatial Signal

Nonparametric hypothesis testing for a spatial signal can involve a large number of hypotheses. For instance, two satellite images of the same scene, taken before and after an event, could be used to test a hypothesis that the event has no environmental impact. This is equivalent to testing that the mean difference of “after-before” is zero at each of the (typically thousands of) pixels that make up the scene. In such a situation, conventional testing procedures that control the overall type I error deteriorate as the number of hypotheses increases. Powerful testing procedures are needed for this problem of testing for the presence of a spatial signal. In this article we propose a procedure called enhanced false discovery rate (EFDR), which is based on controlling the false discovery rate (FDR) and a concept known as generalized degrees of freedom. EFDR differs from the standard FDR procedure by reducing the number of hypotheses tested. This is done in two ways: first, the model is represented more parsimoniously in the wavelet domain; second, an optimal selection of hypotheses is made using a criterion based on generalized degrees of freedom. Not only does the EFDR procedure tell us whether a spatial signal is present, but, if a signal is deemed present, it can also indicate its location and magnitude. We examine EFDR's operating characteristics and in simulations show that it outperforms the standard FDR and conventional testing procedures. Finally, we apply the EFDR procedure to an air temperature dataset generated from the climate system model of the National Center for Atmospheric Research, in which air temperatures in the 1980s are compared to those in the 1990s. We conclude that temperature change has occurred between the two decades, mostly warming in the central part of the United States and in coastal regions of South America at about 20°S.