Number Of False Rejections Research Articles

Variability and stability of the false discovery proportion

Much effort has been done to control the “false discovery rate” (FDR) when $m$ hypotheses are tested simultaneously. The FDR is the expectation of the “false discovery proportion” $\text{FDP}=V/R$ given by the ratio of the number of false rejections $V$ and all rejections $R$. In this paper, we have a closer look at the FDP for adaptive linear step-up multiple tests. These tests extend the well known Benjamini and Hochberg test by estimating the unknown amount $m_{0}$ of the true null hypotheses. We give exact finite sample formulas for higher moments of the FDP and, in particular, for its variance. Using these allows us a precise discussion about the stability of the FDP, i.e., when the FDP is asymptotically close to its mean. We present sufficient and necessary conditions for this stability. They include the presence of a stable estimator for the proportion $m_{0}/m$. We apply our results to convex combinations of generalized Storey type estimators with various tuning parameters and (possibly) data-driven weights. The corresponding step-up tests allow a flexible adaptation. Moreover, these tests control the FDR at finite sample size. We compare these tests to the classical Benjamini and Hochberg test and discuss the advantages of them.

Multiple Comparisons Controlling Expected Number of False Discoveries

In this paper, it is put forward that the task of designing a procedure for a set of multiple comparisons should be considered as a decision-making under uncertainty. Due to this motivation, for the problem of multiple comparisons, we considered another error rate to be controlled, called PFER (per-family error rate), which requests that the expected number of false rejections of a test procedure should be bounded no more than a prespecified level k. Although PFER was proposed by Tukey in 1953, there is not much studying about it so far. We first present Bonferroni procedure (single-step) and then build two step-up procedures with one having generic critical values and another using critical values in BH (Benjamini and Hochberg) type. These procedures are compared through simulations.

Error, power, and cluster separation rates of pairwise multiple testing procedures.

In comparing multiple treatments, 2 error rates that have been studied extensively are the familywise and false discovery rates. Different methods are used to control each of these rates. Yet, it is rare to find studies that compare the same methods on both of these rates, and also on the per-family error rate, the expected number of false rejections. Although the per-family error rate and the familywise error rate are similar in most applications when the latter is controlled at a conventional low level (e.g., .05), the 2 measures can diverge considerably with methods that control the false discovery rate at that same level. Furthermore, we shall consider both rejections of true hypotheses (Type I errors) and rejections of false hypotheses where the observed outcomes are in the incorrect direction (Type III errors). We point out that power estimates based on the number of correct rejections do not consider the pattern of those rejections, which is important in interpreting the total outcome. The present study introduces measures of interpretability based on the pattern of separation of treatments into nonoverlapping sets and compares methods on these measures. In general, range-based (configural) methods are more likely to obtain interpretable patterns based on treatment separation than individual p-value-based measures. Recommendations for practice based on these results are given in the article. Although the article is complex, these recommendations can be understood without the necessity for detailed perusal of the supporting material.

A Package to Study the Performance of Step-Up and Step-Down Test Procedures

The package can be used to analyze the performance of step-up and step-down procedures. It can be used to compare powers, calculate the “false discovery rate”, to study the effects of reduced step procedures, and to calculate P [U ≤ k], where U is the number of rejected true hypotheses. It can be used to determine the maximum number of steps that can be made and still guarantee (with a given probability) that the number of false rejections will not exceed some specified number. The test statistics are assumed to have a multivariate-t distribution. Examples are included.

The beta-binomial distribution for estimating the number of false rejections in microarray gene expression studies

In differential expression analysis of microarray data, it is common to assume independence among null hypotheses (and thus gene expression levels). The independence assumption implies that the number of false rejections V follows a binomial distribution and leads to an estimator of the empirical false discovery rate (eFDR). The number of false rejections V is modeled with the beta-binomial distribution. An estimator of the beta-binomial false discovery rate (bbFDR) is then derived. This approach accounts for how the correlation among non-differentially expressed genes influences the distribution of V . Permutations are used to generate the observed values for V under the null hypotheses and a beta-binomial distribution is fit to the values of V . The bbFDR estimator is compared to the eFDR estimator in simulation studies of correlated non-differentially expressed genes and is found to outperform the eFDR for certain scenarios. As an example, this method is also used to perform an analysis that compares the gene expression of soft-tissue sarcoma samples to normal-tissue samples.

Exact Calculations of Average Power for the Benjamini-Hochberg Procedure

Exact analytic expressions are developed for the average power of the Benjamini and Hochberg false discovery control procedure. The result is based on explicit computation of the joint probability distribution of the total number of rejections and the number of false rejections, and expressed in terms of the cumulative distribution functions of the p-values of the hypotheses. An example of analytic evaluation of the average power is given. The result is confirmed by numerical experiments and applied to a meta-analysis of three clinical studies in mammography.

Using Monte Carlo Methods to Teach Statistics: The MC2G Computer Program

The MC2G computer program (Brooks, 2003) performs Monte Carlo analyses for several statistical tests used with 1 or 2 groups, including t tests, correlations, and nonparametric tests. Several types of results can be generated by MC2G. In particular, if the user enters population values that correspond to a true null hypothesis, the program will calculate Type I error rates based on the number of false rejections in the Monte Carlo samples simulated. If the input values correspond to a false null hypothesis, the program calculates statistical power through Monte Carlo methods. The program also can be used to investigate the sampling distribution of the mean and to determine sample sizes for various population conditions. Results can be viewed on the screen and can be saved to a text file. Both simulated sample data and sample statistics may be saved to comma-delimited text files. Although the program was designed as an instructional tool both for introductory statistics students and for advanced students learning about Monte Carlo methods, it is also suitable for research purposes.

The use of random spliced speech for the recognition of familiar voices

Various techniques for experimental speech signal manipulation to test recognition of familiar voices have been used, but random spliced speech (RSS) has not been tried. This procedure, which consists of randomly recombining digitally segmented speech signals, offers the advantage of more natural-sounding speech and decreased risk of creating artifacts interfering with perception. Although intelligibility is also destroyed by this technique, RSS preserves the spectral cues related to voice quality, as well as F0 level and range. In the present experiment, various 3.5 s RSS samples of 11 males were presented to 34 listeners, each familiar with only one of the voices. In a recognition task with forced yes/no responses, the index of correct identification was well above chance. An analysis of misindentifications revealed that proximity of F0 means led to more errors than did similarity of long term average spectra. Since speakers with F0 mean less than 2.8 semitones apart were frequently confused, it seems that in such an experimental situation the listeners may be forced to rely on isolated cues, such as F0; this may explain the high number of false identifications, even though the number of false rejections is practically nonexistent.