Resampling-based Multiple Testing Research Articles

Resampling-based stepwise multiple testing procedures with applications to clinical trial data.

Endpoints in clinical trials are often highly correlated. However, the commonly used multiple testing procedures in clinical trials either do not take into consideration the correlations among test statistics or can only exploit known correlations. Westfall and Young constructed a resampling-based stepdown method that implicitly utilizes the correlation structure of test statistics in situations with unknown correlations. However, their method requires a "subset pivotality" assumption. Romano and Wolf proposed a more general stepdown method, which does not require such an assumption. There is at present little experience with the application of such methods in analyzing clinical trial data. We advocate the application of resampling-based multiple testing procedures to clinical trials data when appropriate. We have conjectured that the resampling-based stepdown methods can be extended to a stepup procedure under appropriate assumptions and examined the performance of both stepdown and stepup methods under a variety of correlation structures and distribution types. Results from our simulation studies support the use of the resampling-based methods under various scenarios, including binary data and small samples, with strong control of Family wise type I error rate (FWER). Under positive dependence and for binary data even under independence, the resampling-based methods are more powerful than the Holm and Hochberg methods. Last, we illustrate the advantage of the resampling-based stepwise methods with two clinical trial data examples: a cardiovascular outcome trial and an oncology trial.

AP-1/Fos-TGase2 Axis Mediates Wounding-induced Plasmodium falciparum Killing in Anopheles gambiae

Anopheline mosquitoes are the only vectors of human malaria worldwide. It is now widely accepted that mosquito immune responses play a crucial role in restricting Plasmodium development within the vector; therefore, further dissection of the molecular mechanisms underlying these processes should inform new vector control strategies urgently needed to roll back the disease. Here, using genome-wide transcriptional profiling, bioinformatics, and functional gene analysis, we identify a new axis of mosquito resistance to monoclonal Plasmodium falciparum infections that includes the AP-1 transcription factor Fos and the transglutaminase 2 (TGase2), a cross-linking enzyme with known roles in wound responses. We demonstrate that Fos regulates induction of TGase2 expression after wounding but does not affect expression of the components of the well characterized complement-like system. Silencing of Fos or of TGase2 aborts the wounding-induced mosquito killing of P. falciparum. These results reveal multiple signaling pathways that are required for efficient Plasmodium killing in Anopheles gambiae.

Power and Stability Properties of Resampling-Based Multiple Testing Procedures with Applications to Gene Oncology Studies

Resampling-based multiple testing procedures are widely used in genomic studies to identify differentially expressed genes and to conduct genome-wide association studies. However, the power and stability properties of these popular resampling-based multiple testing procedures have not been extensively evaluated. Our study focuses on investigating the power and stability of seven resampling-based multiple testing procedures frequently used in high-throughput data analysis for small sample size data through simulations and gene oncology examples. The bootstrap single-step minP procedure and the bootstrap step-down minP procedure perform the best among all tested procedures, when sample size is as small as 3 in each group and either familywise error rate or false discovery rate control is desired. When sample size increases to 12 and false discovery rate control is desired, the permutation maxT procedure and the permutation minP procedure perform best. Our results provide guidance for high-throughput data analysis when sample size is small.

Stationarity of Asian real exchange rates: An empirical application of multiple testing to nonstationary panels with a structural break

This paper investigates the stationarity of the real exchange rates of the currencies for ten Asian countries against the US dollar during the pre-Lehman period. This paper explicitly investigates the presence of a structural break that occurred at unknown dates across countries and which may have been caused by the Asian financial crisis in 1997–1998. To identify which of the ten countries hold real exchange rate stationarity, that is, long-run Purchasing Power Parity, the resampling-based multiple testing proposed by Romano and Wolf (2005) is employed while dealing with possible cross-sectional correlation among the countries and avoiding the over-rejection of the null hypothesis or the multiplicity problem. Moreover, the paper examines the small-sample property of the multiple testing when there is a structural break in cross-sectionally dependent panels. Finally, the empirical results show that the stationarity hypothesis of the real exchange rate is significantly supported in some Asian countries.

Statistical properties of an early stopping rule for resampling-based multiple testing

Resampling-based methods for multiple hypothesis testing often lead to long run times when the number of tests is large. This paper presents a simple rule that substantially reduces computation by allowing resampling to terminate early on a subset of tests. We prove that the method has a low probability of obtaining a set of rejected hypotheses different from those rejected without early stopping, and obtain error bounds for multiple hypothesis testing. Simulation shows that our approach saves more computation than other available procedures.

Assessing group differences in biodiversity by simultaneously testing a user‐defined selection of diversity indices

Comparing diversities between groups is a task biologists are frequently faced with, for example in ecological field trials or when dealing with metagenomics data. However, researchers often waver about which measure of diversity to choose as there is a multitude of approaches available. As Jost (2008, Molecular Ecology, 17, 4015) has pointed out, widely used measures such as the Shannon or Simpson index have undesirable properties which make them hard to compare and interpret. Many of the problems associated with the use of these 'raw' indices can be corrected by transforming them into 'true' diversity measures. We introduce a technique that allows the comparison of two or more groups of observations and simultaneously tests a user-defined selection of a number of 'true' diversity measures. This procedure yields multiplicity-adjusted P-values according to the method of Westfall and Young (1993, Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment, 49, 941), which ensures that the rate of false positives (type I error) does not rise when the number of groups and/or diversity indices is extended. Software is available in the R package 'simboot'.

The sequential rejection principle of familywise error control

Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.

Resampling-based and other multiple testing strategies with application to combination drug trials with factorial designs

The problem of detecting all effective and superior combinations in a factorial drug efficacy trial is stated in terms of two hypothesis families, full and reduced. The reduced problem formulation allows identification of all simultaneously effective and superior combinations. The full formulation allows individual detection of the efficacy and superiority of combinations resulting in more detailed conclusions. While the full hypothesis family deals with simpler parameters, the true mean effect differences, it has three times as many hypotheses as the reduced family. The reduced family is comprised of hypotheses concerning a gain-parameter, which is defined as the minimum of the true mean differences and leading to a fairly complicated structure. For each problem formulation, Holm's, Hochberg's and two resampling approaches are studied with respect to strong control of overall error rate and several power measures. Holm's and Hochberg's approaches are recommended for the reduced family, while the step-down resampling approach is recommended for the full hypothesis family. Moreover, the correct problem formulation is of great importance, because if the sample size is not sufficient there is a high potential for lack of power associated with the full problem due to a large number of hypotheses.

Comments on: Control of the false discovery rate under dependence using the bootstrap and subsampling

In this enlightening and stimulating paper, Professors Romano, Shaikh, and Wolf construct two novel resampling-based multiple testing methods using the bootstrap and subsampling techniques and theoretically prove that these methods approximately control the FDR under weak regularity conditions. The theoretical results provide a satisfactory solution to an important and challenging problem in multiple testing on developments of FDR controlling procedures by exploiting unknown dependence among the test statistics using resampling techniques. In my comments, I address the related statistical and computational issues when applying their bootstrap method to analyze high-dimensional, low sample size data such as microarray data and suggest several possible extensions.

Multiple Testing with Minimal Assumptions

Resampling-based multiple testing methods that control the Familywise Error Rate in the strong sense are presented. It is shown that no assumptions whatsoever on the data-generating process are required to obtain a reasonably powerful and flexible class of multiple testing procedures. Improvements are obtained with mild assumptions. The methods are applicable to gene expression data in particular, but more generally to any multivariate, multiple group data that may be character or numeric. The role of the disputed "subset pivotality" condition is clarified.

Tumor Suppressor p16INK4A Regulates Polycomb-mediated DNA Hypermethylation in Human Mammary Epithelial Cells

Alterations in DNA methylation are important in cancer, but the acquisition of these alterations is poorly understood. Using an unbiased global screen for CpG island methylation events, we have identified a non-random pattern of DNA hypermethylation acquired in p16-repressed cells. Interestingly, this pattern included loci located upstream of a number of homeobox genes. Upon removal of p16(INK4A) activity in primary human mammary epithelial cells, polycomb repressors, EZH2 and SUZ12, are up-regulated and recruited to HOXA9, a locus expressed during normal breast development and epigenetically silenced in breast cancer. We demonstrate that at this targeted locus, the up-regulation of polycomb repressors is accompanied by the recruitment of DNA methyltransferases and the hypermethylation of DNA, an endpoint, which we show to be dependent on SUZ12 expression. These results demonstrate a causal role of p16(INK4A) disruption in modulating DNA hypermethylation, and identify a dynamic and active process whereby epigenetic modulation of gene expression is activated as an early event in breast tumor progression.

Kernel estimation for adjusted [formula omitted]-values in multiple testing

Multiple testing procedures are frequently applied to biomedical and genomic research, for instance, identification of differentially expressed genes in microarray experiments. Resampling methods are commonly used to compute adjusted p -values in multiple hypothesis testing problems. Importantly, the resampling-based multiple testing procedures are sensitive to the number of permutations, especially for the MinP adjustment procedure. The single-step MinP adjusted p -values are derived from the distribution of the minimum of the p -values. Because of computational complexity, the adjusted p -values are often computed using the distribution of the maximum of the test statistics (MaxT). This paper proposes an approach based on the kernel density estimation (KDE) technique to reduce the number of permutations for implementing the single-step MinP adjustment. Simulation studies are conducted to demonstrate that the KDE method is more powerful than the MinP adjustment method under independent and correlated models. The three resampling-based single-step adjustment procedures, MaxT, MinP, and KDE, are applied to two published microarray data sets, the colon tumor data set consisting of 40 tumor and 22 normal colon tissue samples on 2000 human genes (endpoints) and the leukemia data set consisting of 27 acute lymphoblastic leukemia and 11 acute myeloid leukemia samples on 3051 genes. The MaxT adjusted p -values are very robust to the number of permutations. The MaxT adjusted p -values are stable with 10,000 permutations, while the MinP adjusted p -values are step functions. As the number of permutations increases, the number of ties decrease. The adjusted p -values are stable with 500,000 permutations. For the KDE method, the adjusted p -values are stable at 50,000 permutations. At 1000,000 permutations, the three procedures have similar adjusted p -values.

The File-Drawer Problem, Revisited

It has been contended that drug companies have hidden, in the “file drawer,” the results of unsuccessful clinical trials while publishing the results of more successful trials ([1][1]). As others have pointed out, no one's interest (drug company, doctor, patient) is served by the marketing of ineffective drugs, but nevertheless, the scientific enterprise is set back when full results are not published. The remedy proposed is to make all clinical trials data available (“The old file-drawer problem,” D. Kennedy, Editorial, 23 July, p. [451][2]); indeed, some companies are already planning to post their clinical trials data. If there is minor mischief in the drug company file-drawer, there is major mischief with many epidemiology studies. Although there are legitimate reservations ([2][3]), why shouldn't epidemiologists also open their data sets ([3][4])? Arguably, the scientific need is greater than that for clinical trials. Whereas clinical trials test prespecified, well-defined hypotheses, in epidemiological investigations, hundreds to thousands of questions may be asked, but only those results that pass the minimal threshold of “statistically significant, P < 0.05” are reported. Statistical analyses are often not adjusted to reflect multiple testing, even though procedures are readily available ([4][5]). If hundreds to thousands of results are locked in the epidemiology file drawer, even readers who wish to make the adjustment for themselves cannot. The worst-case scenario is that the vast majority of results reported from epidemiology studies are false positives. To cite (but not single out) only one example of epidemiology file-drawer cases, antibiotic use was associated with breast cancer ([5][6]). The antibiotic-breast cancer association was culled from a very large search process. If analyses were adjusted to reflect the number of questions, there might well be no significant association. The central point, however, is that keeping these results in the file drawer prevents others from doing alternative analyses, and again the loser is the scientific enterprise. 1. 1.[↵][7] 1. L. Abboud , “Lilly plans broad access to results on its drug trials,” Wall Street J. , 3 Aug. 2004, pp. B1-B2. 2. 2.[↵][8] 1. E. Marshall , Science 290, 28 (2000). [OpenUrl][9][PubMed][10][Web of Science][11] 3. 3.[↵][12] 1. Board on Life Sciences , Sharing Publication-Related Data and Materials: Responsibilities of Authorship in the Life Sciences (National Academies Press, Washington, DC, 2003) (see [www.nap.edu/books/0309088593/html/][13]). 4. 4.[↵][14] 1. P. H. Westfall, 2. S. S. Young , Resampling-Based Multiple Testing (Wiley, New York, 1993). 5. 5.[↵][15] 1. C. M. Velicer 2. et al. , JAMA 29, 827 (2004). [OpenUrl][16] # Response {#article-title-2} I think most would disagree with Young's characterization of the unpublished clinical trials problem as “minor mischief.” His larger point, about epidemiological trials, is well taken—but forcing these out of hiding will be administratively difficult. Epidemiological studies undertaken in connection with some regulatory procedures or a court action can be subject to required disclosure. Many and probably most, however, are undertaken in pursuit of a problem that interests the investigator. It might be useful, as Young suggests, to have all that stuff out there. Alas, we don't have a process. [1]: #ref-1 [2]: /lookup/doi/10.1126/science.305.5683.451 [3]: #ref-2 [4]: #ref-3 [5]: #ref-4 [6]: #ref-5 [7]: #xref-ref-1-1 View reference 1. in text [8]: #xref-ref-2-1 View reference 2. in text [9]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.issn%253D0036-8075%26rft.aulast%253DMarshall%26rft.auinit1%253DE.%26rft.volume%253D290%26rft.issue%253D5489%26rft.spage%253D28%26rft.epage%253D29%26rft.atitle%253DEthics.%2BEpidemiologists%2Bwary%2Bof%2Bopening%2Bup%2Btheir%2Bdata.%26rft_id%253Dinfo%253Apmid%252F11183141%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [10]: /lookup/external-ref?access_num=11183141&link_type=MED&atom=%2Fsci%2F306%2F5699%2F1133.4.atom [11]: /lookup/external-ref?access_num=000089724800008&link_type=ISI [12]: #xref-ref-3-1 View reference 3. in text [13]: http://www.nap.edu/books/0309088593/html/ [14]: #xref-ref-4-1 View reference 4. in text [15]: #xref-ref-5-1 View reference 5. in text [16]: {openurl}?query=rft.jtitle%253DJAMA%26rft.volume%253D29%26rft.spage%253D827%26rft.atitle%253DJAMA%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx

Choice of a null distribution in resampling-based multiple testing

This paper investigates different choices of test statistic null distribution for resampling-based multiple testing in the context of single parameter hypotheses. We show that the test statistic null distribution for strongly controlling type I error may be obtained by projecting the true test statistic distribution onto the space of mean zero distributions. For common choices of test statistics, this distribution is asymptotically multivariate normal with the covariance of the vector influence curve for the parameter estimator. Applying the ordinary non-parametric or model-based bootstrap to mean zero centered test statistics produces an estimated test statistic null distribution which provides asymptotic strong control. In contrast, the usual practice of obtaining an estimated test statistic null distribution via an estimated data null distribution (e.g. null restricted bootstrap) only provides an asymptotically correct test statistic null distribution if the covariance of the vector influence curve is the same under the chosen data null distribution as under the true data distribution. This condition is the formal analogue of the subset pivotality condition (Westfall and Young, Resampling-based Multiple Testing: Examples and Methods for p-value adjustment, Wiley, New York, 1993). We demonstrate the use of our proposed ordinary bootstrap null distribution with a single-step multiple testing method which is equivalent to constructing an error-specific confidence region for the true parameter and checking if it contains the hypothesized value. We also study the two sample problem and show that the permutation method produces an asymptotically correct null distribution if (i) the sample sizes are equal or (ii) the populations have the same covariance structure.

Resampling-based multiple testing for microarray data analysis

The burgeoning field of genomics has revived interest in multiple testing procedures by raising new methodological and computational challenges. For example, microarray experiments generate large multiplicity problems in which thousands of hypotheses are tested simultaneously. Westfall and Young (1993) propose resampling-based p-value adjustment procedures which are highly relevant to microarray experiments. This article discusses dierent criteria for error control in resampling-based multiple testing, including (a) the family wise error rate of Westfall and Young (1993) and (b) the false discovery rate developed by Benjamini and Hochberg (1995), both from a frequentist viewpoint; and (c) the positive false discovery rate of Storey (2002a), which has a Bayesian motivation. We also introduce our recently developed fast algorithm for implementing the minP adjustment to control family-wise error rate. Adjusted p-values for dierent approaches are applied to gene expression data from two recently published microarray studies. The properties of these procedures for multiple testing are compared.

A Permutational Step-Up Method of Testing Multiple Outcomes

This paper describes a permutational step-up multiple comparison procedure to adjust the p values from k related hypotheses. The method is applicable when two groups of subjects are being compared on each of k outcomes. It is related to the step-down method of Westfall and Young (1993, Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment, Volume 1) and Troendle (1995, Journal of the American Statistical Association 90, 370-378) and is an alternative to the analytic step-up method of Dunnett and Tamhane (1992, Journal of the American Statistical Association 87, 162-170), which requires a specific distribution and correlation structure. By conditioning on the data observed, this permutational method avoids any specific distribution or correlation assumption. It is shown very generally that the method asymptotically controls the familywise probability of a type I error.

Resampling-Based Multiple Testing

(1995). Resampling-Based Multiple Testing. Journal of Quality Technology: Vol. 27, No. 3, pp. 272-273.

Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.

Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.

Resampling-Based Multiple Testing.

Resampling-Based Multiple Testing.

Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.

Resampling-Based Multiple Testing: Examples and Methods for p-Value Adjustment.