Class Of Multiple Testing Procedures Research Articles

Differentially private false discovery rate control

Differential privacy provides a rigorous framework for privacy-preserving data analysis. This paper proposes the first differentially private procedure for controlling the false discovery rate (FDR) in multiple hypothesis testing. Inspired by the Benjamini-Hochberg procedure (BHq), our approach is to first repeatedly add noise to the logarithms of the p-values to ensure differential privacy and to select an approximately smallest p-value serving as a promising candidate at each iteration; the selected p-values are further supplied to the BHq and our private procedure releases only the rejected ones. Moreover, we develop a new technique that is based on a backward submartingale for proving FDR control of a broad class of multiple testing procedures, including our private procedure, and both the BHq step- up and step-down procedures. As a novel aspect, the proof works for arbitrary dependence between the true null and false null test statistics, while FDR control is maintained up to a small multiplicative factor.

An enhanced mixture method for constructing gatekeeping procedures in clinical trials

ABSTRACTIt is increasingly common to encounter complex multiplicity problems with several multiplicity components in confirmatory Phase III clinical trials. These components are often based on several endpoints (primary and secondary endpoints) and several dose-control comparisons. When constructing a multiplicity adjustment in these settings, it is important to control the Type I error rate over all multiplicity components. An important class of multiple testing procedures, known as gatekeeping procedures, was derived using the mixture method that enables clinical trial sponsors to set up efficient multiplicity adjustments that account for clinically relevant logical relationships among the hypotheses of interest. An enhanced version of this mixture method is introduced in this paper to construct more powerful gatekeeping procedures for a specific type of logical relationships that rely on transitive serial restrictions. Restrictions of this kind are very common in Phase III clinical trials and the proposed method is applicable to a broad class of multiplicity problems. Several examples are provided to illustrate the new method and results of simulation trials are presented to compare the performance of gatekeeping procedures derived using this method and other available methods.

Closure properties of classes of multiple testing procedures

Statistical discoveries are often obtained through multiple hypothesis testing. A variety of procedures exists to evaluate multiple hypotheses, for instance the ones of Benjamini–Hochberg, Bonferroni, Holm or Sidak. We are particularly interested in multiple testing procedures with two desired properties: (solely) monotonic and well-behaved procedures. This article investigates to which extent the classes of (monotonic or well-behaved) multiple testing procedures, in particular the subclasses of so-called step-up and step-down procedures, are closed under basic set operations, specifically the union, intersection, difference and the complement of sets of rejected or non-rejected hypotheses. The present article proves two main results: First, taking the union or intersection of arbitrary (monotonic or well-behaved) multiple testing procedures results in new procedures which are monotonic but not well-behaved, whereas the complement or difference generally preserves neither property. Second, the two classes of (solely monotonic or well-behaved) step-up and step-down procedures are closed under taking the union or intersection, but not the complement or difference.

Multiple testing procedures for adaptive enrichment designs: combining group sequential and reallocation approaches.

Adaptive enrichment designs involve preplanned rules for modifying enrollment criteria based on accrued data in an ongoing trial. For example, enrollment of a subpopulation where there is sufficient evidence of treatment efficacy, futility, or harm could be stopped, while enrollment for the remaining subpopulations is continued. We propose a new class of multiple testing procedures tailored to adaptive enrichment designs. The procedures synthesize ideas from two general approaches. As in the modified group sequential approach, the procedures gain power by leveraging the covariance among statistics for different stages and different hypotheses. As in the alpha reallocation approach, the procedures lower rejection thresholds for the remaining null hypotheses after others have been rejected. The proposed procedures are proved to have power greater than or equal to several existing methods, and to strongly control the familywise Type I error rate when statistics are normally distributed. The methods are illustrated through simulations of a trial for a surgical intervention for stroke, involving two subpopulations.

MMCTest—A Safe Algorithm for Implementing Multiple Monte Carlo Tests

ABSTRACTConsider testing multiple hypotheses using tests that can only be evaluated by simulation, such as permutation tests or bootstrap tests. This article introduces MMCTest, a sequential algorithm that gives, with arbitrarily high probability, the same classification as a specific multiple testing procedure applied to ideal p‐values. The method can be used with a class of multiple testing procedures that include the Benjamini and Hochberg false discovery rate procedure and the Bonferroni correction controlling the familywise error rate. One of the key features of the algorithm is that it stops sampling for all the hypotheses that can already be decided as being rejected or non‐rejected. MMCTest can be interrupted at any stage and then returns three sets of hypotheses: the rejected, the non‐rejected and the undecided hypotheses. A simulation study motivated by actual biological data shows that MMCTest is usable in practice and that, despite the additional guarantee, it can be computationally more efficient than other methods.

Selection of Hypothesis Weights and Ordering When Testing Multiple Hypotheses in Clinical Trials

This article discusses the problem of selecting free parameters of multiple testing procedures in confirmatory Phase III clinical trials with multiple objectives, including hypothesis weights and hypothesis ordering. We identify classes of multiple testing procedures that provide different interpretations of these parameters. This includes basic single-step procedures (Bonferroni procedure) that employ fixed hypothesis weights, as well as more powerful stepwise procedures (Holm, fallback, and chain procedures) that reweight the hypotheses during the testing process. We examine the behavior of different classes of multiple testing procedures in problems with unequally weighted hypotheses and a priori ordered hypotheses and provide practical guidelines for the choice of hypothesis weights and hypothesis ordering. The concepts discussed in the article are illustrated using case studies based on clinical trials with multiple endpoints, multiple dose–placebo comparisons, and multiple patient populations.

Key multiplicity issues in clinical drug development

Much progress has been made over the past decade with the development of novel methods for addressing increasingly more complex multiplicity problems arising in confirmatory Phase III clinical trials. This includes traditional problems with a single source of multiplicity, for example, analysis of multiple endpoints or dose-placebo contrasts. In addition, more advanced problems with several sources of multiplicity have attracted attention in clinical drug development. These problems include two or more families of objectives such as multiple endpoints evaluated at multiple dose levels or in multiple patient populations. This paper provides a review of concepts that play a central role in defining and solving multiplicity problems (error rate definitions) and introduces main classes of multiple testing procedures widely used in clinical trials (nonparametric, semiparametric, and parametric procedures). The paper also presents recent advances in multiplicity research, including gatekeeping procedures for clinical trials with multiple sets of objectives. The concepts and methods introduced in the paper are illustrated using several case studies on the basis of real clinical trials. Software implementation of commonly used multiple testing and gatekeeping procedures is discussed.

Superchain procedures in clinical trials with multiple objectives

This paper introduces a new class of multiple testing procedures for addressing multiplicity problems arising in clinical trials with multiple objectives grouped into families. The families may correspond to equally important sets of objectives (co-primary endpoints) or ordered sets of objectives (primary and secondary endpoints). The procedures, termed superchain procedures, serve as an extension of several classes of other multiple testing procedures, including chain procedures and parallel gatekeeping procedures. Superchain procedures exhibit several desirable features, including flexible decision rules that can be tailored to a broad class of win criteria in confirmatory clinical trials. Additionally, superchain procedures enable the trial's sponsor to efficiently incorporate available distributional information to improve overall power. Finally, superchain procedures rely on straightforward testing algorithms, which facilitates their development. We illustrate the new methodology by using clinical trial examples with two and three families of objectives.

Chain Procedures: A Class of Flexible Closed Testing Procedures With Clinical Trial Applications

We define a class of multiple testing procedures for testing a family of hypotheses based on a prespecified or data-driven testing sequence. These procedures, termed chain procedures, are characterized by independent sets of parameters which govern the initial allocation of the overall α level among the null hypotheses of interest and the process for iteratively reallocating available (or unspent) α among the remaining eligible null hypotheses. As a result, chain procedures are more flexible than popular stepwise procedures such as the Holm or fallback procedures. While presenting the broad class of chain procedures, this article focuses on the development of parametric chain procedures for problems with a known joint distribution of the hypothesis test statistics. Chain procedures are closed testing procedures and thus control the familywise error rate in the strong sense. Further, we discuss optimal selection of parameters of chain procedures based on clinically relevant application-specific criteria. Finally, we illustrate application of the chain testing method using a clinical trial example aimed at the development of a tailored therapy.

Multiple Testing with Minimal Assumptions

Resampling-based multiple testing methods that control the Familywise Error Rate in the strong sense are presented. It is shown that no assumptions whatsoever on the data-generating process are required to obtain a reasonably powerful and flexible class of multiple testing procedures. Improvements are obtained with mild assumptions. The methods are applicable to gene expression data in particular, but more generally to any multivariate, multiple group data that may be character or numeric. The role of the disputed "subset pivotality" condition is clarified.

Tree‐structured gatekeeping tests in clinical trials with hierarchically ordered multiple objectives

This paper discusses a new class of multiple testing procedures, tree-structured gatekeeping procedures, with clinical trial applications. These procedures arise in clinical trials with hierarchically ordered multiple objectives, for example, in the context of multiple dose-control tests with logical restrictions or analysis of multiple endpoints. The proposed approach is based on the principle of closed testing and generalizes the serial and parallel gatekeeping approaches developed by Westfall and Krishen (J. Statist. Planning Infer. 2001; 99:25-41) and Dmitrienko et al. (Statist. Med. 2003; 22:2387-2400). The proposed testing methodology is illustrated using a clinical trial with multiple endpoints (primary, secondary and tertiary) and multiple objectives (superiority and non-inferiority testing) as well as a dose-finding trial with multiple endpoints.