Familywise Error Control Research Articles

Smooth Matérn Gaussian random fields: Euler characteristic, expected number and height distribution of critical points

This paper studies Gaussian random fields with Matérn covariance functions with smooth parameter ν>2. Two cases of parameter spaces, the Euclidean space and N-dimensional sphere, are considered. For such smooth Gaussian fields, we have derived the explicit formulae for the expected Euler characteristic of the excursion set, the expected number and height distribution of critical points. The results are valuable for approximating the excursion probability in family-wise error control and for computing p-values in peak inference.

Cluster extent inference revisited: quantification and localisation of brain activity

Abstract Cluster inference based on spatial extent thresholding is a popular analysis method multiple testing in spatial data, and is frequently used for finding activated brain areas in neuroimaging. However, the method has several well-known issues. While powerful for finding regions with some activation, the method as currently defined does not allow any further quantification or localisation of signal. In this paper, we repair this gap. We show that cluster-extent inference can be used (1) to infer the presence of signal in any region of interest and (2) to quantify the percentage of activation in such regions. These additional inferences come for free, i.e. they do not require any further adjustment of the alpha-level of tests, while retaining full family-wise error control. We achieve this extension of the possibilities of cluster inference by embedding the method into a closed testing procedure, and solving the graph-theoretic k-separator problem that results from this embedding. We demonstrate the usefulness of the improved method in a large-scale application to neuroimaging data from the Neurovault database.

Multiplicity Control vs Replication

This paper presents a side-by-side consideration of multiplicity control procedures and replication as solutions to the problem of multiplicity. Several independent theoretical arguments are presented which demonstrate that replication serves several important functions, and that multiplicity control procedures have a number of serious flaws. Subsequently, the results of a simulation study are provided, showing that under typical conditions, replication provides similar familywise error control and power as multiplicity control procedures. Taken together, these theoretical and statistical arguments lead to the conclusion that researchers who are concerned about the problem of multiplicity should shift their attention away from multiplicity control procedures and towards increased use of replication.

Multivariate one-sided testing in matched observational studies as an adversarial game

Summary We present a multivariate one-sided sensitivity analysis for matched observational studies, appropriate when the researcher has specified that a given causal mechanism should manifest itself in effects on multiple outcome variables in a known direction. The test statistic can be thought of as the solution to an adversarial game, where the researcher determines the best linear combination of test statistics to combat nature’s presentation of the worst-case pattern of hidden bias. The corresponding optimization problem is convex, and can be solved efficiently even for reasonably sized observational studies. Asymptotically, the test statistic converges to a chi-bar-squared distribution under the null, a common distribution in order-restricted statistical inference. The test attains the largest possible design sensitivity over a class of coherent test statistics, and facilitates one-sided sensitivity analyses for individual outcome variables while maintaining familywise error control through its incorporation into closed testing procedures.

A consistent variable screening procedure with family-wise error control

ABSTRACTVariable selection in ultra-high-dimensional data analysis is of critical importance in diverse scientific fields. There are many methods proposed for this problem in the past few decades. However, no attention has been paid on the model size produced by screening procedures and the difficulty of determining the screening threshold for practical use, which is key to real applications. Specifically, the pre-defined model size in a screening procedure lacks theoretical support, and users have no idea about the true positives and false positives in the final results. To solve this problem, we developed a new consistent independence screening procedure called HHG-CIS. The screening threshold of HHG-CIS can be set to explicitly control the family-wise error rate, thus it is easy to use in practice. Furthermore, HHG-CIS is model-free, robust to outliers, and outperforms other methods in models with interactions. We showed these advantages of HHG-CIS through extensive simulations and a real data application.

Familywise error control in multi-armed response-adaptive trials.

Response‐adaptive designs allow the randomization probabilities to change during the course of a trial based on cumulated response data so that a greater proportion of patients can be allocated to the better performing treatments. A major concern over the use of response‐adaptive designs in practice, particularly from a regulatory viewpoint, is controlling the type I error rate. In particular, we show that the naïve z‐test can have an inflated type I error rate even after applying a Bonferroni correction. Simulation studies have often been used to demonstrate error control but do not provide a guarantee. In this article, we present adaptive testing procedures for normally distributed outcomes that ensure strong familywise error control by iteratively applying the conditional invariance principle. Our approach can be used for fully sequential and block randomized trials and for a large class of adaptive randomization rules found in the literature. We show there is a high price to pay in terms of power to guarantee familywise error control for randomization schemes with extreme allocation probabilities. However, for proposed Bayesian adaptive randomization schemes in the literature, our adaptive tests maintain or increase the power of the trial compared to the z‐test. We illustrate our method using a three‐armed trial in primary hypercholesterolemia.

A more powerful familywise error control procedure for evaluating mean equivalence

When one wishes to show no meaningful differences among group means, equivalence tests should be used, as a nonsignificant test of mean difference does not provide evidence supporting equivalence. This research proposes two modified stepwise procedures for controlling the familywise Type I error rate, based on the Bonferroni-type correction of k2/4 (where k is the number of groups to be compared) proposed by Caffo, Lauzon and Rohmel (2013 Correction to “easy multiplicity control in equivalence testing using two One-Sided tests. The American Statistician 67 (2):115–6) Bonferroni-type correction of k2/4 (where k is the number of groups to be compared). Using a Monte Carlo simulation method, we show that adopting a stepwise procedure increases power, while maintaining the familywise error rate at or below α. Implications for applied research and directions for future study are discussed.

Hommel's procedure in linear time.

Hommel's and Hochberg's procedures for familywise error control are both derived as shortcuts in a closed testing procedure with the Simes local test. Hommel's shortcut is exact but takes quadratic time in the number of hypotheses. Hochberg's shortcut takes only linear time after the P-values are sorted, but is conservative. In this paper, we present an exact shortcut in linear time on sorted P-values, combining the strengths of both procedures. The novel shortcut also applies to a robust variant of Hommel's procedure that does not require the assumption of the Simes inequality.

Faster family-wise error control for neuroimaging with a parametric bootstrap.

In neuroimaging, hundreds to hundreds of thousands of tests are performed across a set of brain regions or all locations in an image. Recent studies have shown that the most common family-wise error (FWE) controlling procedures in imaging, which rely on classical mathematical inequalities or Gaussian random field theory, yield FWE rates (FWER) that are far from the nominal level. Depending on the approach used, the FWER can be exceedingly small or grossly inflated. Given the widespread use of neuroimaging as a tool for understanding neurological and psychiatric disorders, it is imperative that reliable multiple testing procedures are available. To our knowledge, only permutation joint testing procedures have been shown to reliably control the FWER at the nominal level. However, these procedures are computationally intensive due to the increasingly available large sample sizes and dimensionality of the images, and analyses can take days to complete. Here, we develop a parametric bootstrap joint testing procedure. The parametric bootstrap procedure works directly with the test statistics, which leads to much faster estimation of adjusted p-values than resampling-based procedures while reliably controlling the FWER in sample sizes available in many neuroimaging studies. We demonstrate that the procedure controls the FWER in finite samples using simulations, and present region- and voxel-wise analyses to test for sex differences in developmental trajectories of cerebral blood flow.

Higher Criticism for Large-Scale Inference, Especially for Rare and Weak Effects

In modern high-throughput data analysis, researchers perform a large number of statistical tests, expecting to find perhaps a small fraction of significant effects against a predominantly null background. Higher Criticism (HC) was introduced to determine whether there are any nonzero effects; more recently, it was applied to feature selection, where it provides a method for selecting useful predictive features from a large body of potentially useful features, among which only a rare few will prove truly useful. In this article, we review the basics of HC in both the testing and feature selection settings. HC is a flexible idea, which adapts easily to new situations; we point out simple adaptions to clique detection and bivariate outlier detection. HC, although still early in its development, is seeing increasing interest from practitioners; we illustrate this with worked examples. HC is computationally effective, which gives it a nice leverage in the increasingly more relevant "Big Data" settings we see today. We also review the underlying theoretical "ideology" behind HC. The Rare/Weak (RW) model is a theoretical framework simultaneously controlling the size and prevalence of useful/significant items among the useless/null bulk. The RW model shows that HC has important advantages over better known procedures such as False Discovery Rate (FDR) control and Family-wise Error control (FwER), in particular, certain optimality properties. We discuss the rare/weak phase diagram, a way to visualize clearly the class of RW settings where the true signals are so rare or so weak that detection and feature selection are simply impossible, and a way to understand the known optimality properties of HC.

Multiple hypothesis testing in genomics

This paper presents an overview of the current state of the art in multiple testing in genomics data from a user's perspective. We describe methods for familywise error control, false discovery rate control and false discovery proportion estimation and confidence, both conceptually and practically, and explain when to use which type of error rate. We elaborate on the assumptions underlying the methods and discuss pitfalls in the interpretation of results. In our discussion, we take into account the exploratory nature of genomics experiments, looking at selection of genes before or after testing, and at the role of validation experiments.

The Inheritance Procedure: Multiple Testing of Tree-structured Hypotheses

Hypotheses tests in bioinformatics can often be set in a tree structure in a very natural way, e.g. when tests are performed at probe, gene, and chromosome level. Exploiting this graph structure in a multiple testing procedure may result in a gain in power or increased interpretability of the results.We present the inheritance procedure, a method of familywise error control for hypotheses structured in a tree. The method starts testing at the top of the tree, following up on those branches in which it finds significant results, and following up on leaf nodes in the neighborhood of those leaves. The method is a uniform improvement over a recently proposed method by Meinshausen. The inheritance procedure has been implemented in the globaltest package which is available on www.bioconductor.org.

On familywise type I error control for multiplicity in equivalence trials with three or more treatments

For the all pairwise comparisons for equivalence of k (k≥2) treatments Lauzon and Caffo proposed simply to divide the type I error level α by k-1 to achieve a Bonferroni-based familywise error control when declaring pairs of two treatments equivalent. This rule is shown to be too liberal for k≥4. It works for k=3 yet for reasons not considered by Lauzon and Caffo. Based on the two one-sided testing procedures and using the closure test principle we develop valid alternatives based on Bonferroni's inequality. The set H of intersection hypotheses reveals a rich structure, leading to the possibility to present H as a directed acyclic graph (DAG). This in turn allows using some graph theoretical theorems and eases proving properties of the resulting multiple testing problems.

Is “X”-WAS the Future for All of Epidemiology?

Is “X”-WAS the Future for All of Epidemiology?

Familywise Error Control in Multi-Armed Response-Adaptive Two-Stage Designs

For comparing multiple treatments against a single control with normally distributed observations, we consider two-stage designs of the following form: During the first stage, control and treatments are allocated by response-adaptive randomization; after completion of the first stage, some treatments are selected to proceed to the second stage; during the second stage, control and selected treatments are allocated by block randomization. Tests for such designs that use the data from both stages have been based on simulation under the global null hypothesis. We present an approach that does not rely on simulation and protects the familywise error rate in the strong sense. The main idea is to view the trial as a data-dependent modification of a simpler design, for which we know the distributions of its test statistics. To account for the data-dependent modification, we use the conditional invariance principle (Brannath et al., 2007).

The sequential rejection principle of familywise error control

Closed testing and partitioning are recognized as fundamental principles of familywise error control. In this paper, we argue that sequential rejection can be considered equally fundamental as a general principle of multiple testing. We present a general sequentially rejective multiple testing procedure and show that many well-known familywise error controlling methods can be constructed as special cases of this procedure, among which are the procedures of Holm, Shaffer and Hochberg, parallel and serial gatekeeping procedures, modern procedures for multiple testing in graphs, resampling-based multiple testing procedures and even the closed testing and partitioning procedures themselves. We also give a general proof that sequentially rejective multiple testing procedures strongly control the familywise error if they fulfill simple criteria of monotonicity of the critical values and a limited form of weak familywise error control in each single step. The sequential rejection principle gives a novel theoretical perspective on many well-known multiple testing procedures, emphasizing the sequential aspect. Its main practical usefulness is for the development of multiple testing procedures for null hypotheses, possibly logically related, that are structured in a graph. We illustrate this by presenting a uniform improvement of a recently published procedure.

Multiplicity Control in Structural Equation Modeling

Researchers conducting structural equation modeling analyses rarely, if ever, control for the inflated probability of Type I errors when evaluating the statistical significance of multiple parameters in a model. In this study, the Type I error control, power and true model rates of famsilywise and false discovery rate controlling procedures were compared with rates when no multiplicity control was imposed. The results indicate that Type I error rates become severely inflated with no multiplicity control, but also that familywise error controlling procedures were extremely conservative and had very little power for detecting true relations. False discovery rate controlling procedures provided a compromise between no multiplicity control and strict familywise error control and with large sample sizes provided a high probability of making correct inferences regarding all the parameters in the model.

Multiplicity Control in Structural Equation Modeling

Researchers conducting structural equation modeling analyses rarely, if ever, control for the inflated probability of Type I errors when evaluating the statistical significance of multiple parameters in a model. In this study, the Type I error control, power and true model rates of famsilywise and false discovery rate controlling procedures were compared with rates when no multiplicity control was imposed. The results indicate that Type I error rates become severely inflated with no multiplicity control, but also that familywise error controlling procedures were extremely conservative and had very little power for detecting true relations. False discovery rate controlling procedures provided a compromise between no multiplicity control and strict familywise error control and with large sample sizes provided a high probability of making correct inferences regarding all the parameters in the model.

A Stepwise SPA Test for Data Snooping and its Application on Fund Performance Evaluation

In this study, we propose a Stepwise SPA Test which is powerful in searching for predictive models or profitable investment targets with appropriate family-wise error control. Our testing method, built on White's Reality Check (2000), Hansen's SPA test (2005), and Romano and Wolf's stepwise procedure (2005), aims to conduct large-scale joint hypothesis testing in a given data set. Based on the Monte Carlo simulations, we show that the proposed test is more powerful than the other three tests in limited samples. We then apply the test to examine the performance of mutual funds and hedge funds, and obtain some interesting results. We find that only eight mutual funds are found to beat the S&P 500 index, and only few hedge funds outperform the risk-free rate. With these two empirical cases, we substantiate the empirical value of our test in fund performance evaluation.

A semi-Bayesian study of Duncan's Bayesian multiple comparison procedure

Duncan's Bayesian decision-theoretic multiple comparison procedure requires a decision on the relative magnitudes of losses due to Type I and Type II errors. In this paper, the relative losses are chosen so that the procedure results in weak control of familywise error at the 0.05 level, i.e. the probability that all hypotheses are accepted is 0.95 when all hypotheses are true. Duncan's Bayesian formulation requires prior distributions and specification of associated hyperparameters for the variances of the population means and of the errors. With noninformative priors, the required ratio of these values can be estimated from the sample. From a frequentist point of view, this obviates the necessity for any prior specification for these distributions. However, Duncan's assumption of a prior normal distribution for the population means is required and is retained. A simulation study then compares the modified method, with respect to Bayes risk and average power, to several frequentist-based multiple comparison procedures for testing hypotheses concerning all pairwise comparisons among a set of means. Results indicate considerable similarity in both risk and average power between Duncan's modified procedure and the Benjamini and Hochberg (1995, J. Roy. Statist. Soc. 57, 289–300) FDR-controlling procedure, with the same weak familywise error control. Both risk and power of these procedures are close to the risk and power of individual t -tests of the mean comparisons, and considerably superior on both measures to the properties of the best symmetric simultaneous testing procedure, based on the range of normally distributed observations.