Control Of Familywise Error Rate Research Articles

A Preplanned Multi-Stage Platform Trial for Discovering Multiple Superior Treatments With Control of FWER and Power.

There is a growing interest in the implementation of platform trials, which provide the flexibility to incorporate new treatment arms during the trial and the ability to halt treatments early based on lack of benefit or observed superiority. In such trials, it can be important to ensure that error rates are controlled. This paper introduces a multi-stage design that enables the addition of new treatment arms, at any point, in a preplanned manner within a platform trial, while still maintaining control over the family-wise error rate. This paper focuses on finding the required sample size to achieve a desired level of statistical power when treatments are continued to be tested even after a superior treatment has already been found. This may be of interest if there are treatments from different sponsors which are also superior to the current control or multiple doses being tested. The calculations to determine the expected sample size is given. A motivating trial is presented in which the sample size of different configurations is studied. In addition, the approach is compared to running multiple separate trials and it is shown that in many scenarios if family-wise error rate control is needed there may not be benefit in using a platform trial when comparing the sample size of thetrial.

On a Holm-related MTP for rejecting at least k hypotheses: general validity, optimality property, confidence regions, and applications

ABSTRACT This article concerns p-value-based multiple testing procedures (MTPs) that can be used in a confirmatory clinical study under minimal assumptions in case the requirement for study-success is that at least k out of m primary/important hypotheses become rejected. Recently, a simple, generally valid Holm-type MTP was discussed that can be used for such a requirement for any k from one to m. It can only reject at least k (or zero) hypotheses, but this increases the power to reject k or more hypotheses compared to Holm’s step-down MTP. The present article provides a simple formulation and proof of strong family-wise error rate (FWER) control for a stepwise MTP that is sharper in that for any k strictly between one and m it: (a) always rejects at least as much, and (b) can potentially reject fewer than k hypotheses. This sharper MTP too is generally valid, without any assumption about logical or stochastic relationships. It has a gatekeeping step, followed by m steps where ordered primary p-values are compared to critical constants and rejections are made in a step-down manner. These constants have the optimality property that under a natural monotonicity restriction, they cannot be increased without losing the general strong FWER control. Confidence regions like those for Holm’s MTP are provided. Applications are discussed in connection with three interesting approaches proposed earlier for confirmatory studies: (a) the Superiority-Noninferiority approach; (b) Fallback tests for co-primary endpoints; and (c) Multistage gatekeeping MTPs that utilize so-called k-truncated Holm MTPs in some stages.

A BLAST from the past: revisiting blastp's E-value.

The Basic Local Alignment Search Tool, BLAST, is an indispensable tool for genomic research. BLAST established itself as the canonical tool for sequence similarity search in large part thanks to its meaningful statistical analysis. Specifically, BLAST reports the E-value of each reported alignment, which is defined as the expected number of optimal local alignments that will score at least as high as the observed alignment score, assuming that the query and the database sequences are randomly generated. Here we critically evaluate the E-values provided by the standard protein BLAST (blastp), showing that they can be at times significantly conservative while at others too liberal. We offer an alternative approach based on generating a small sample from the null distribution of random optimal alignments, and testing whether the observed alignment score is consistent with it. In contrast with blastp, our significance analysis seems valid, in the sense that it did not deliver inflated significance estimates in any of our extensive experiments. Moreover, although our method is slightly conservative, it is often significantly less so than the blastp E-value. Indeed, in cases where blastp's analysis is valid (ie, not too liberal), our approach seems to deliver a greater number of correct alignments. One advantage of our approach is that it works with any reasonable choice of substitution matrix and gap penalties, avoiding blastp's limited options of matrices and penalties. In addition, we can formulate the problem using a canonical family-wise error rate control setup, thereby dispensing with E-values, which can at times be difficult to interpret. The Apache licensed source code is available at (https://github.com/batmen-lab/SGPvalue). Supplementary data are available at Bioinformatics online.

Multiple testing of interval composite null hypotheses using randomized p-values

Equivalence tests are statistical hypothesis testing procedures that aim to establish practical equivalence rather than the usual statistical significant difference. These testing procedures are frequent in “bioequivalence studies," where one would wish to show that, for example, an existing drug and a new one under development have comparable therapeutic effects. In this article, we propose a two-stage randomized (RAND2) p-value that depends on a uniformly most powerful (UMP) p-value and an arbitrary tuning parameter c∈[0,1]\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$c\\in [0,1]$$\\end{document} for testing an interval composite null hypothesis. We investigate the behavior of the distribution function of the two p-values under the null hypothesis and alternative hypothesis for a fixed significance level t∈(0,1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$t\\in (0,1)$$\\end{document} and varying sample sizes. We evaluate the performance of the two p-values in estimating the proportion of true null hypotheses in multiple testing. We conduct a family-wise error rate control using an adaptive Bonferroni procedure with a plug-in estimator to account for the multiplicity that arises from our multiple hypotheses under consideration. The various claims in this research are verified using a simulation study and real-world data analysis.

Summary statistics knockoffs inference with family-wise error rate control.

Testing multiple hypotheses of conditional independence with provable error rate control is a fundamental problem with various applications. To infer conditional independence with family-wise error rate (FWER) control when only summary statistics of marginal dependence are accessible, we adopt GhostKnockoff to directly generate knockoff copies of summary statistics and propose a new filter to select features conditionally dependent on the response. In addition, we develop a computationally efficient algorithm to greatly reduce the computational cost of knockoff copies generation without sacrificing power and FWER control. Experiments on simulated data and a real dataset of Alzheimer's disease genetics demonstrate the advantage of the proposed method over existing alternatives in both statistical power and computational efficiency.

An exhaustive ADDIS principle for online FWER control.

In this paper, we consider online multiple testing with familywise error rate (FWER) control, where the probability of committing at least one type I error will remain under control while testing a possibly infinite sequence of hypotheses over time. Currently, adaptive-discard (ADDIS) procedures seem to be the most promising online procedures with FWER control in terms of power. Now, our main contribution is a uniform improvement of the ADDIS principle and thus of all ADDIS procedures. This means, the methods we propose reject as least as much hypotheses as ADDIS procedures and in some cases even more, while maintaining FWER control. In addition, we show that there is no other FWER controlling procedure that enlarges the event of rejecting any hypothesis. Finally, we apply the new principle to derive uniform improvements of the ADDIS-Spending andADDIS-Graph.

Statistical inference for diagnostic test accuracy studies with multiple comparisons.

Diagnostic accuracy studies assess the sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test are usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance, if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regard to statistical error rates in least-favourable and realistic scenarios. This covers parametric and non-parametric methods and one Bayesian approach. All methods have been implemented in the new open-source R package cases which allows us to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT and Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power.

Financial Toxicity in the Spanish Speaking Population

Advances in therapy have significantly improved cancer survival outcomes but with a harmful patient consequence: financial toxicity. Financial toxicity consists of a patient's many economic burdens from cancer treatment, encompassing direct healthcare costs as well as additional devastating financial consequences, such as job loss and medical bankruptcy. With rising cancer treatment costs, financial toxicity has become a problem that is increasingly impactful in both cancer treatment and survivorship, amplifying disparities, negatively impacting quality of life, treatment adherence and survival. The long-term research goal that this study will contribute to is to develop culturally appropriate and effective patient navigation resources and psychosocial support resources for interventions to increase coping and resilience to financial toxicity in high-risk populations. This study will seek to generate the early data needed to help develop the content and method of delivering strategies, tailored for an underserved, Spanish-speaking Latino population. Eligibility Participant must be 18 years of age or older at the time of study participation, able to read and speak Spanish, have diagnosis of cancer within the past 2 years or be on active treatment for their cancer; all disease sites and all stages are eligible for enrollment. We will exclude patients unable to give consent. Methods This is a prospective observational study of cancer patients and survivors, with participants to be recruited from Scripps MD Anderson Cancer Center. Participating clinics will be screened in the electronic medical record for individuals who meet above eligibility criteria. At the time of clinic visits, eligible individuals will be approached. Individuals who express that they would like to participate in the survey study will receive the study packet with the consent statement and baseline survey questions. Study participants will also be invited to participate in the OPTIONAL patient navigation portion of the study. Participants who are not active on the MyScripps patient portal will be invited to join the patient navigation portion of the study. Those elect to participate will be provided a device if they do not have their own. Participants will be given a Greenphire Clincard on which they will receive a total of $75 in three installments. Research team members will enter the item responses into the REDCap database. They will undergo a follow-up survey during a range from 4 months to 9 months from the initial survey date. Those participating in the patient navigation portion of the study will be contacted by the navigator on a monthly basis through the electronic portal or telephone. Statistics We plan to enroll N= 80 Spanish-speaking patients in the survey study. For correlation testing, we hypothesize a moderate effect size (correlation = 0.32) for the association between the primary outcome measure, the baseline measure of self-efficacy (Hope State score) and the baseline financial toxicity score (Economic Strain and Resilience in Cancer). For feasibility testing of navigation participation, feasibility will be assessed through descriptive statistics. We hypothesize that it is feasible for 50% of invited patients to participate. When the sample size is 40, a two-sided 95% confidence interval for participant completion rate will extend 0.15 from the observed proportion for an expected proportion of 0.50. Analysis The primary outcome measure is financial toxicity, measured using the ENRICh instrument at baseline. The primary predictor is the baseline Hope State score. This modeled association is of primary interest in the analysis. We will conduct descriptive and univariate/multivariate analyses on this association of primary interest, including Chi-square test, t-test. Other univariate and multivariate analyses of associations between financial toxicity and other measures obtained from survey measures, as well as clinical, demographic, socioeconomic, and care delivery variables, considered descriptive and exploratory, will be performed. Accounting for multiple comparisons will be based on stepdown procedures and familywise error rate control; we plan to control this overall error rate using the Holm-Sidak adjustment, or procedure with equitable properties.

Improved family-wise error rate control in multiple equivalence testing.

Equivalence testing is an important component of safety assessments, used for example by the European Food Safety Authority, to allow new food or feed products on the market. The aim of such tests is to demonstrate equivalence of characteristics of test and reference crops. Equivalence tests are typically univariate and applied to each measured analyte (characteristic) separately without multiplicity correction. This increases the probability of making false claims of equivalence (type I errors) when evaluating multiple analytes simultaneously. To solve this problem, familywise error rate (FWER) control using Hochberg's method has been proposed. This paper demonstrates that, in the context of equivalence testing, other FWER-controlling methods are more powerful than Hochberg's. Particularly, it is shown that Hommel's method is guaranteed to perform at least as well as Hochberg's and that an "adaptive" version of Bonferroni's method, which uses an estimator of the proportion of non-equivalent characteristics, often substantially outperforms Hommel's method. Adaptive Bonferroni takes better advantage of the particular context of food safety where a large proportion of true equivalences is expected, a situation where other methods are particularly conservative. The different methods are illustrated by their application to two compositional datasets and further assessed and compared using simulated data.

Familywise error rate control for block response-adaptive randomization.

Response-adaptive randomization allows the probabilities of allocating patients to treatments in a clinical trial to change based on the previously observed response data, in order to achieve different experimental goals. One concern over the use of such designs in practice, particularly from a regulatory viewpoint, is controlling the type I error rate. To address this, Robertson and Wason (Biometrics, 2019) proposed methodology that guarantees familywise error rate control for a large class of response-adaptive designs by re-weighting the usual -test statistic. In this article, we propose an improvement of their method that is conceptually simpler, in the context where patients are allocated to the experimental treatment arms in a trial in blocks (i.e. groups) using response-adaptive randomization. We show the modified method guarantees that there will never be negative weights for the contribution of each block of data to the adjusted test statistics, and can also provide a substantial power advantage in practice.

Reverse graphical approaches for multiple test procedures

ABSTRACT The graphical approach has been proposed as a general framework for clinical trial designs involving multiple hypotheses, where decisions are made only based on the observed marginal p -values. The graphical approach starts from a graph that includes all hypotheses as vertices and gradually removes some vertices when their corresponding hypotheses are rejected. In this paper, we propose a reverse graphical approach, which starts from a set of singleton graphs and gradually adds vertices into graphs until rejection of a set of hypotheses is made. Proofs of familywise error rate control are provided. A simulation study is conducted for statistical power analysis, and a case study is included to illustrate how the proposed approach can be applied to clinical studies.

Practical basket design for binary outcomes with control of family-wise error rate

BackgroundA basket trial is a type of clinical trial in which eligibility is based on the presence of specific molecular characteristics across subpopulations with different cancer types. The existing basket designs with Bayesian hierarchical models often improve the efficiency of evaluating therapeutic effects; however, these models calibrate the type I error rate based on the results of simulation studies under various selected scenarios. The theoretical control of family-wise error rate (FWER) is important for decision-making regarding drug approval.MethodsIn this study, we propose a new Bayesian two-stage design with one interim analysis for controlling FWER at the target level, along with the formulations of type I and II error rates. Since the difficulty lies in the complexity of the theoretical formulation of the type I error rate, we devised the simulation-based method to approximate the type I error rate.ResultsThe proposed design enabled adjustment of the cutoff value to control the FWER at the target value in the final analysis. The simulation studies demonstrated that the proposed design can be used to control the well-approximated FWER below the target value even in situations where the number of enrolled patients differed among subpopulations.ConclusionsThe accrual number of patients is sometimes unable to reach the pre-defined value; therefore, existing basket designs may not ensure defined operating characteristics before beginning the trial. The proposed design that enables adjustment of the cutoff value to control FWER at the target value based on the results in the final analysis would be a better alternative.

Discovering Significant Sequential Patterns in Data Stream by an Efficient Two-Phase Procedure

One essential topic of mining sequential patterns in the data stream is to optimize the time-space computations. However, more importantly, it should pay more attention to the significance of mining results as a large portion of them just response to the user-defined constraints purely by accident and they may have no statistical significance. In this paper, we propose FSSPDS, an efficient two-phase algorithm to discover the significant sequential patterns (SSPs) in the data stream with typical sliding windows, which has never been considered in existing problems. First, for generating SSPs candidates with high-quality, FSSPDS takes testable support and pattern length constraints into account and insignificant patterns were removed timely by a pattern-growth method. In the second phase, appropriate permutation testing is used to test the significance of the SSPs candidates. Exact permutation p values are obtained in a novel combination way based on unconditional Barnard’s test statistic which better reflects the process of data generations and collections. Experimental evaluations show that FSSPDS allows the discovery of SSPs in the data stream and rivals the state-of-the-art approaches efficiently under the control of family-wise error rate (FWER), especially for time efficiency, which was approximately an order of magnitude higher.

Pairwise comparisons using ranks in block designs

Friedman’s rank test and the associated aligned rank test are the standard rank alternatives to the classical linear models F statistic for the randomized complete block design (RCBD). However, in current practice there are no good rank alternatives to the Tukey–Kramer all pairwise comparisons procedure for the normal linear model. For example, the standard rank method found in Section 7.3 of Hollander et al. [John Wiley & Sons, Inc., Hoboken, New Jersey; 2014] is based on the permutation distribution under the complete null of no differences and thus cannot strongly control the family-wise error rate (FWER). It also has low power. However, we show that the closed method introduced by Marcus et al. [Biometrika 63 (3);1976:655–60] applied to aligned ranks has strong FWER control and good power compared to the Tukey–Kramer method for long-tailed error distributions.

Efficiency of a randomized confirmatory basket trial design constrained to control the family wise error rate by indication.

Basket trials pool histologic indications sharing molecular pathophysiology, improving development efficiency. Currently, basket trials have been confirmatory only for exceptional therapies. Our previous randomized basket design may be generally suitable in the resource-intensive confirmatory phase, maintains high power even with modest effect sizes, and provides nearly k-fold increased efficiency for k indications, but controls false positives for the pooled result only. Since family wise error rate by indications may sometimes be required, we now simulate a variant of this basket design controlling family wise error rate at 0.025k, the total family wise error rate of k separate randomized trials. We simulated this modified design under numerous scenarios varying design parameters. Only designs controlling family wise error rate and minimizing estimation bias were allowable. Optimal performance results when . We report efficiency (expected # true positives/expected sample size) relative to k parallel studies, at 90% power ("uncorrected") or at the power achieved in the basket trial ("corrected," because conventional designs could also increase efficiency by sacrificing power). Efficiency and power (percentage active indications identified) improve with a higher percentage of initial indications active. Up to 92% uncorrected and 38% corrected efficiency improvement is possible. Even under family wise error rate control, randomized confirmatory basket trials substantially improve development efficiency. Initial indication selection is critical.

Estimation and inference on treatment effects under treatment-based sampling designs

Summary Causal inference in a programme evaluation setting faces the problem of external validity when the treatment effect in the target population is different from the treatment effect identified from the population of which the sample is representative. This paper focuses on a situation where such discrepancy arises by a stratified sampling design based on the individual treatment status and other characteristics. In such settings, the design probability is known from the sampling design but the target population depends on the underlying population share vector, which is often unknown, and, except for special cases, the treatment effect parameters are not identified. In this paper we propose a method of constructing confidence sets that are valid for a given range of population shares. When a benchmark population share vector and a corresponding estimator of a treatment effect parameter are given, we develop a method to discover the scope of external validity with familywise error rate control. Finally, we derive an optimal sampling design that minimizes the semiparametric efficiency bound given a population share associated with a target population. We provide Monte Carlo simulation results and an empirical application to demonstrate the usefulness of our proposals.

Identifying pleiotropic genes for complex phenotypes with summary statistics from a perspective of composite null hypothesis testing.

Pleiotropy has important implication on genetic connection among complex phenotypes and facilitates our understanding of disease etiology. Genome-wide association studies provide an unprecedented opportunity to detect pleiotropic associations; however, efficient pleiotropy test methods are still lacking. We here consider pleiotropy identification from a methodological perspective of high-dimensional composite null hypothesis and propose a powerful gene-based method called MAIUP. MAIUP is constructed based on the traditional intersection-union test with two sets of independent P-values as input and follows a novel idea that was originally proposed under the high-dimensional mediation analysis framework. The key improvement of MAIUP is that it takes the composite null nature of pleiotropy test into account by fitting a three-component mixture null distribution, which can ultimately generate well-calibrated P-values for effective control of family-wise error rate and false discover rate. Another attractive advantage of MAIUP is its ability to effectively address the issue of overlapping subjects commonly encountered in association studies. Simulation studies demonstrate that compared with other methods, only MAIUP can maintain correct type I error control and has higher power across a wide range of scenarios. We apply MAIUP to detect shared associated genes among 14 psychiatric disorders with summary statistics and discover many new pleiotropic genes that are otherwise not identified if failing to account for the issue of composite null hypothesis testing. Functional and enrichment analyses offer additional evidence supporting the validity of these identified pleiotropic genes associated with psychiatric disorders. Overall, MAIUP represents an efficient method for pleiotropy identification.

Group sequential Holm and Hochberg procedures.

The problem of testing multiple hypotheses using a group sequential procedure often arises in clinical trials. We review several group sequential Holm (GSHM) type procedures proposed in the literature and clarify the relationships between them. In particular, we show which procedures are equivalent or, if different, which are more powerful and what are their pros and cons. We propose a step-up group sequential Hochberg (GSHC) procedure as a reverse application of a particular step-down GSHM procedure. We conducted an extensive simulation study to evaluate the familywise error rate (FWER) and power properties of that GSHM procedure and the GSHC procedure and found that the GSHC procedure controls FWER more closely and is more powerful. All procedures are illustrated with a common numerical example, the data for which are chosen to bring out the differences between them. A real case study is also presented to illustrate application of these procedures. R programs for applying the proposed procedures, additional simulation results, and the proof of the FWER control of the GSHC procedure in a special case are provided in Supplementary Material.

Optimizing subgroup selection in two-stage adaptive enrichment and umbrella designs.

We design two‐stage confirmatory clinical trials that use adaptation to find the subgroup of patients who will benefit from a new treatment, testing for a treatment effect in each of two disjoint subgroups. Our proposal allows aspects of the trial, such as recruitment probabilities of each group, to be altered at an interim analysis. We use the conditional error rate approach to implement these adaptations with protection of overall error rates. Applying a Bayesian decision‐theoretic framework, we optimize design parameters by maximizing a utility function that takes the population prevalence of the subgroups into account. We show results for traditional trials with familywise error rate control (using a closed testing procedure) as well as for umbrella trials in which only the per‐comparison type 1 error rate is controlled. We present numerical examples to illustrate the optimization process and the effectiveness of the proposed designs.

Predictive Genetic Variations in the Kynurenine Pathway for Interferon-α-Induced Depression in Patients with Hepatitis C Viral Infection.

Importance: The high incidence of major depressive episodes during interferon-α (IFN-α) therapy is considered the most powerful supportive evidence for the inflammation theory of depression. As the kynurenine pathway plays an important role connecting inflammation and depression, it is plausible to investigate this pathway for predictive genetic markers for IFN-α-induced depression. Methods: In this prospective case-control study, we assessed 291 patients with chronic hepatitis C viral infection taking IFN-α therapy and analyzed the single nucleotide polymorphisms (SNPs) in genes in the kynurenine pathway. Our case group contained patients who developed IFN-α-induced depression during the treatment, and others were defined as the control group. Genomic DNA was extracted from leukocytes in the peripheral blood and analyzed by Affymetrix TWB array. We first tested allelic, dominant, and recessive models on each of our SNPs using Fisher’s exact test. We then conducted 5000 gene-wide max(T) permutations based on the best model of each SNP to provide strong gene-wide family-wise error rate control. Finally, we preformed logistic regression for the significant SNPs acquired in previous procedures, with sex and education level as covariates to build predictive models. Additional haplotype analyses were conducted with Haploview 4.2 to investigate the combining effect of multiple significant SNPs within a gene. Results: With sex and education level as covariates, rs8082252 (p = 0.0015, odds ratio = 2.716), rs8082142 (p = 0.0031, odds ratio = 2.499) in arylformamidase (AFMID), and rs12477181 (p = 0.0004, odds ratio = 0.3478) in kynureninase (KYNU) were significant in logistic regression models with dominant modes of inheritance. Haplotype analyses showed the two significant SNPs in AFMID to be in the same haploblock and highly correlated (r2 = 0.99). There were two significant haplotypes (by the sequence of rs8082252, rs8082142): AT (χ2 = 7.734, p = 0.0054) and GC (χ2 = 6.874, p = 0.0087). Conclusions: This study provided supportive evidence of the involvement of the kynurenine pathway in IFN-α-induced depression. SNPs in this pathway were also predictive of this disease.