Error Rate Control Research Articles

Note on a stepwise procedure for rejecting at least k out of m hypotheses: A simple Holm-type formulation and proof.

Mielke etal. (2021) proposed a stepwise multiple testing procedure (MTP) based on marginal p-values for rejecting at least k out of m null hypotheses. Briefly, the MTP cannot reject less than k hypotheses, but this property improves the power to reject k or more hypotheses relative to the stepdown MTP of Holm (1979). Mielke etal. discussed why such an MTP is of interest in the context of biosimilarity developments. This article describes how a slight modification of Holm's simple direct arguments can be used as an alternative to the closed-testing arguments of Mielke etal. to show the strong control of the family-wise error rate. This modification is based on a Holm-type formulation of the rejection algorithm where each step involves a single ordered p-value. With k equal to one, the stepwise MTP reduces to Holm's stepdown MTP. The MTP is valid quite generally. A version of the MTP with weights for hypotheses is also described and discussed. As in the case without weights: (a) a modification of Holm's arguments can be used; (b) with k equal to one, the MTP reduces to Holm's MTP with weights; and (c) the MTP is valid quite generally.

Mapping pleiotropic loci using a fast-sequential testing algorithm

Pleiotropy (i.e., genes with effects on multiple traits) leads to genetic correlations between traits and contributes to the development of many syndromes. Identifying variants with pleiotropic effects on multiple health-related traits can improve the biological understanding of gene action and disease etiology, and can help to advance disease-risk prediction. Sequential testing is a powerful approach for mapping genes with pleiotropic effects. However, the existing methods and the available software do not scale to analyses involving millions of SNPs and large datasets. This has limited the adoption of sequential testing for pleiotropy mapping at large scale. In this study, we present a sequential test and software that can be used to test pleiotropy in large systems of traits with biobank-sized data. Using simulations, we show that the methods implemented in the software are powerful and have adequate type-I error rate control. To demonstrate the use of the methods and software, we present a whole-genome scan in search of loci with pleiotropic effects on seven traits related to metabolic syndrome (MetS) using UK-Biobank data (n~300 K distantly related white European participants). We found abundant pleiotropy and report 170, 44, and 18 genomic regions harboring SNPs with pleiotropic effects in at least two, three, and four of the seven traits, respectively. We validate our results using previous studies documented in the GWAS-catalog and using data from GTEx. Our results confirm previously reported loci and lead to several novel discoveries that link MetS-related traits through plausible biological pathways.

Proof of Concept: Drug Selection? Or Dose Selection? Thoughts on Multiplicity Issues.

In new drug development process, one of the most important milestones for a drug candidate is to establish Proof of Concept (PoC) at early Phase II stage. Among many challenges in PoC clinical trial design and analysis, the application of multiplicity comparison procedures (MCP) is frequently discussed when multiple doses or drugs are included in one PoC study. In such discussion, one fundamental question of applying multiplicity adjustment is which error one should consider to control and at what level. Should it be the experiment-wise error or the compound-wise error? In this paper, the multiplicity issues in two cases of PoC studies are used as examples to discuss the concept of different types of error and the level of the error rate control. With a clear understanding of the type of error and error rate control, the debate of applications of the multiplicity adjustment procedures in the PoC studies can be reconciled.

Alternative models and randomization techniques for Bayesian response-adaptive randomization with binary outcomes.

Bayesian response-adaptive designs, which data adaptively alter the allocation ratio in favor of the better performing treatment, are often criticized for engendering a non-trivial probability of a subject imbalance in favor of the inferior treatment, inflating type I error rate, and increasing sample size requirements. The implementation of these designs using the Thompson sampling methods has generally assumed a simple beta-binomial probability model in the literature; however, the effect of these choices on the resulting design operating characteristics relative to other reasonable alternatives has not been fully examined. Motivated by the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial, we posit that a logistic probability model coupled with an urn or permuted block randomization method will alleviate some of the practical limitations engendered by the conventional implementation of a two-arm Bayesian response-adaptive design with binary outcomes. In this article, we discuss up to what extent this solution works and when it does not. A computer simulation study was performed to evaluate the relative merits of a Bayesian response-adaptive design for the Advanced R2 Eperfusion STrategies for Refractory Cardiac Arrest trial using the Thompson sampling methods based on a logistic regression probability model coupled with either an urn or permuted block randomization method that limits deviations from the evolving target allocation ratio. The different implementations of the response-adaptive design were evaluated for type I error rate control across various null response rates and power, among other performance metrics. The logistic regression probability model engenders smaller average sample sizes with similar power, better control over type I error rate, and more favorable treatment arm sample size distributions than the conventional beta-binomial probability model, and designs using the alternative randomization methods have a negligible chance of a sample size imbalance in the wrong direction. Pairing the logistic regression probability model with either of the alternative randomization methods results in a much improved response-adaptive design in regard to important operating characteristics, including type I error rate control and the risk of a sample size imbalance in favor of the inferior treatment.

Psychologists Should Use Brunner-Munzel’s Instead of Mann-Whitney’s U Test as the Default Nonparametric Procedure

To investigate whether a variable tends to be larger in one population than in another, the t test is the standard procedure. In some situations, the parametric t test is inappropriate, and a nonparametric procedure should be used instead. The default nonparametric procedure is Mann-Whitney’s U test. Despite being a nonparametric test, Mann-Whitney’s test is associated with a strong assumption, known as exchangeability. I demonstrate that if exchangeability is violated, Mann-Whitney’s test can lead to wrong statistical inferences even for large samples. In addition, I argue that in psychology, exchangeability is typically not met. As a remedy, I introduce Brunner-Munzel’s test and demonstrate that it provides good Type I error rate control even if exchangeability is not met and that it has similar power as Mann-Whitney’s test. Consequently, I recommend using Brunner-Munzel’s test by default. To facilitate this, I provide advice on how to perform and report on Brunner-Munzel’s test.

Optimizing subgroup selection in two-stage adaptive enrichment and umbrella designs.

We design two‐stage confirmatory clinical trials that use adaptation to find the subgroup of patients who will benefit from a new treatment, testing for a treatment effect in each of two disjoint subgroups. Our proposal allows aspects of the trial, such as recruitment probabilities of each group, to be altered at an interim analysis. We use the conditional error rate approach to implement these adaptations with protection of overall error rates. Applying a Bayesian decision‐theoretic framework, we optimize design parameters by maximizing a utility function that takes the population prevalence of the subgroups into account. We show results for traditional trials with familywise error rate control (using a closed testing procedure) as well as for umbrella trials in which only the per‐comparison type 1 error rate is controlled. We present numerical examples to illustrate the optimization process and the effectiveness of the proposed designs.

AI Uncertainty Based on Rademacher Complexity and Shannon Entropy

In this paper from communication channel coding perspective we are able to present both a theoretical and practical discussion of AI’s uncertainty, capacity and evolution for pattern classification based on the classical Rademacher complexity and Shannon entropy. First AI capacity is defined as in communication channels. It is shown qualitatively that the classical Rademacher complexity and Shannon rate in communication theory is closely related by their definitions. Secondly based on the Shannon mathematical theory on communication coding, we derive several sufficient and necessary conditions for an AI’s error rate approaching zero in classifications problems. A 1/2 criteria on Shannon entropy is derived in this paper so that error rate can approach zero or is zero for AI pattern classification problems. Last but not least, we show our analysis and theory by providing examples of AI pattern classifications with error rate approaching zero or being zero. Impact Statement: Error rate control of AI pattern classification is crucial in many lives related AI applications. AI uncertainty, capacity and evolution are investigated in this paper. Sufficient/necessary conditions for AI’s error rate approaching zero are derived based on Shannon’s communication coding theory. Zero error rate and zero error rate approaching AI design methodology for pattern classifications are illustrated using Shannon’s coding theory. Our method shows how to control the error rate of AI, how to measure the capacity of AI and how to evolve AI into higher levels. Index Terms: Rademacher Complexity, Shannon Theory, Shannon Entropy, Vapnik-Cheronenkis (VC) dimension.

Predictive Genetic Variations in the Kynurenine Pathway for Interferon-α-Induced Depression in Patients with Hepatitis C Viral Infection.

Importance: The high incidence of major depressive episodes during interferon-α (IFN-α) therapy is considered the most powerful supportive evidence for the inflammation theory of depression. As the kynurenine pathway plays an important role connecting inflammation and depression, it is plausible to investigate this pathway for predictive genetic markers for IFN-α-induced depression. Methods: In this prospective case-control study, we assessed 291 patients with chronic hepatitis C viral infection taking IFN-α therapy and analyzed the single nucleotide polymorphisms (SNPs) in genes in the kynurenine pathway. Our case group contained patients who developed IFN-α-induced depression during the treatment, and others were defined as the control group. Genomic DNA was extracted from leukocytes in the peripheral blood and analyzed by Affymetrix TWB array. We first tested allelic, dominant, and recessive models on each of our SNPs using Fisher’s exact test. We then conducted 5000 gene-wide max(T) permutations based on the best model of each SNP to provide strong gene-wide family-wise error rate control. Finally, we preformed logistic regression for the significant SNPs acquired in previous procedures, with sex and education level as covariates to build predictive models. Additional haplotype analyses were conducted with Haploview 4.2 to investigate the combining effect of multiple significant SNPs within a gene. Results: With sex and education level as covariates, rs8082252 (p = 0.0015, odds ratio = 2.716), rs8082142 (p = 0.0031, odds ratio = 2.499) in arylformamidase (AFMID), and rs12477181 (p = 0.0004, odds ratio = 0.3478) in kynureninase (KYNU) were significant in logistic regression models with dominant modes of inheritance. Haplotype analyses showed the two significant SNPs in AFMID to be in the same haploblock and highly correlated (r2 = 0.99). There were two significant haplotypes (by the sequence of rs8082252, rs8082142): AT (χ2 = 7.734, p = 0.0054) and GC (χ2 = 6.874, p = 0.0087). Conclusions: This study provided supportive evidence of the involvement of the kynurenine pathway in IFN-α-induced depression. SNPs in this pathway were also predictive of this disease.

Accurate error control in high-dimensional association testing using conditional false discovery rates.

High-dimensional hypothesis testing is ubiquitous in the biomedical sciences, and informative covariates may be employed to improve power. The conditional false discovery rate (cFDR) is a widely used approach suited to the setting where the covariate is a set of p-values for the equivalent hypotheses for a second trait. Although related to the Benjamini–Hochberg procedure, it does not permit any easy control of type-1 error rate and existing methods are over-conservative. We propose a newmethod for type-1 error rate control based on identifyingmappings from the unit square to the unit interval defined by the estimated cFDR and splitting observations so that each map is independent of the observations it is used to test. We also propose an adjustment to the existing cFDR estimator which further improves power. We show by simulation that the new method more than doubles potential improvement in power over unconditional analyses compared to existing methods. We demonstrate our method on transcriptome-wide association studies and show that the method can be used in an iterative way, enabling the use of multiple covariates successively. Our methods substantially improve the power and applicability of cFDR analysis.

Multiscale Null Hypothesis Testing for Network-valued Data: Analysis of Brain Networks of Patients with Autism

Abstract Networks are a natural way of representing the human brain for studying its structure and function and, as such, have been extensively used. In this framework, case–control studies for understanding autism pertain to comparing samples of healthy and autistic brain networks. In order to understand the biological mechanisms involved in the pathology, it is key to localize the differences on the brain network. Motivated by this question, we hereby propose a general non-parametric finite-sample exact statistical framework that allows to test for differences in connectivity within and between prespecified areas inside the brain network, with strong control of the family-wise error rate. We demonstrate unprecedented ability to differentiate children with non-syndromic autism from children with both autism and tuberous sclerosis complex using electroencephalography data. The implementation of the method is available in the R package nevada.

Two-stage penalized regression screening to detect biomarker-treatment interactions in randomized clinical trials.

High-dimensional biomarkers such as genomics are increasingly being measured in randomized clinical trials. Consequently, there is a growing interest in developing methods that improve the power to detect biomarker-treatment interactions. We adapt recently proposed two-stage interaction detecting procedures in the setting of randomized clinical trials. We also propose a new stage 1 multivariate screening strategy using ridge regression to account for correlations among biomarkers. For this multivariate screening, we prove the asymptotic between-stage independence, required for familywise error rate control, under biomarker-treatment independence. Simulation results show that in various scenarios, the ridge regression screening procedure can provide substantially greater power than the traditional one-biomarker-at-a-time screening procedure in highly correlated data. We also exemplify our approach in two real clinical trial dataapplications.

An automation-based adaptive seamless design for dose selection and confirmation with improved power and efficiency.

In a drug development program, the efficacy and safety of multiple doses can be evaluated in patients through a phase 2b dose ranging study. With a demonstrated dose response in the trial, promising doses are identified. Their effectiveness then is further investigated and confirmed in phase 3 studies. Although this two-step approach serves the purpose of the program, in general, it is inefficient because of its prolonged development duration and the exclusion of the phase 2b data in the final efficacy evaluation and confirmation which are only based on phase 3 data. To address the issue, we propose a new adaptive design, which seamlessly integrates the dose finding and confirmation steps under one pivotal study. Unlike existing adaptive seamless phase 2b/3 designs, the proposed design combines the response adaptive randomization, sample size modification, and multiple testing techniques to achieve better efficiency. The design can be easily implemented through an automated randomization process. At the end, a number of targeted doses are selected and their effectiveness is confirmed with guaranteed control of family-wise error rate.

Online control of the familywise error rate.

Biological research often involves testing a growing number of null hypotheses as new data are accumulated over time. We study the problem of online control of the familywise error rate, that is testing an a priori unbounded sequence of hypotheses (p-values) one by one over time without knowing the future, such that with high probability there are no false discoveries in the entire sequence. This paper unifies algorithmic concepts developed for offline (single batch) familywise error rate control and online false discovery rate control to develop novel online familywise error rate control methods. Though many offline familywise error rate methods (e.g., Bonferroni, fallback procedures and Sidak's method) can trivially be extended to the online setting, our main contribution is the design of new, powerful, adaptive online algorithms that control the familywise error rate when the p-values are independent or locally dependent in time. Our numerical experiments demonstrate substantial gains in power, that are also formally proved in an idealized Gaussian sequence model. A promising application to the International Mouse Phenotyping Consortium is described.

Compilation of a Coupled Hyper-Chaotic Lorenz System Based on DNA Strand Displacement Reaction Network.

Ideal formal chemical reaction network is an effective programming language to design complex system dynamical behavior. In this article, a coupled hyper-chaotic Lorenz system can be described by the ordinary differential equations of an ideal formal reaction network, which is constructed by catalysis, annihilation and adjust reaction modules, where the variables of system are represented by the difference in concentration of two chemical species. The ideal formal reaction network can be implemented by DNA strand displacement reaction network. Through Lyapunov exponent, we have analyzed hyper-chaotic dynamical behavior of coupled Lorenz system. In discussion and analysis, we have analyzed effect of noise, reaction rate control error and concentration detection error to DNA strand displacement reaction network.

H-MAGMA, inheriting a shaky statistical foundation, yields excess false positives.

All data used in the manuscript are available at https://github.com/ryurko/HMAGMA-comment. Supplementary Fig. 1. Comparison of covariance approximations for two-sided tests (green-solid) versus Brown's one-sided approximation described in MAGMA paper (red-dashed) and the code implemented in MAGMA with r2 (cyan-dashed). Supplementary Fig. 2. Comparison of the type 1 error rate control, plus/minus two standard errors, at target a = 0:05 (denoted by dashed red line) by gene-level method for seven example genes of different sizes. Supplementary Fig. 3. Comparison of multiple testing family-wise error rate (FWER), plus/minus two standard errors, at target a = 0:05 (denoted with dashed red line) by gene-level method. Supplementary Fig. 4. a, Comparison of the proportion of simulations with zero to six false positives for each method, excluding MAGMA: paper, using the Bonferroni correction at target a = 0:05 (plus/minus two standard errors). Supplementary Fig. 5. Comparison of gene-set analysis type 1 error, plus/minus two-standard errors, at target a = 0:05 (denoted with dashed red line) by gene-level method for each considered gene-set analysis method (from left to right) MAGMA: gene-set, Fisher's combination test, and Stouffer's z-test. Supplementary Table 1. Comparison of number of BH discoveries (union across adult and fetal Hi-C data annotations) with H-MAGMA versus corrected Monte Carlo based approach. Supplementary Fig. 6. Comparison of null histograms by gene-set analysis method (columns) and methods for computing gene-level p-values (rows) for gene-sets of size G = 45 genes. Red dashed lines indicate expected 2.5% and 97.5% quantiles for uniform p-values across 200,000 simulations. Supplementary Fig. 7. Comparison of distributions for MAGMA gene-set analysis standardized test statistic by methods for computing gene-level p-values. Red density curve displays the expected t(G 1) distribution, with the red vertical dashed line denoting the center at zero. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

Covariate adaptive familywise error rate control for genome-wide association studies.

The familywise error rate has been widely used in genome-wide association studies. With the increasing availability of functional genomics data, it is possible to increase detection power by leveraging these genomic functional annotations. Previous efforts to accommodate covariates in multiple testing focused on false discovery rate control, while covariate-adaptive procedures controlling the familywise error rate remain underdeveloped. Here, we propose a novel covariate-adaptive procedure to control the familywise error rate that incorporates external covariates which are potentially informative of either the statistical power or the prior null probability. An efficient algorithm is developed to implement the proposed method. We prove its asymptotic validity and obtain the rate of convergence through a perturbation-type argument. Our numerical studies show that the new procedure is more powerful than competing methods and maintains robustness across different settings. We apply the proposed approach to the UK Biobank data and analyse 27 traits with 9 million single-nucleotide polymorphisms tested for associations. Seventy-five genomic annotations are used as covariates. Our approach detects more genome-wide significant loci than other methods in 21 out of the 27 traits.

Adaptive enrichment trials: What are the benefits?

When planning a Phase III clinical trial, suppose a certain subset of patients is expected to respond particularly well to the new treatment. Adaptive enrichment designs make use of interim data in selecting the target population for the remainder of the trial, either continuing with the full population or restricting recruitment to the subset of patients. We define a multiple testing procedure that maintains strong control of the familywise error rate, while allowing for the adaptive sampling procedure. We derive the Bayes optimal rule for deciding whether or not to restrict recruitment to the subset after the interim analysis and present an efficient algorithm to facilitate simulation‐based optimisation, enabling the construction of Bayes optimal rules in a wide variety of problem formulations. We compare adaptive enrichment designs with traditional nonadaptive designs in a broad range of examples and draw clear conclusions about the potential benefits of adaptive enrichment.

Sample size re-estimation for pivotal clinical trials

It is well known that if the hypothesis test is left unchanged, the Type I error rate may be inflated for sample size re-estimation (SSR) designs. To address this issue, three main approaches have been proposed in the literature: combination test, conditional error and conventional test with sample size increase in the allowable region (AR) only. These three seemingly different approaches are in fact connected. For each combination test, there is a corresponding conditional error function and AR. Designing adaptation rules in this AR with conventional test guarantees the Type I error rate control but at the same time always leads to smaller power comparing to the corresponding combination test (or conditional error) approach. In cases where conventional test is still preferable, step-wise type adaptation rules that do not fully reside in the AR can be alternatively considered.We believe controversies in the statistical community on the efficiency comparisons between group sequential (GS) and SSR design stem partially from the misalignment of performance metrics and conditional versus unconditional evaluations. We advocate summary metrics, such as median, variance or tail probabilities of the sample size in addition to expectation and personalizing efficiency definition for each trial sponsor. Conditional metrics by favorable, promising and unfavorable zones of the interim results provide additional insights and should always be incorporated into the decision-making process.

Permutation testing in high-dimensional linear models: an empirical investigation

Permutation testing in linear models, where the number of nuisance coefficients is smaller than the sample size, is a well-studied topic. The common approach of such tests is to permute residuals after regressing on the nuisance covariates. Permutation-based tests are valuable in particular because they can be highly robust to violations of the standard linear model, such as non-normality and heteroscedasticity. Moreover, in some cases they can be combined with existing, powerful permutation-based multiple testing methods. Here, we propose permutation tests for models where the number of nuisance coefficients exceeds the sample size. The performance of the novel tests is investigated with simulations. In a wide range of simulation scenarios our proposed permutation methods provided appropriate type I error rate control, unlike some competing tests, while having good power.

Empirical Likelihood Test for Regression Coefficients in High Dimensional Partially Linear Models

This paper considers tests for regression coefficients in high dimensional partially linear Models. The authors first use the B-spline method to estimate the unknown smooth function so that it could be linearly expressed. Then, the authors propose an empirical likelihood method to test regression coefficients. The authors derive the asymptotic chi-squared distribution with two degrees of freedom of the proposed test statistics under the null hypothesis. In addition, the method is extended to test with nuisance parameters. Simulations show that the proposed method have a good performance in control of type-I error rate and power. The proposed method is also employed to analyze a data of Skin Cutaneous Melanoma (SKCM).