Error Rate Control Research Articles

Estimation and inference on treatment effects under treatment-based sampling designs

Summary Causal inference in a programme evaluation setting faces the problem of external validity when the treatment effect in the target population is different from the treatment effect identified from the population of which the sample is representative. This paper focuses on a situation where such discrepancy arises by a stratified sampling design based on the individual treatment status and other characteristics. In such settings, the design probability is known from the sampling design but the target population depends on the underlying population share vector, which is often unknown, and, except for special cases, the treatment effect parameters are not identified. In this paper we propose a method of constructing confidence sets that are valid for a given range of population shares. When a benchmark population share vector and a corresponding estimator of a treatment effect parameter are given, we develop a method to discover the scope of external validity with familywise error rate control. Finally, we derive an optimal sampling design that minimizes the semiparametric efficiency bound given a population share associated with a target population. We provide Monte Carlo simulation results and an empirical application to demonstrate the usefulness of our proposals.

Personalized Risk-Based Screening Design for Comparative Two-Arm Group Sequential Clinical Trials.

Personalized medicine has been emerging to take into account individual variability in genes and environment. In the era of personalized medicine, it is critical to incorporate the patients’ characteristics and improve the clinical benefit for patients. The patients’ characteristics are incorporated in adaptive randomization to identify patients who are expected to get more benefit from the treatment and optimize the treatment allocation. However, it is challenging to control potential selection bias from using observed efficacy data and the effect of prognostic covariates in adaptive randomization. This paper proposes a personalized risk-based screening design using Bayesian covariate-adjusted response-adaptive randomization that compares the experimental screening method to a standard screening method based on indicators of having a disease. Personalized risk-based allocation probability is built for adaptive randomization, and Bayesian adaptive decision rules are calibrated to preserve error rates. A simulation study shows that the proposed design controls error rates and yields a much smaller number of failures and a larger number of patients allocated to a better intervention compared to existing randomized controlled trial designs. Therefore, the proposed design performs well for randomized controlled clinical trials under personalized medicine.

Hybrid FAHP and TOPSIS to determine recommendation for improving SMEs facing Covid-19 pandemic

The Covid-19 pandemic has had an impact on domestic economy, such as a decrease in people’s consumption and purchasing power, a decline in company performance, as well as SMEs. This situation, SMEs sector controls 99.99% of all existing businesses, employs 97.16% of private sector workforce, and contributes 57.5% to Indonesia’s gross product. Various government program efforts in helping Batik SMEs players face Covid-19 pandemic so that they are right on target. The large number of SMEs in Bangkalan is around 166,768, causing department to find it difficult to determine recommendations for improvement for each MSME. Based on these problems, a cooperative model for the SMEs industry is needed so that official work program runs smoothly on target. The method used is FAHP and TOPSIS integration method, FAHP method is used to determine weighting and TOPSIS method is used for ranking and recommendations for improvement. The advantage of FAHP is consistency index to control error rate of decision. TOPSIS method is capable of making multi-criteria decisions based on alternative SMEs industry mapping process. The purpose is make a recommendation model for improvement using hybrid method of FAHP and TOPSIS with balanced scorecard indicators for learning and growth perspective. The research contribution is to make decisions based on group recommendations and consistency ratio (CR) is less than 0.1. The findings research, that the influential indicators are batik business ownership and variations of batik motifs. The recommendations for improvement, the average MSME still does not have branding.

A multiple testing framework for diagnostic accuracy studies with co-primary endpoints.

Major advances have been made regarding the utilization of machine learning techniques for disease diagnosis and prognosis based on complex and high-dimensional data. Despite all justified enthusiasm, overoptimistic assessments of predictive performance are still common in this area. However, predictive models and medical devices based on such models should undergo a throughout evaluation before being implemented into clinical practice. In this work, we propose a multiple testing framework for (comparative) phase III diagnostic accuracy studies with sensitivity and specificity as co-primary endpoints. Our approach challenges the frequent recommendation to strictly separate model selection and evaluation, that is, to only assess a single diagnostic model in the evaluation study. We show that our parametric simultaneous test procedure asymptotically allows strong control of the family-wise error rate. A multiplicity correction is also available for point and interval estimates. Moreover, we demonstrate in an extensive simulation study that our multiple testing strategy on average leads to a better final diagnostic model and increased statistical power. To plan such studies, we propose a Bayesian approach to determine the optimal number of models to evaluate simultaneously. For this purpose, our algorithm optimizes the expected final model performance given previous (hold-out) data from the model development phase. We conclude that an assessment of multiple promising diagnostic models in the same evaluation study has several advantages when suitable adjustments for multiple comparisons are employed.

Benchmarking of a Bayesian single cell RNAseq differential gene expression test for dose-response study designs.

The application of single-cell RNA sequencing (scRNAseq) for the evaluation of chemicals, drugs, and food contaminants presents the opportunity to consider cellular heterogeneity in pharmacological and toxicological responses. Current differential gene expression analysis (DGEA) methods focus primarily on two group comparisons, not multi-group dose–response study designs used in safety assessments. To benchmark DGEA methods for dose–response scRNAseq experiments, we proposed a multiplicity corrected Bayesian testing approach and compare it against 8 other methods including two frequentist fit-for-purpose tests using simulated and experimental data. Our Bayesian test method outperformed all other tests for a broad range of accuracy metrics including control of false positive error rates. Most notable, the fit-for-purpose and standard multiple group DGEA methods were superior to the two group scRNAseq methods for dose–response study designs. Collectively, our benchmarking of DGEA methods demonstrates the importance in considering study design when determining the most appropriate test methods.

An order restricted multi-arm multi-stage clinical trial design.

One family of designs that can noticeably improve efficiency in later stages of drug development are multi-arm multi-stage (MAMS) designs. They allow several arms to be studied concurrently and gain efficiency by dropping poorly performing treatment arms during the trial aswell as by allowing to stop early for benefit. Conventional MAMS designs were developed for the setting, in which treatment arms are independent and hence can be inefficient when an order in the effects of the arms can be assumed (eg,when considering different treatment durations or different doses). In this work, we extend the MAMS framework to incorporate the order of treatment effects when no parametric dose-response or duration-responsemodel is assumed. The design can identify all promising treatments with high probability. We show that the design provides strong control of the family-wise error rate and illustrate the design in a study of symptomatic asthma. Via simulations we show that the inclusion of the ordering information leads to better decision-making compared to a fixed sample and aMAMS design. Specifically, in the considered settings, reductions in sample size of around 15% were achieved in comparison to a conventional MAMS design.

Intelligent Network Traffic Control Based on Deep Reinforcement Learning

This paper studies the problems of load balancing and flow control in data center network, and analyzes several common flow control schemes in data center intelligent network and their existing problems. On this basis, the network traffic control problem is modeled with the goal of deep reinforcement learning strategy optimization, and an intelligent network traffic control method based on deep reinforcement learning is proposed. At the same time, for the flow control order problem in deep reinforcement learning algorithm, a flow scheduling priority algorithm is proposed innovatively. According to the decision output, the corresponding flow control and control are carried out, so as to realize the load balance of the network. Finally, experiments show, the network traffic bandwidth loss rate of the proposed intelligent network traffic control method is low. Under the condition of random 60 traffic density, the average bisection bandwidth obtained by the proposed intelligent network traffic control method is 4.0mbps and the control error rate is 2.25%. The intelligent network traffic control method based on deep reinforcement learning has high practicability in the practical application process, and fully meets the research requirements.

Threshold Selection in Feature Screening for Error Rate Control

Hard thresholding rule is commonly adopted in feature screening procedures to screen out unimportant predictors for ultrahigh-dimensional data. However, different thresholds are required to adapt to different contexts of screening problems and an appropriate thresholding magnitude usually varies from the model and error distribution. With an ad-hoc choice, it is unclear whether all of the important predictors are selected or not, and it is very likely that the procedures would include many unimportant features. We introduce a data-adaptive threshold selection procedure with error rate control, which is applicable to most kinds of popular screening methods. The key idea is to apply the sample-splitting strategy to construct a series of statistics with marginal symmetry property and then to utilize the symmetry for obtaining an approximation to the number of false discoveries. We show that the proposed method is able to asymptotically control the false discovery rate and per family error rate under certain conditions and still retains all of the important predictors. Three important examples are presented to illustrate the merits of the new proposed procedures. Numerical experiments indicate that the proposed methodology works well for many existing screening methods. Supplementary materials for this article are available online.

Bayesian multivariate probability of success using historical data with type I error rate control.

In clinical trials, it is common to have multiple clinical outcomes (e.g., coprimary endpoints or a primary and multiple secondary endpoints). It is often desirable to establish efficacy in at least one of multiple clinical outcomes, which leads to a multiplicity problem. In the frequentist paradigm, the most popular methods to correct for multiplicity are typically conservative. Moreover, despite guidance from regulators, it is difficult to determine the sample size of a future study with multiple clinical outcomes. In this article, we introduce a Bayesian methodology for multiple testing that asymptotically guarantees type I error control. Using a seemingly unrelated regression model, correlations between outcomes are specifically modeled, which enables inference on the joint posterior distribution of the treatment effects. Simulation results suggest that the proposed Bayesian approach is more powerful than the method of Holm (1979), which is commonly utilized in practice as a more powerful alternative to the ubiquitous Bonferroni correction. We further develop multivariate probability of success, a Bayesian method to robustly determine sample size in the presence of multiple outcomes.

Multiple hypothesis testing with persistent homology

In this paper we propose a computationally efficient multiple hypothesis testing procedure for persistent homology. The computational efficiency of our procedure is based on the observation that one can empirically simulate a null distribution that is universal across many hypothesis testing applications involving persistence homology. Our observation suggests that one can simulate the null distribution efficiently based on a small number of summaries of the collected data and use this null in the same way that p-value tables were used in classical statistics. To illustrate the efficiency and utility of the null distribution we provide procedures for rejecting acyclicity with both control of the Family-Wise Error Rate (FWER) and the False Discovery Rate (FDR). We will argue that the empirical null we propose is very general conditional on a few summaries of the data based on simulations and limit theorems for persistent homology for point processes.

Online Change Point Detection for Weighted and Directed Random Dot Product Graphs

Given a sequence of random (directed and weighted) graphs, we address the problem of online monitoring and detection of changes in the underlying data distribution. Our idea is to endow sequential change-point detection (CPD) techniques with a graph representation learning substrate based on the versatile Random Dot Product Graph (RDPG) model. We consider efficient, online updates of a judicious monitoring function, which quantifies the discrepancy between the streaming graph observations and the nominal RDPG. This reference distribution is inferred via spectral embeddings of the first few graphs in the sequence. We characterize the distribution of this running statistic to select thresholds that guarantee error-rate control, and under simplifying approximations we offer insights on the algorithm’s detection resolution and delay. The end result is a lightweight online CPD algorithm, that is also explainable by virtue of the well-appreciated interpretability of RDPG embeddings. This is in stark contrast with most existing graph CPD approaches, which either rely on extensive computation, or they store and process the entire observed time series. An apparent limitation of the RDPG model is its suitability for undirected and unweighted graphs only, a gap we aim to close here to broaden the scope of the CPD framework. Unlike previous proposals, our non-parametric RDPG model for weighted graphs does not require a priori specification of the weights’ distribution to perform inference and estimation. This network modeling contribution is of independent interest beyond CPD. We offer an open-source implementation of the novel online CPD algorithm for weighted and direct graphs, whose effectiveness and efficiency are demonstrated via (reproducible) synthetic and real network data experiments.

The cluster depth tests: Toward point-wise strong control of the family-wise error rate in massively univariate tests with application to M/EEG

The cluster mass test has been widely used for massively univariate tests in M/EEG, fMRI and, recently, pupillometry analysis. It is a powerful method for detecting effects while controlling weakly the family-wise error rate (FWER), although its correct interpretation can only be performed at the cluster level without any point-wise conclusion. It implies that the discoveries of a cluster mass test cannot be precisely localized in time or in space. We propose a new multiple comparisons procedure, the cluster depth tests, that both controls the FWER while allowing an interpretation at the time point level. We show the conditions for a strong control of the FWER, and a simulation study shows that the cluster depth tests achieve large power and guarantee the FWER even in the presence of physiologically plausible effects. By having an interpretation at the time point/voxel level, the cluster depth tests make it possible to take full advantage of the high temporal resolution of EEG recording and give a precise timing of the start and end of the significant effects.

Ranking procedures for repeated measures designs with missing data: Estimation, testing and asymptotic theory.

We develop purely nonparametric methods for the analysis of repeated measures designs with missing values. Hypotheses are formulated in terms of purely nonparametric treatment effects. In particular, data can have different shapes even under the null hypothesis and therefore, a solution to the nonparametric Behrens-Fisher problem in repeated measures designs will be presented. Moreover, global testing and multiple contrast test procedures as well as simultaneous confidence intervals for the treatment effects of interest will be developed. All methods can be applied for the analysis of metric, discrete, ordinal, and even binary data in a unified way. Extensive simulation studies indicate a satisfactory control of the nominal type-I error rate, even for small sample sizes and a high amount of missing data (up to 30%). We apply the newly developed methodology to a real data set, demonstrating its application and interpretation.

Error rate control for classification rules in multiclass mixture models.

In the context of finite mixture models one considers the problem of classifying as many observations as possible in the classes of interest while controlling the classification error rate in these same classes. Similar to what is done in the framework of statistical test theory, different type I and type II-like classification error rates can be defined, along with their associated optimal rules, where optimality is defined as minimizing type II error rate while controlling type I error rate at some nominal level. It is first shown that finding an optimal classification rule boils down to searching an optimal region in the observation space where to apply the classical Maximum A Posteriori (MAP) rule. Depending on the misclassification rate to be controlled, the shape of the optimal region is provided, along with a heuristic to compute the optimal classification rule in practice. In particular, a multiclass FDR-like optimal rule is defined and compared to the thresholded MAP rules that is used in most applications. It is shown on both simulated and real datasets that the FDR-like optimal rule may be significantly less conservative than the thresholded MAP rule.

Data-driven selection of the number of change-points via error rate control

In multiple change-point analysis, one of the main difficulties is to determine the number of change-points. Various consistent selection methods, including the use of Schwarz information criterion and cross-validation, have been proposed to balance the model fitting and complexity. However, there is lack of systematic approaches to provide theoretical guarantee of significance in determining the number of changes. In this paper, we introduce a data-adaptive selection procedure via error rate control based on order-preserving sample-splitting, which is applicable to most existing change-point methods. The key idea is to construct a series of statistics with global symmetry property and then utilize the symmetry to derive a data-driven threshold. Under this general framework, we are able to rigorously investigate the false discovery proportion control, and show that the proposed method controls the false discovery rate (FDR) asymptotically under mild conditions while retaining the true change-points. Numerical experiments indicate that our selection procedure works well for many change-detection methods and is able to yield accurate FDR control in finite samples.

Improved group sequential Holm procedures for testing multiple correlated hypotheses over time

ABSTRACT Clinical trials can typically feature two different types of multiple inference: testing of more than one null hypothesis and testing at multiple time points. These modes of multiplicity are closely related mathematically but distinct statistically and philosophically. Regulatory agencies require strong control of the family-wise error rate (FWER), the risk of falsely rejecting any null hypothesis at any analysis. The correlations between test statistics at interim analyses and the final analysis are therefore routinely used in group sequential designs to achieve less conservative critical values. However, the same type of correlations between different comparisons, endpoints or sub-populations are less commonly used. As a result, FWER is in practice often controlled conservatively for commonly applied procedures. Repeated testing of the same null hypothesis may give changing results, when the hypothesis is rejected at an interim but accepted at the final analysis. The mathematically correct overall rejection is at odds with an inference theoretic approach and with common sense. We discuss these two issues, of incorporating correlations and how to interpret time-changing conclusions, and provide case studies where power can be increased while adhering to sound statistical principles.

Leveraging auxiliary data from arbitrary distributions to boost GWAS discovery with Flexible cFDR.

Genome-wide association studies (GWAS) have identified thousands of genetic variants that are associated with complex traits. However, a stringent significance threshold is required to identify robust genetic associations. Leveraging relevant auxiliary covariates has the potential to boost statistical power to exceed the significance threshold. Particularly, abundant pleiotropy and the non-random distribution of SNPs across various functional categories suggests that leveraging GWAS test statistics from related traits and/or functional genomic data may boost GWAS discovery. While type 1 error rate control has become standard in GWAS, control of the false discovery rate can be a more powerful approach. The conditional false discovery rate (cFDR) extends the standard FDR framework by conditioning on auxiliary data to call significant associations, but current implementations are restricted to auxiliary data satisfying specific parametric distributions, typically GWAS p-values for related traits. We relax these distributional assumptions, enabling an extension of the cFDR framework that supports auxiliary covariates from arbitrary continuous distributions ("Flexible cFDR"). Our method can be applied iteratively, thereby supporting multi-dimensional covariate data. Through simulations we show that Flexible cFDR increases sensitivity whilst controlling FDR after one or several iterations. We further demonstrate its practical potential through application to an asthma GWAS, leveraging various functional genomic data to find additional genetic associations for asthma, which we validate in the larger, independent, UK Biobank data resource.

Identifying pleiotropic genes for complex phenotypes with summary statistics from a perspective of composite null hypothesis testing.

Pleiotropy has important implication on genetic connection among complex phenotypes and facilitates our understanding of disease etiology. Genome-wide association studies provide an unprecedented opportunity to detect pleiotropic associations; however, efficient pleiotropy test methods are still lacking. We here consider pleiotropy identification from a methodological perspective of high-dimensional composite null hypothesis and propose a powerful gene-based method called MAIUP. MAIUP is constructed based on the traditional intersection-union test with two sets of independent P-values as input and follows a novel idea that was originally proposed under the high-dimensional mediation analysis framework. The key improvement of MAIUP is that it takes the composite null nature of pleiotropy test into account by fitting a three-component mixture null distribution, which can ultimately generate well-calibrated P-values for effective control of family-wise error rate and false discover rate. Another attractive advantage of MAIUP is its ability to effectively address the issue of overlapping subjects commonly encountered in association studies. Simulation studies demonstrate that compared with other methods, only MAIUP can maintain correct type I error control and has higher power across a wide range of scenarios. We apply MAIUP to detect shared associated genes among 14 psychiatric disorders with summary statistics and discover many new pleiotropic genes that are otherwise not identified if failing to account for the issue of composite null hypothesis testing. Functional and enrichment analyses offer additional evidence supporting the validity of these identified pleiotropic genes associated with psychiatric disorders. Overall, MAIUP represents an efficient method for pleiotropy identification.

A Stepwise Procedure for Toxicity Studies Based on Ratio of two means

This paper proposes a stepwise confidence set procedure for identifying equivalence or safety of compounds in a toxicity study under heteroscedasticity of variances for a normally distributed data. The problem of statistical methodology for drug safety is the control of the familywise error rate (FWER). Hence, we construct a confidence set procedure for toxicological evaluation and incorporating the partitioning principle with a case of heteroscedascity of variances under normal assumption. Our simulation studies demonstrated that the power of the procedures for heterogeneity of variances increases with increasing in ratio of means.

Note on a stepwise procedure for rejecting at least k out of m hypotheses: A simple Holm-type formulation and proof.

Mielke etal. (2021) proposed a stepwise multiple testing procedure (MTP) based on marginal p-values for rejecting at least k out of m null hypotheses. Briefly, the MTP cannot reject less than k hypotheses, but this property improves the power to reject k or more hypotheses relative to the stepdown MTP of Holm (1979). Mielke etal. discussed why such an MTP is of interest in the context of biosimilarity developments. This article describes how a slight modification of Holm's simple direct arguments can be used as an alternative to the closed-testing arguments of Mielke etal. to show the strong control of the family-wise error rate. This modification is based on a Holm-type formulation of the rejection algorithm where each step involves a single ordered p-value. With k equal to one, the stepwise MTP reduces to Holm's stepdown MTP. The MTP is valid quite generally. A version of the MTP with weights for hypotheses is also described and discussed. As in the case without weights: (a) a modification of Holm's arguments can be used; (b) with k equal to one, the MTP reduces to Holm's MTP with weights; and (c) the MTP is valid quite generally.