Error Rate Control Research Articles

Three-outcome designs for external pilot trials with progression criteria

BackgroundWhether or not to progress from a pilot study to a definitive trial is often guided by pre-specified quantitative progression criteria with three possible outcomes. Although the choice of these progression criteria will help to determine the statistical properties of the pilot trial, there is a lack of research examining how they, or the pilot sample size, should be determined.MethodsWe review three-outcome trial designs originally proposed in the phase II oncology setting and extend these to the case of external pilots, proposing a unified framework based on univariate hypothesis tests and the control of frequentist error rates. We apply this framework to an example and compare against a simple two-outcome alternative.ResultsWe find that three-outcome designs can be used in the pilot setting, although they are not generally more efficient than simpler two-outcome alternatives. We show that three-outcome designs can help allow for other sources of information or other stakeholders to feed into progression decisions in the event of a borderline result, but this will come at the cost of a larger pilot sample size than the two-outcome case. We also show that three-outcome designs can be used to allow adjustments to be made to the intervention or trial design before commencing the definitive trial, providing the effect of the adjustment can be accurately predicted at the pilot design stage. An R package, tout, is provided to optimise progression criteria and pilot sample size.ConclusionsThe proposed three-outcome framework provides a way to optimise pilot trial progression criteria and sample size in a way that leads to desired operating characteristics. It can be applied whether or not an adjustment following the pilot trial is anticipated, but will generally lead to larger sample size requirements than simpler two-outcome alternatives.

Analysis of factors associated with use of real-world data in single technology appraisals of cancer drugs by the National Institute for Health and Care Excellence

ObjectivesThis study investigates factors associated with use of real-world data (RWD) in economic modelling for single technology appraisals (STAs) of cancer drugs by the National Institute for Health and Care Excellence (NICE) to improve systematic understanding of the use of RWD. MethodsThe data were extracted from STAs of cancer drugs, for which NICE issued guidance between January 2011 and December 2022 (n=267). Binary regression was used to test hypotheses concerning the greater or lesser use of RWD. Bonferroni-Holm correction was used to control error rates in multiple hypotheses tests. Several explanatory variables were considered in this analysis, including time (Time), incidence rate of disease (IR), availability of direct treatment comparison (AD), generalisability of trial data (GE), maturity of survival data in trial (MS) and previous technology recommendations by NICE (PR). The primary outcome variable was any use of RWD. Secondary outcome variables were specific uses of RWD in economic models. ResultsAD had a statistical negative association with any use of RWD whereas no associations with non-parametric and parametric use of RWD were found. Time had several statistical associations with use of RWD (validating survival distributions for the intervention, estimating progression-free survival for the intervention, estimating overall survival for comparators and transition probabilities). ConclusionsRWD were more likely to be used in economic modelling of cancer drugs when randomised controlled trials failed to provide relevant clinical information of the drug for appraisals, particularly in the absence of direct treatment comparisons. These results, based on analysis of data systematically collected from previous appraisals, suggest that uses of RWD were associated with data gaps in the economic modelling. While this result may support some of the claimed advantages of using RWD when evidence is absent, the question, the extent to which use of RWD in indirect treatment comparisons reduces uncertainty is still to be determined.

Testing for a treatment effect in aselected subgroup.

There is a growing interest in clinical trials that investigate how patients may respond differently to an experimental treatment depending on the basis of some biomarker measured on a continuous scale, and in particular to identify some threshold value for the biomarker above which a positive treatment effect can be considered to have been demonstrated. This can be statistically challenging when the same data are used both to select the threshold and to test the treatment effect in the subpopulation that it defines. This paper describes a hierarchical testing framework to give familywise type I error rate control in this setting and proposes two specific tests that can be used within this framework. One, a simple test based on the estimated value from a linear regression model with treatment by biomarker interaction, is powerful but can lead to type I error rate inflation if the assumptions of the linear model are not met. The other is more robust to these assumptions, but can be slightly less powerful when the assumptions hold.

Patients and Healthcare Professionals’ Perceptions of Stethoscope Barriers

Stethoscope hygiene is rarely performed despite evidence that they are a potential source of nosocomial infections. Aseptic touch-free disposable stethoscope diaphragm barriers (touch-free barriers) provide a bacteriologically impervious surface; however, little is known about perceptions of touch-free barriers. The purpose of this study was to determine patients’ and healthcare professionals’ perceptions of stethoscope hygiene habits, cleanliness, and patient safety, after respondents viewed videos of stethoscope hygiene with alcohol versus a touch-free barrier, or no hygiene/no barrier. We conducted a cross-sectional study using survey methodology and convenience sampling. Participants viewed three short videos that showed a provider using each of three stethoscope hygiene techniques (barrier, alcohol, or none), followed by completion of a brief survey. Pearson’s chi-square, Fisher’s exact, Wilcoxon rank sum or Kruskal-Wallis tests were used as appropriate. Bonferroni correction was used to control error rate at 0.05 for each question. The sample size of 240 participants included 119 patients and 121 healthcare professionals. Patients rarely witnessed healthcare professionals cleaning stethoscopes, rated clean stethoscopes as important, and were more likely to choose touch-free barrier as easiest for providers. Healthcare professionals rated stethoscope hygiene as important, reported frequent stethoscope hygiene, and felt that touch-free barrier would improve workflow. Touch-free barriers were viewed positively by respondents. Hospitals should consider touch-free barriers as an important strategy to reduce nosocomial infections.

The assessment of replicability using the sum of p-values.

Statistical significance of both the original and the replication study is a commonly used criterion to assess replication attempts, also known as the two-trials rule in drug development. However, replication studies are sometimes conducted although the original study is non-significant, in which case Type-I error rate control across both studies is no longer guaranteed. We propose an alternative method to assess replicability using the sum of -values from the two studies. The approach provides a combined -value and can be calibrated to control the overall Type-I error rate at the same level as the two-trials rule but allows for replication success even if the original study is non-significant. The unweighted version requires a less restrictive level of significance at replication if the original study is already convincing which facilitates sample size reductions of up to 10%. Downweighting the original study accounts for possible bias and requires a more stringent significance level and larger sample sizes at replication. Data from four large-scale replication projects are used to illustrate and compare the proposed method with the two-trials rule, meta-analysis and Fisher's combination method.

Addressing researcher degrees of freedom through minP adjustment

When different researchers study the same research question using the same dataset they may obtain different and potentially even conflicting results. This is because there is often substantial flexibility in researchers’ analytical choices, an issue also referred to as “researcher degrees of freedom”. Combined with selective reporting of the smallest p-value or largest effect, researcher degrees of freedom may lead to an increased rate of false positive and overoptimistic results. In this paper, we address this issue by formalizing the multiplicity of analysis strategies as a multiple testing problem. As the test statistics of different analysis strategies are usually highly dependent, a naive approach such as the Bonferroni correction is inappropriate because it leads to an unacceptable loss of power. Instead, we propose using the “minP” adjustment method, which takes potential test dependencies into account and approximates the underlying null distribution of the minimal p-value through a permutation-based procedure. This procedure is known to achieve more power than simpler approaches while ensuring a weak control of the family-wise error rate. We illustrate our approach for addressing researcher degrees of freedom by applying it to a study on the impact of perioperative paO2\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$paO_2$$\\end{document} on post-operative complications after neurosurgery. A total of 48 analysis strategies are considered and adjusted using the minP procedure. This approach allows to selectively report the result of the analysis strategy yielding the most convincing evidence, while controlling the type 1 error—and thus the risk of publishing false positive results that may not be replicable.

Multiple testing of interval composite null hypotheses using randomized p-values

Equivalence tests are statistical hypothesis testing procedures that aim to establish practical equivalence rather than the usual statistical significant difference. These testing procedures are frequent in “bioequivalence studies," where one would wish to show that, for example, an existing drug and a new one under development have comparable therapeutic effects. In this article, we propose a two-stage randomized (RAND2) p-value that depends on a uniformly most powerful (UMP) p-value and an arbitrary tuning parameter c∈[0,1]\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$c\\in [0,1]$$\\end{document} for testing an interval composite null hypothesis. We investigate the behavior of the distribution function of the two p-values under the null hypothesis and alternative hypothesis for a fixed significance level t∈(0,1)\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$t\\in (0,1)$$\\end{document} and varying sample sizes. We evaluate the performance of the two p-values in estimating the proportion of true null hypotheses in multiple testing. We conduct a family-wise error rate control using an adaptive Bonferroni procedure with a plug-in estimator to account for the multiplicity that arises from our multiple hypotheses under consideration. The various claims in this research are verified using a simulation study and real-world data analysis.

Summary statistics knockoffs inference with family-wise error rate control.

Testing multiple hypotheses of conditional independence with provable error rate control is a fundamental problem with various applications. To infer conditional independence with family-wise error rate (FWER) control when only summary statistics of marginal dependence are accessible, we adopt GhostKnockoff to directly generate knockoff copies of summary statistics and propose a new filter to select features conditionally dependent on the response. In addition, we develop a computationally efficient algorithm to greatly reduce the computational cost of knockoff copies generation without sacrificing power and FWER control. Experiments on simulated data and a real dataset of Alzheimer's disease genetics demonstrate the advantage of the proposed method over existing alternatives in both statistical power and computational efficiency.

Variable Selection for High-dimensional Cox Model with Error Rate Control

Variable Selection for High-dimensional Cox Model with Error Rate Control

Adding experimental treatment arms to multi-arm multi-stage platform trials in progress.

Multi-arm multi-stage (MAMS) platform trials efficiently compare several treatments with a common control arm. Crucially MAMS designs allow for adjustment for multiplicity if required. If for example, the active treatment arms in a clinical trial relate to different dose levels or different routes of administration of a drug, the strict control of the family-wise error rate (FWER) is paramount. Suppose a further treatment becomes available, it is desirable to add this to the trial already in progress; to access both the practical and statistical benefits of the MAMS design. In any setting where control of the error rate is required, we must add corresponding hypotheses without compromising the validity of the testing procedure.To strongly control the FWER, MAMS designs use pre-planned decision rules that determine the recruitment of the next stage of the trial based on the available data. The addition of a treatment arm presents an unplanned change to the design that we must account for in the testing procedure. We demonstrate the use of the conditional error approach to add hypotheses to any testing procedure that strongly controls the FWER. We use this framework to add treatments to a MAMS trial in progress. Simulations illustrate the possible characteristics of such procedures.

Upstrapping to determine futility: predicting future outcomes nonparametrically from past data

BackgroundClinical trials often involve some form of interim monitoring to determine futility before planned trial completion. While many options for interim monitoring exist (e.g., alpha-spending, conditional power), nonparametric based interim monitoring methods are also needed to account for more complex trial designs and analyses. The upstrap is one recently proposed nonparametric method that may be applied for interim monitoring.MethodsUpstrapping is motivated by the case resampling bootstrap and involves repeatedly sampling with replacement from the interim data to simulate thousands of fully enrolled trials. The p-value is calculated for each upstrapped trial and the proportion of upstrapped trials for which the p-value criteria are met is compared with a pre-specified decision threshold. To evaluate the potential utility for upstrapping as a form of interim futility monitoring, we conducted a simulation study considering different sample sizes with several different proposed calibration strategies for the upstrap. We first compared trial rejection rates across a selection of threshold combinations to validate the upstrapping method. Then, we applied upstrapping methods to simulated clinical trial data, directly comparing their performance with more traditional alpha-spending and conditional power interim monitoring methods for futility.ResultsThe method validation demonstrated that upstrapping is much more likely to find evidence of futility in the null scenario than the alternative across a variety of simulations settings. Our three proposed approaches for calibration of the upstrap had different strengths depending on the stopping rules used. Compared to O’Brien-Fleming group sequential methods, upstrapped approaches had type I error rates that differed by at most 1.7% and expected sample size was 2–22% lower in the null scenario, while in the alternative scenario power fluctuated between 15.7% lower and 0.2% higher and expected sample size was 0–15% lower.ConclusionsIn this proof-of-concept simulation study, we evaluated the potential for upstrapping as a resampling-based method for futility monitoring in clinical trials. The trade-offs in expected sample size, power, and type I error rate control indicate that the upstrap can be calibrated to implement futility monitoring with varying degrees of aggressiveness and that performance similarities can be identified relative to considered alpha-spending and conditional power futility monitoring methods.

Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test.

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

An exhaustive ADDIS principle for online FWER control.

In this paper, we consider online multiple testing with familywise error rate (FWER) control, where the probability of committing at least one type I error will remain under control while testing a possibly infinite sequence of hypotheses over time. Currently, adaptive-discard (ADDIS) procedures seem to be the most promising online procedures with FWER control in terms of power. Now, our main contribution is a uniform improvement of the ADDIS principle and thus of all ADDIS procedures. This means, the methods we propose reject as least as much hypotheses as ADDIS procedures and in some cases even more, while maintaining FWER control. In addition, we show that there is no other FWER controlling procedure that enlarges the event of rejecting any hypothesis. Finally, we apply the new principle to derive uniform improvements of the ADDIS-Spending andADDIS-Graph.

Ferbat's test: A Monte Carlo multiple comparison procedure with a control

The present work presented an alternative MCC to the Dunnett's test, called Ferbat's test. The test replaced the root of the mean square of the residue used within Dunnett's test with another non-biased $\sigma$ estimator. The distribution of the test statistics was determined by simulation using the Monte Carlo method. Comparing the performance evaluation of these two tests, the Ferbat's test performed better in some scenarios, such as control of the experimentwise error rate for all simulated situations and higher power when the number of treatments was small and when the number of replications increases. In the other evaluation situations, the tests presented equivalent performance.

Statistical inference for diagnostic test accuracy studies with multiple comparisons.

Diagnostic accuracy studies assess the sensitivity and specificity of a new index test in relation to an established comparator or the reference standard. The development and selection of the index test are usually assumed to be conducted prior to the accuracy study. In practice, this is often violated, for instance, if the choice of the (apparently) best biomarker, model or cutpoint is based on the same data that is used later for validation purposes. In this work, we investigate several multiple comparison procedures which provide family-wise error rate control for the emerging multiple testing problem. Due to the nature of the co-primary hypothesis problem, conventional approaches for multiplicity adjustment are too conservative for the specific problem and thus need to be adapted. In an extensive simulation study, five multiple comparison procedures are compared with regard to statistical error rates in least-favourable and realistic scenarios. This covers parametric and non-parametric methods and one Bayesian approach. All methods have been implemented in the new open-source R package cases which allows us to reproduce all simulation results. Based on our numerical results, we conclude that the parametric approaches (maxT and Bonferroni) are easy to apply but can have inflated type I error rates for small sample sizes. The two investigated Bootstrap procedures, in particular the so-called pairs Bootstrap, allow for a family-wise error rate control in finite samples and in addition have a competitive statistical power.

Error‐controlled feature selection for ultrahigh‐dimensional and highly correlated feature space using deep learning

AbstractDeep learning has been at the center of analytics in recent years due to its impressive empirical success in analyzing complex data objects. Despite this success, most existing tools behave like black‐box machines, thus the increasing interest in interpretable, reliable, and robust deep learning models applicable to a broad class of applications. Feature‐selected deep learning has emerged as a promising tool in this realm. However, the recent developments do not accommodate ultrahigh‐dimensional and highly correlated features or high noise levels. In this article, we propose a novel screening and cleaning method with the aid of deep learning for a data‐adaptive multi‐resolutional discovery of highly correlated predictors with a controlled error rate. Extensive empirical evaluations over a wide range of simulated scenarios and several real datasets demonstrate the effectiveness of the proposed method in achieving high power while keeping the false discovery rate at a minimum.

Two-stage stratified designs with survival outcomes and adjustment for misclassification in predictive biomarkers.

Biomarker stratified clinical trial designs are versatile tools to assess biomarker clinical utility and address its relationship with clinical endpoints. Due to imperfect assays and/or classification rules, biomarker status is prone to errors. To account for biomarker misclassification, we consider a two-stage stratified design for survival outcomes with an adjustment for misclassification in predictive biomarkers. Compared to continuous and/or binary outcomes, the test statistics for survival outcomes with an adjustment for biomarker misclassification is much more complicated and needs to take special care. We propose to use the information from the observed biomarker status strata to construct adjusted log-rank statistics for true biomarker status strata. These adjusted log-rank statistics are then used to develop sequential tests for the global (composite) hypothesis and component-wise hypothesis. We discuss the power analysis with the control of the type-I error rate by using the correlations between the adjusted log-rank statistics within and between the design stages. Our method is illustrated with examples of the recent successful development of immunotherapy in nonsmall-cell lung cancer.

A conformal regressor for predicting negative conversion time of Omicron patients.

In light of the situation and the characteristics of Omicron, the country has continuously optimized the rules for the prevention and control of COVID-19. The global epidemic is still spreading, and new cases of infection continue to emerge in China. To facilitate the infected person to estimate the course of virus infection, a prediction model for predicting negative conversion time is proposed in this article. The clinical features of Omicron-infected patients in Shandong Province in the first half of 2022 are retrospectively studied. These features are grouped by disease diagnosis result, clinical sign, traditional Chinese medicine symptoms, and drug use. These features are input to the eXtreme Gradient Boosting (XGBoost) model, and the output is the predicted number of negative conversion days. At the same time, XGBoost is used as the underlying algorithm of the conformal prediction (CP) framework, which can realize the probability interval estimation with a controllable error rate. The results show that the proposed model has a mean absolute error of 3.54days and has the shortest interval prediction result. This shows that the method in this paper can carry more decision-making information and help people better understand the disease and self-estimate the course of the disease to a certain extent.

Profitability of technical trading rules in the Chinese stock market

This article examines the profitability of technical analysis on the Shanghai Stock Exchange Composite Index (SSEC) and the Growth Enterprise Market (GEM) Index, employing stepwise generalized error rate control procedures to address the data-snooping bias. Our comprehensive study encompasses 38,456 trading rules, extending beyond prior research by incorporating simple rules alongside their contrarian and complex counterparts. Upon rectifying data-snooping bias, we show that only a few complex rules outperform SSEC in different testing sub-periods, and no trading rules outperform GEM. As the outperforming rules are based on relatively high frequent trades, most lucrative rules yield negative returns in the out-of-sample periods once transaction costs were considered. Our new findings shed light on the efficiency of the Chinese stock market from 2010 to 2021.

Bayesian response adaptive randomization design with a composite endpoint of mortality and morbidity.

Allocating patients to treatment arms during a trial based on the observed responses accumulated up to the decision point, and sequential adaptation of this allocation, could minimize the expected number of failures or maximize total benefits to patients. In this study, we developed a Bayesian response-adaptive randomization (RAR) design targeting the endpoint of organ support-free days (OSFD) for patients admitted to the intensive care units. The OSFD is a mixture of mortality and morbidity assessed by the number of days of free of organ support within a predetermined post-randomization time-window. In the past, researchers treated OSFD as an ordinal outcome variable where the lowest category is death. We propose a novel RAR design for a composite endpoint of mortality and morbidity, for example, OSFD, by using a Bayesian mixture model with a Markov chain Monte Carlo sampling to estimate the posterior probability distribution of OSFD and determine treatment allocation ratios at each interim. Simulations were conducted to compare the performance of our proposed design under various randomization rules and different alpha spending functions. The results show that our RAR design using Bayesian inference allocated more patients to the better performing arm(s) compared to other existing adaptive rules while assuring adequate power and type I error rate control across a range of plausible clinical scenarios.