Standard Permutation Test Research Articles

Permutation-based multiple testing corrections for -values and confidence intervals for cluster randomized trials.

In this article, we derive and compare methods to derive P-values and sets of confidence intervals with strong control of the family-wise error rates and coverage for estimates of treatment effects in cluster randomized trials with multiple outcomes. There are few methods for P-value corrections and deriving confidence intervals, limiting their application in this setting. We discuss the methods of Bonferroni, Holm, and Romano and Wolf and adapt them to cluster randomized trial inference using permutation-based methods with different test statistics. We develop a novel search procedure for confidence set limits using permutation tests to produce a set of confidence intervals under each method of correction. We conduct a simulation-based study to compare family-wise error rates, coverage of confidence sets, and the efficiency of each procedure in comparison to no correction using both model-based standard errors and permutation tests. We show that the Romano-Wolf type procedure has nominal error rates and coverage under non-independent correlation structures and is more efficient than the other methods in a simulation-based study. We also compare results from the analysis of a real-world trial.

Combining Paired and Two-Sample Data Using a Permutation Test

This paper presents a permutation test for the incomplete pairs setting. This situation arises in both observational and experimental studies when some of the data are in the form of a paired sample and the rest of the data comprise two independent samples. The proposed method uses the data from the two types of samples to test the difference between the mean responses. Our test statistic combines the observed mean difference for the complete pairs with the difference between the two means of the independent samples. The randomizations are carried out as is typically done with standard permutation tests for paired and independent samples. We show by a simulation study that our statistic performs well in comparison to other methods.

The Chi-Square Test of Distance Correlation

Distance correlation has gained much recent attention in the data science community: the sample statistic is straightforward to compute and asymptotically equals zero if and only if independence, making it an ideal choice to discover any type of dependency structure given sufficient sample size. One major bottleneck is the testing process: because the null distribution of distance correlation depends on the underlying random variables and metric choice, it typically requires a permutation test to estimate the null and compute the p-value, which is very costly for large amount of data. To overcome the difficulty, in this article, we propose a chi-squared test for distance correlation. Method-wise, the chi-squared test is nonparametric, extremely fast, and applicable to bias-corrected distance correlation using any strong negative type metric or characteristic kernel. The test exhibits a similar testing power as the standard permutation test, and can be used for K-sample and partial testing. Theory-wise, we show that the underlying chi-squared distribution well approximates and dominates the limiting null distribution in upper tail, prove the chi-squared test can be valid and universally consistent for testing independence, and establish a testing power inequality with respect to the permutation test. Supplementary files for this article are available online.

When Your Permutation Test is Doomed to Fail

A two-tailed test comparing the means of two independent populations is perhaps the most commonly used hypothesis test in quantitative research, featured centrally in medical research, A/B testing, and throughout the sciences. When data are skewed, the standard two-tailed t test is not appropriate and the permutation test comparing the two means (or medians) has been a widely recommended alternative, with statistical authors and statistical software packages touting the permutation test’s utility, particularly for small samples. In this presentation, we illustrate that when the two samples are skewed and the sample sizes are unequal, the two-tailed permutation test (as traditionally implemented) can in some cases have power equal to zero, even when the k highest values in the combined data are all found in the group with k observations. Further, in many cases the standard permutation test exhibits decreasing power as the total sample size increases! We illustrate the causes of these perverse properties via both simulation and real-world examples, and we recommend approaches for ameliorating or avoiding these potential problems.

Remote Tool-Based Adjudication for Grading Diabetic Retinopathy.

PurposeTo present and evaluate a remote, tool-based system and structured grading rubric for adjudicating image-based diabetic retinopathy (DR) grades.MethodsWe compared three different procedures for adjudicating DR severity assessments among retina specialist panels, including (1) in-person adjudication based on a previously described procedure (Baseline), (2) remote, tool-based adjudication for assessing DR severity alone (TA), and (3) remote, tool-based adjudication using a feature-based rubric (TA-F). We developed a system allowing graders to review images remotely and asynchronously. For both TA and TA-F approaches, images with disagreement were reviewed by all graders in a round-robin fashion until disagreements were resolved. Five panels of three retina specialists each adjudicated a set of 499 retinal fundus images (1 panel using Baseline, 2 using TA, and 2 using TA-F adjudication). Reliability was measured as grade agreement among the panels using Cohen's quadratically weighted kappa. Efficiency was measured as the number of rounds needed to reach a consensus for tool-based adjudication.ResultsThe grades from remote, tool-based adjudication showed high agreement with the Baseline procedure, with Cohen's kappa scores of 0.948 and 0.943 for the two TA panels, and 0.921 and 0.963 for the two TA-F panels. Cases adjudicated using TA-F were resolved in fewer rounds compared with TA (P < 0.001; standard permutation test).ConclusionsRemote, tool-based adjudication presents a flexible and reliable alternative to in-person adjudication for DR diagnosis. Feature-based rubrics can help accelerate consensus for tool-based adjudication of DR without compromising label quality.Translational RelevanceThis approach can generate reference standards to validate automated methods, and resolve ambiguous diagnoses by integrating into existing telemedical workflows.

Meta-analysis of Voxel-Based Neuroimaging Studies using Seed-based d Mapping with Permutation of Subject Images (SDM-PSI).

Most methods for conducting meta-analysis of voxel-based neuroimaging studies do not assess whether effects are not null, but whether there is a convergence of peaks of statistical significance, and reduce the assessment of the evidence to a binary classification exclusively based on p-values (i.e., voxels can only be "statistically significant" or "non-statistically significant"). Here, we detail how to conduct a meta-analysis using Seed-based d Mapping with Permutation of Subject Images (SDM-PSI), a novel method that uses a standard permutation test to assess whether effects are not null. We also show how to grade the strength of the evidence according to a set of criteria that considers a range of statistical significance levels (from more liberal to more conservative), the amount of data or the detection of potential biases (e.g., small-study effect and excess of significance). To exemplify the procedure, we detail the conduction of a meta-analysis of voxel-based morphometry studies in obsessive-compulsive disorder, and we provide all the data already extracted from the manuscripts to allow the reader to replicate the meta-analysis easily. SDM-PSI can also be used for meta-analyses of functional magnetic resonance imaging, diffusion tensor imaging, position emission tomography and surface-based morphometry studies.

AP-SKAT: highly-efficient genome-wide rare variant association test.

BackgroundGenome-wide association studies have revealed associations between single-nucleotide polymorphisms (SNPs) and phenotypes such as disease symptoms and drug tolerance. To address the small sample size for rare variants, association studies tend to group gene or pathway level variants and evaluate the effect on the set of variants. One of such strategies, known as the sequential kernel association test (SKAT), is a widely used collapsing method. However, the reported p-values from SKAT tend to be biased because the asymptotic property of the statistic is used to calculate the p-value. Although this bias can be corrected by applying permutation procedures for the test statistics, the computational cost of obtaining p-values with high resolution is prohibitive.ResultsTo address this problem, we devise an adaptive SKAT procedure termed AP-SKAT that efficiently classifies significant SNP sets and ranks them according to the permuted p-values. Our procedure adaptively stops the permutation test when the significance level is outside some confidence interval of the estimated p-value for a binomial distribution. To evaluate the performance, we first compare the power and sample size calculation and the type I error rates estimate of SKAT, SKAT-O, and the proposed procedure using genotype data in the SKAT R package and from 1000 Genome Project. Through computational experiments using whole genome sequencing and SNP array data, we show that our proposed procedure is highly efficient and has comparable accuracy to the standard procedure.ConclusionsFor several types of genetic data, the developed procedure could achieve competitive power and sample size under small and large sample size conditions with controlling considerable type I error rates, and estimate p-values of significant SNP sets that are consistent with those estimated by the standard permutation test within a realistic time. This demonstrates that the procedure is sufficiently powerful for recent whole genome sequencing and SNP array data with increasing numbers of phenotypes. Additionally, this procedure can be used in other association tests by employing alternative methods to calculate the statistics.

Why We Should Not Be Indifferent to Specification Choices for Difference-in-Differences.

To evaluate the effects of specification choices on the accuracy of estimates in difference-in-differences (DID) models. Process-of-care quality data from Hospital Compare between 2003 and 2009. We performed a Monte Carlo simulation experiment to estimate the effect of an imaginary policy on quality. The experiment was performed for three different scenarios in which the probability of treatment was (1) unrelated to pre-intervention performance; (2) positively correlated with pre-intervention levels of performance; and (3) positively correlated with pre-intervention trends in performance. We estimated alternative DID models that varied with respect to the choice of data intervals, the comparison group, and the method of obtaining inference. We assessed estimator bias as the mean absolute deviation between estimated program effects and their true value. We evaluated the accuracy of inferences through statistical power and rates of false rejection of the null hypothesis. Performance of alternative specifications varied dramatically when the probability of treatment was correlated with pre-intervention levels or trends. In these cases, propensity score matching resulted in much more accurate point estimates. The use of permutation tests resulted in lower false rejection rates for the highly biased estimators, but the use of clustered standard errors resulted in slightly lower false rejection rates for the matching estimators. When treatment and comparison groups differed on pre-intervention levels or trends, our results supported specifications for DID models that include matching for more accurate point estimates and models using clustered standard errors or permutation tests for better inference. Based on our findings, we propose a checklist for DID analysis.

Maintaining the exchangeability assumption for a two-group permutation test in the non-randomized setting

In this note, we develop a new two-group bootstrap-permutation test that utilizes the tail-extrapolated quantile function estimator for the bootstrap component. This test is an extension of the standard two-group permutation test, that through its construction is defined to meet the exchangeability assumption, and thus it guarantees that the type I error is appropriately bounded by definition. This methodology is particularly useful in the non-randomized two-group setting for which the exchangeability assumption for the traditional two-group permutation test is untestable. We develop some theoretical results for the new test, followed by a simulation study and an example.

PERMORY: an LD-exploiting permutation test algorithm for powerful genome-wide association testing

In genome-wide association studies (GWAS) examining hundreds of thousands of genetic markers, the potentially high number of false positive findings requires statistical correction for multiple testing. Permutation tests are considered the gold standard for multiple testing correction in GWAS, because they simultaneously provide unbiased type I error control and high power. At the same time, they demand heavy computational effort, especially with large-scale datasets of modern GWAS. In recent years, the computational problem has been circumvented by using approximations to permutation tests, which, however, may be biased. We have tackled the original computational problem of permutation testing in GWAS and herein present a permutation test algorithm one or more orders of magnitude faster than existing implementations, which enables efficient permutation testing on a genome-wide scale. Our algorithm does not rely on any kind of approximation and hence produces unbiased results identical to a standard permutation test. A noteworthy feature of our algorithm is a particularly effective performance when analyzing high-density marker sets. Freely available on the web at http://www.permory.org.

Testing and interval estimation for two-sample survival comparisons with small sample sizes and unequal censoring

While the commonly used log-rank test for survival times between 2 groups enjoys many desirable properties, sometimes the log-rank test and its related linear rank tests perform poorly when sample sizes are small. Similar concerns apply to interval estimates for treatment differences in this setting, though their properties are less well known. Standard permutation tests are one option, but these are not in general valid when the underlying censoring distributions in the comparison groups are unequal. We develop 2 methods for testing and interval estimation, for use with small samples and possibly unequal censoring, based on first imputing survival and censoring times and then applying permutation methods. One provides a heuristic justification for the approach proposed recently by Heinze and others (2003, Exact log-rank tests for unequal follow-up. Biometrics 59, 1151-1157). Simulation studies show that the proposed methods have good Type I error and power properties. For accelerated failure time models, compared to the asymptotic methods of Jin and others (2003, Rank-based inference for the accelerated failure time model. Biometrika 90, 341-353), the proposed methods yield confidence intervals with better coverage probabilities in small-sample settings and similar efficiency when sample sizes are large. The proposed methods are illustrated with data from a cancer study and an AIDS clinical trial.

Rotation Testing in Gene Set Enrichment Analysis for Small Direct Comparison Experiments

Gene Set Enrichment Analysis (GSEA) is a method for analysing gene expression data with a focus on a priori defined gene sets. The permutation test generally used in GSEA for testing the significance of gene set enrichment involves permutation of a phenotype vector and is developed for data from an indirect comparison design, i.e. unpaired data. In some studies the samples representing two phenotypes are paired, e.g. samples taken from a patient before and after treatment, or if samples representing two phenotypes are hybridised to the same two-channel array (direct comparison design). In this paper we will focus on data from direct comparison experiments, but the methods can be applied to paired data in general. For these types of data, a standard permutation test for paired data that randomly re-signs samples can be used. However, if the sample size is very small, which is often the case for a direct comparison design, a permutation test will give very imprecise estimates of the p-values. Here we propose using a rotation test rather than a permutation test for estimation of significance in GSEA of direct comparison data with a limited number of samples. Our proposed rotation test makes GSEA applicable to direct comparison data with few samples, by depending on rotations of the data instead of permutations. The rotation test is a generalisation of the permutation test, and can in addition be used on indirect comparison data and for testing significance of other types of test statistics outside the GSEA framework.

A Randomization Test for Controlling Population Stratification in Whole-Genome Association Studies

Population stratification can be a serious obstacle in the analysis of genomewide association studies. We propose a method for evaluating the significance of association scores in whole-genome cohorts with stratification. Our approach is a randomization test akin to a standard permutation test. It conditions on the genotype matrix and thus takes into account not only the population structure but also the complex linkage disequilibrium structure of the genome. As we show in simulation experiments, our method achieves higher power and significantly better control over false-positive rates than do existing methods. In addition, it can be easily applied to whole-genome association studies.

An ‘exact’ two-group median test with an extension to censored data

In this note, we develop an ‘exact’ bootstrap test for comparing medians that is applicable for both censored and uncensored data. It is deemed ‘exact’ in the sense that an analytic solution is available such that no Monte-Carlo methods are necessary. The advantage of our analytic bootstrap hypothesis testing approach, as compared to standard parametric tests or permutation tests (oftentimes denoted randomization tests), is that the nonparametric bootstrap approach is completely nonparametric and asymptotically correct. In addition, we will illustrate that it has good finite sample behavior and is robust to departures from the classic exchangeability assumption that is a key component of permutation testing.

An enhanced sign test for dependent binary data with small numbers of clusters

The classical sign test is proper for hypotheses about a specified success probability, when based on independent trials. For such a hypothesis we introduce a new exact test that is appropriate with clustered binary data. It combines a permutation approach and an exact parametric bootstrap calculation. Simulation studies show it to be superior to a sign test based on aggregated cluster level data. The new test is more powerful than or comparable to a standard permutation test whenever (1) the number of clusters is small or (2) for larger cluster numbers under strong clustering. The results from a chemical repellency trial are used to illustrate three legitimate test methods.

A Fast Method for Computing High-Significance Disease Association in Large Population-Based Studies

Because of rapid progress in genotyping techniques, many large-scale, genomewide disease-association studies are now under way. Typically, the disorders examined are multifactorial, and, therefore, researchers seeking association must consider interactions among loci and between loci and other factors. One of the challenges of large disease-association studies is obtaining accurate estimates of the significance of discovered associations. The linkage disequilibrium between SNPs makes the tests highly dependent, and dependency worsens when interactions are tested. The standard way of assigning significance (P value) is by a permutation test. Unfortunately, in large studies, it is prohibitively slow to compute low P values by this method. We present here a faster algorithm for accurately calculating low P values in case-control association studies. Unlike with several previous methods, we do not assume a specific distribution of the traits, given the genotypes. Our method is based on importance sampling and on accounting for the decay in linkage disequilibrium along the chromosome. The algorithm is dramatically faster than the standard permutation test. On data sets mimicking medium-to-large association studies, it speeds up computation by a factor of 5,000-100,000, sometimes reducing running times from years to minutes. Thus, our method significantly increases the problem-size range for which accurate, meaningful association results are attainable.

Non-parametric inference for balanced randomization designs

This paper compares the properties of two balanced randomization schemes which allocate the known number n of subjects among several treatments. According to the first procedure, the so-called truncated multinomial randomization design, the allocation process starts with the uniform distribution, until a treatment receives the prescribed number of subjects, after which this uniform distribution switches to the remaining treatments, and so on. The second scheme, the random allocation rule, selects at random any assignment of the given number of subjects per treatment. The limiting behavior of these two procedures is shown to be quite different in the sense that for the random allocation rule the instant, at which a treatment gets its prescribed number of subjects, comes much later (after n - O ( 1 ) rather than n - O ( n ) subject assignments.) The large sample distribution of standard permutation tests is obtained, and formulas for the accidental bias and for the selection bias of both procedures are derived.

Efficient Computation of Significance Levels for Multiple Associations in Large Studies of Correlated Data, Including Genomewide Association Studies

Large exploratory studies, including candidate-gene-association testing, genomewide linkage-disequilibrium scans, and array-expression experiments, are becoming increasingly common. A serious problem for such studies is that statistical power is compromised by the need to control the false-positive rate for a large family of tests. Because multiple true associations are anticipated, methods have been proposed that combine evidence from the most significant tests, as a more powerful alternative to individually adjusted tests. The practical application of these methods is currently limited by a reliance on permutation testing to account for the correlated nature of single-nucleotide polymorphism (SNP)-association data. On a genomewide scale, this is both very time-consuming and impractical for repeated explorations with standard marker panels. Here, we alleviate these problems by fitting analytic distributions to the empirical distribution of combined evidence. We fit extreme-value distributions for fixed lengths of combined evidence and a beta distribution for the most significant length. An initial phase of permutation sampling is required to fit these distributions, but it can be completed more quickly than a simple permutation test and need be done only once for each panel of tests, after which the fitted parameters give a reusable calibration of the panel. Our approach is also a more efficient alternative to a standard permutation test. We demonstrate the accuracy of our approach and compare its efficiency with that of permutation tests on genomewide SNP data released by the International HapMap Consortium. The estimation of analytic distributions for combined evidence will allow these powerful methods to be applied more widely in large exploratory studies.

Bias properties and nonparametric inference for truncated binomial randomization

This paper examines certain properties of the truncated binomial randomization scheme for clinical trials. In particular, we derive the degree to which the scheme is susceptible to accidental bias and discuss the large sample properties of standard nonparametric permutation tests under a randomization model.

The impact of treatment allocation procedures on nominal significance levels and bias

Complete randomization, the simplest method for allocating treatments to patients in clinical trials, can serve as a basis for inferential procedures using standard permutation tests, because the method ensures that each sequence of allocations is equally likely. No other method of allocation possesses this property. However, many clinical trials employ allocation methods that force balance of covariates across treatment groups. With these methods, some allocation sequences are impossible or highly unlikely so that standard permutation tests are technically invalidated. In this article we investigate whether standard permutation tests for binary outcomes are likely to yield distorted nominal p values in practical applications of these alternative allocation methods. A sample of completed trials conducted by the Eastern Cooperative Oncology Group serves as a basis on which to construct simulations. Our results indicate that nominal p values can be conservative, but are not likely to be severely distorted if the analysis is stratified by important covariates used as allocation prompts. Moreover the inherent conservativeness of exact methods due to discreteness tends to dominate any additional conservativeness due to nonrandom designs. In addition, we investigate the relationship of treatment allocation methods with bias in estimates from a logistic model when important covariates are unknown. This bias is the same for all asymptotically balanced allocation methods and is significant but not disastrous.