Permutation Distribution Research Articles

JMASM20: Exact Permutation Critical Values For The Kruskal-Wallis One-Way ANOVA

The exhaustive enumeration of all the permutations of the observations in an experiment is the only possible way of truly constructing exact tests of significance. The permutation paradigm requires no distributional assumptions and works well with values that are normal, almost normal and non-normally distributed. The Kruskal-Wallis test does not require the assumptions that the samples are from normal populations and that the samples have the same standard deviation. In this article, the exact permutation distribution of the Kruskal-Wallis test statistic is generated empirically by actually obtaining all the distinct permutations of an experiment. The tables of exact critical values for the Kruskal-Wallis one-way ANOVA are produced.

Likelihood approaches to the non‐parametric two‐sample problem for right‐censored data

The classical two-sample problem with random right-censoring is considered. We show that non- parametric likelihood techniques can be used to obtain tests for either the identity hypothesis or the non-parametric Behrens-Fisher hypothesis (NBFH). In the case of the identity hypothesis, a special imputed permutation distribution is used to estimate the distribution under the null hypothesis. In the case of the NBFH, simulation from the constrained non-parametric maximum likelihood estimate is used. Simulation shows that the tests using either approximation have excellent control of the type I error rate, even with quite small sample sizes. Further, for Lehmann-type alternatives the likelihood-based methods have similar power to the logrank test, while for the non-Lehmann-type alternatives tried here the likelihood-based methods have superior power.

Multivariate Nonparametric Tests of Independence

New test statistics are proposed for testing whether two random vectors are independent. Gieser and Randles, as well as Taskinen, Kankainen, and Oja have introduced and discussed multivariate extensions of the quadrant test of Blomqvist. This article serves as a sequel to this work and presents new multivariate extensions of Kendall's tau and Spearman's rho statistics. Two different approaches are discussed. First, interdirection proportions are used to estimate the cosines of angles between centered observation vectors and between differences of observation vectors. Second, covariances between affine-equivariant multivariate signs and ranks are used. The test statistics arising from these two approaches appear to be asymptotically equivalent if each vector is elliptically symmetric. The spatial sign versions are easy to compute for data in common dimensions, and they provide practical, robust alternatives to normal-theory methods. Asymptotic theory is developed to approximate the finite-sample null distributions as well, as to calculate limiting Pitman efficiencies. Small-sample null permutation distributions are also described. A simple simulation study is used to compare the proposed tests with the classical Wilks test. Finally, the theory is illustrated by an example.

Permutation tests for homogeneity based on the empirical characteristic function

In this paper, omnibus tests are proposed for testing the homogeneity of two populations. The tests are based on weighted integrals involving the empirical characteristic function. The consistency of the tests as well as their asymptotic distribution under the null hypothesis are investigated. As the decay of the weight functions tends to infinity, the test statistics approach limit values which are related to moment differences between the two populations. The test procedure is based on resampling from the permutation distribution of the test statistic. The new tests are compared with other omnibus tests for homogeneity via a Monte Carlo procedure.

On the Exact Conditional Distribution for Peto's Combined Test for Carcinogenicity Assays

The most frequently used approach to analyze data from carcinogenicity assays is the method introduced by Peto et al. [Peto, R., Pike, M. C., Day, N. E., Gray, R. G., Lee, P. N., Parish, S., Peto, J., Richards, S., Wahrendorf, J. (1980). Guidelines for simple, sensitive significance tests for carcinogenicity effects in long term animal experiments. In: Long Term and Short Term Screening Assays for Carcinogens: A Critical Appraisal. IARC Monograph on the Evaluation of the Carcenogenic Risk of Chemicals to Humans. Geneva: WHO, pp. 311–426, Annex to supplement 2]. These authors combine the logrank statistic used in the fatal context and the Mantel–Haenszel statistic appropriate for the incidental context into an overall test statistic. In this article, we investigate exact conditional distributions of this combined test statistic due to Peto et al. (1980), as well as that of its two components, within a rigorous mathematical framework. We derive exact distributions under different conditioning events, and we discuss their advantages and disadvantages. One of these conditioning events will lead to the well-known permutational distribution. These exact conditional distributions will be valid under the assumption of equal censoring distributions across the groups. An interesting result is that this assumption may be relaxed in the incidental context, at the expense of using a more restrictive condition than the one needed to obtain the permutational distribution. Clearly, less strict conditions lead to less discrete distributions and would therefore be preferable. However, relaxing the assumption of equal censoring across the experimental groups is an important goal because the censoring distributions usually differ across the experimental groups. We also show how these exact conditional distributions can be related to k × r tables and extended hypergeometric distributions. This is a very useful result because it allows one to apply efficient computer algorithms for k × r tables, such as those published by Mehta and Patel [Mehta, C. R., Patel, N. R. (1983). A network algorithm for performing Fisher's exact test in rxc contingency tables. JASA 78:427–434] or Mehta et al. [Mehta, C. R., Patel, N. R., Tsiatis, A. A. (1984). Exact significance testing to establish treatment equivalence with ordered categorical data. Biometrics 40:819–825]. Further results include limit theorems about the exact conditional distributions, which converge to the same limit as the unconditional distribution of the combined test statistic, as derived by Heimann and Neuhaus [Heimann, G., Neuhaus, G. (2001). On the asymptotic distribution for Peto's combined test for carcinogenicity assays under equal and unequal censoring. Biometrika 88:435–445] in an earlier article. We also discuss some previous proposals on how to obtain “exact” p values, and we show why these are wrong.

ABSORPTION OF INTRAVESICALLY APPLIED DRUGS: COMPARISON OF NORMAL AND ILEAL AUGMENTED RABBIT BLADDER

Drug absorption within urinary diversions has been reported to cause prolonged and higher grade toxicity. Therefore, continuous urine drainage has been recommended during chemotherapy in patients with continent urinary diversion. We developed an animal model in which to examine the significance of drug absorption in normal rabbit bladders compared with ileal augmented bladders. Ten rabbits with ileal bladder augmentation and 5 control animals were used for absorption studies with methotrexate and ofloxacin. One, 4 and 12 months after surgery the rabbits received an intravesical instillation of either drug. During 2 hours blood samples were drawn. To avoid overfilling the bladder by urine it was emptied after 30, 60, 90 and 120 minutes, and refilled with fresh solution to yield a relatively constant drug amount. After 12 months the animals were sacrificed. Area of the native bladder wall and of the intestinal segment was measured to allow the correction of absorbed drug amounts per surface area. Moreover, the median villous heights of native and augmented ileum were compared histologically. Serum levels of methotrexate and ofloxacin were determined by high performance liquid chromatography. The maximum serum concentration of the 2 drugs was typically seen after 60 to 120 minutes. During the 1-year period peak serum concentrations of ofloxacin remained consistently higher in ileal augmented than in control rabbits, although this did not achieve statistical significance at all instillation time points. For methotrexate a statistically significant difference was not shown for either time point. When absorption was corrected for total bladder surface area, the enhancement of ofloxacin absorption by ileal augmentation weakened and attained statistical significance only at the 1-month time point. Histological examinations after 12 months showed that augmented intestinal mucosa had a significantly smaller villous height than native ileum. Our data demonstrate that bladder surface is the most important factor for increased absorption but time dependent histological changes of the integrated intestinal mucosa also influence absorption. There is a broad interindividual discrepancy. Therefore, general rules in patients with urinary diversion may not be justified.

Nonparametric models for clustered data: Application to root survival study

Root survival data are discrete, have censoring and there is dependence due to clustering. The assumptions of the common semiparametric models seem to be violated for the data set under consideration. A fully nonparametric model for such clustered data is proposed and methods for testing hypotheses about the main effects and interaction are developed. Simulation studies indicate that the procedures have reasonable accuracy for moderate sample sizes. For small sample sizes, bootstrapping from the permutation distribution performs rather well for testing simple effects. The nonparametric analysis of the data set is compared with that based on semiparametric models, and the effect of allowing for the dependence caused by the clustering is examined.

Further developments on sufficient conditions for negative dependence of random variables

Let W =(W 1,…,W n) be a random vector of n independent random variables, and let R =(R 1,…,R n) be another random vector having the permutation distribution on {1,2,…, n}, independent of W . If the W i 's are ordered in the likelihood ratio order [resp. the hazard rate order, the reversed hazard rate order, and the usual stochastic order], it is shown that ( W R 1 , W R 2 ,…, W R n ) is negatively regression dependent [resp. negatively right tail dependent, negatively left tail dependent, and negatively associated]. Several applications of the main results are also given.

An exact maternal-fetal genotype incompatibility (MFG) test.

The maternal-fetal genotype incompatibility (MFG) test can be used for a variety of genetic applications concerning disease risk in offspring including testing for the presence of alleles that act directly through offspring genotypes (child allelic effects), alleles that act through maternal genotypes (maternal allelic effects), or maternal-fetal genotype incompatibilities. The log-linear version of the MFG model divides the genotype data into many cells, where each cell represents one of the possible mother, father, and child genotype combinations. Currently, tests of hypotheses about different allelic effects are accomplished by an asymptotic MFG test, but it is unknown if this is appropriate under conditions that produce small cell counts. In this report, we develop an exact MFG test that is based on the permutation distribution of cell counts. We determine by simulation the type I error and power of both the exact MFG test and the asymptotic MFG test for four different biologically relevant scenarios: a test of child allelic effects in the presence of maternal allelic effects, a test of maternal allelic effects in the presence of child allelic effects, and tests of maternal-fetal genotype incompatibility with and without child allelic effects. These simulations show that, in general, the exact test is slightly conservative whereas the asymptotic test is slightly anti-conservative. However, the asymptotic MFG test produces significantly inflated type I error rates under conditions with extreme null allele frequencies and sample sizes of 75, 100, and 150. Under these conditions, the exact test is clearly preferred over the asymptotic test. Under all other conditions that we tested, the user can safely choose either the exact test or the asymptotic test.

Tests for covariate-associated heterogeneity in IBD allele sharing of affected relatives.

Linkage studies that aim to map susceptibility genes for complex diseases commonly test for excess allele sharing among affected relatives. Conventional methods based on identical-by-descent IBD allele sharing do not allow for possible differences among families, such as arise in the case of locus heterogeneity, and thus have reduced ability to detect linkage in the presence of such heterogeneity. We investigated two approaches to test for heterogeneity in allele sharing, using a family-level covariate that may be associated with different disease mechanisms leading to differences in allele sharing. Likelihood ratio tests for heterogeneity were formulated based on an extension of the linear and exponential likelihood models developed by Kong and Cox. Alternatively, we examined the asymptotic and permutation distributions of T-tests for differences between mean allele-sharing linkage scores from two covariate-defined family subgroups, assuming exchangeability. The size and power of heterogeneity tests were evaluated for S(all) and S(pairs) allele-sharing scoring functions using data sets of families with affected sibling and cousin pairs, generated under a model of locus heterogeneity. In certain simulation scenarios, the likelihood ratio test statistics did not follow the expected asymptotic distributions. The type I error estimates for the T-statistics conformed to nominal 5 and 1% levels in all scenarios considered, and corresponding power was comparable to that of the likelihood ratio tests. Application of these tests for heterogeneity detected significant differences in allele sharing between subgroups of families with inflammatory bowel disease.

Constrained clustering of irregularly sampled spatial data

Many spatial data such as those in climatology or environmental monitoring are collected over irregular geographical locations. Furthermore, it is common to have multivariate observations at each location. We propose a method of segmentation of a region of interest based on such data that can be carried out in two steps: (1) clustering or classification of irregularly sample points and (2) segmentation of the region based on the classified points. We develop a spatially-constrained clustering algorithm for segmentation of the sample points by incorporating a geographical-constraint into the standard clustering methods. Both hierarchical and nonhierarchical methods are considered. The latter is a modification of the seeded region growing method known in image analysis. Both algorithms work on a suitable neighbourhood structure, which can for example be defined by the Delaunay triangulation of the sample points. The number of clusters is estimated by testing the significance of successive change in the within-cluster sum-of-squares relative to a null permutation distribution. The methodology is validated on simulated data and used in construction of a climatology map of Ireland based on meteorological data of daily rainfall records from 1294 stations over the period of 37 years.

Age-adjusted exact trend tests in the event of rare occurrences.

Preclinical animal carcinogenicity studies are usually concerned with testing the statistical significance of a dose-response relationship. When the response consists of a rare event such as the development of a certain type of tumor, exact statistical methods are often employed. The exact randomization trend test based on the multivariate hypergeometric distribution is less powerful in the presence of treatment-related risks other than the specified response. Particularly, the loss of power becomes more pronounced when competing risks cause progressively higher mortality rates with increasing dose, which is usual in practice. An age-adjusted form of the randomization test is proposed to adjust for this effect. Permutational distribution for Peto's cause-of-death (COD) test is also explored and compared with its asymptotic counterpart by simulation. The use of COD information has been a controversial issue due to the subjectivity in the pathologists' determinations as well as for economic reasons. The proposed age-adjusted exact test does not require COD, and it is shown to compare favorably to the COD tests via an extensive Monte Carlo simulation. Applications of the methods to two real data sets are included.

Permutation tests for univariate or multivariate analysis of variance and regression

The most appropriate strategy to be used to create a permutation distribution for tests of individual terms in complex experimental designs is currently unclear. There are often many possibilities, including restricted permutation or permutation of some form of residuals. This paper provides a summary of recent empirical and theoretical results concerning available methods and gives recommendations for their use in univariate and multivariate applications. The focus of the paper is on complex designs in analysis of variance and multiple regression (i.e., linear models). The assumption of exchangeability required for a permutation test is assured by random allocation of treatments to units in experimental work. For observational data, exchangeability is tantamount to the assumption of independent and identically distributed errors under a null hypothesis. For partial regression, the method of permutation of residuals under a reduced model has been shown to provide the best test. For analysis of variance, one must first identify exchangeable units by considering expected mean squares. Then, one may generally produce either (i) an exact test by restricting permutations or (ii) an approximate test by permuting raw data or some form of residuals. The latter can provide a more powerful test in many situations.

One-Sided two-sample and trend tests based on a modified baumgartner-weiss-schindler statistic

The nonparametric two-sample test recently proposed by Baumgartner,Weiß,and Schindler [1] is not suitable for one-tailed analyses. We propose a modification of the original Baumgartner-Weiß-Schmdler statistic which enables one-sided tests. The test based on the new modified statistic has a less conservative size and is, according to simulation results, more powerful than the one-sided Wilcoxon-Mann-Whitney test. With the modified statistic a Jonckheere-type trend test can be constructed. Simulation studies indicate that this new trend test is superior to the often employed Jonckheere-Terpstra test. Furthermore, the new tests based on the modified statistic are also powerful in case of heteroscedasticity. For all tests the exact permutation distribution was used for inference

LogXact 4.1 for Windows

Cytel Corporation is the innovator of several software programs for analyzing data using exact methods. Their product list includes StatXact for exact tests, LogXact for exact logistic regression (especially useful for unbalanced data), EaSt for planning/monitoring group sequential clinical trials, and the Egret software package for design/analysis of epidemiological analysis. A new version of their software program for analyzing logistic regression with exact methods was released this year. Version 4.1 of LogXact is written for the Microsoft Windows environment. It will run on Windows 3.1 or higher. This package was originally intended for small sample logistic regression. The previous releases of LogXact produced estimates implementing a method described by Cox (1970). The unique feature of this special-purpose package is the support for exact small sample p values and confidence intervals in addition to the commonly available approximations based on chi-squared and normal distributions. The software will perform unconditional or conditional maximum likelihood inference and conditional exact inference. In addition, the present release adds Monte Carlo methods such that one can use the software's exact methods in analyzing larger datasets. Finally, version 4.1 also includes exact inference for Poisson regression. Using the Cox method for determining estimates based on permutation distributions limited the inital releases of the software to small datasets. The new version of LogXact 4.1 adds a Monte Carlo simulation method to the logistic regression collection of tools. This new tool allows the same capabilities for handling ill-conditioned predictors without the need to completely enumerate the permutation set. Instead, one can specify the number of permutations (selected via Monte Carlo) to consider in applying the estimation methods. The maturity of this special-purpose data analysis package allows application to a larger number of datasets and should now interest those users who have heretofore considered the package unable to support their larger (dataset) analysis needs. The exact method is based on permutation distributions of the sufficient statistics and is an alternative to maximum likelihood that produces results when maximum likelihood cannot. A common example in logistic regression is the presence of a perfect predictor; for example, when we have a binary predictor for which the outcome is always positive (or negative) in the presence of the predictor. Maximum likelihood methods will either drive the coefficient to infinity or recognize the ill-conditioned problem and drop the predictor from consideration. An example will illustrate the utility of the methods. Consider the synthetic data with 16 observations where frequency is the number of observations with the associated covariate pattern and result.

Testing Equality of Survival Functions Based on Both Paired and Unpaired Censored Data

We introduce two test procedures for comparing two survival distributions on the basis of randomly right-censored data consisting of both paired and unpaired observations. Our procedures are based on generalizations of a pooled rank test statistic previously proposed for uncensored data. One generalization adapts the Prentice-Wilcoxon score, while the other adapts the Akritas score. The use of these particular scoring systems in pooled rank tests with randomly right-censored paired data has been advocated by several researchers. Our test procedures utilize the permutation distributions of the test statistics based on a novel manner of permuting the scores. Permutation versions of tests for right-censored paired data and for two independent right-censored samples that use the proposed scoring systems are obtained as special cases of our test procedures. Simulation results show that our test procedures have high power for detecting scale and location shifts in exponential and log-logistic distributions for the survival times. We also demonstrate the advantages of our test procedures in terms of utilizing randomly occurring unpaired observations that are discarded in test procedures for paired data. The tests are applied to skin graft data previously reported elsewhere.

Efficient Monte Carlo Methods for Conditional Logistic Regression

Exact inference for the logistic regression model is based on generating the permutation distribution of the sufficient statistics for the regression parameters of interest conditional on the sufficient statistics for the remaining (nuisance) parameters. Despite the availability of fast numerical algorithms for the exact computations, there are numerous instances where a data set is too large to be analyzed by the exact methods, yet too sparse or unbalanced for the maximum likelihood approach to be reliable. What is needed is a Monte Carlo alternative to the exact conditional approach which can bridge the gap between the exact and asymptotic methods of inference. The problem is technically hard because conventional Monte Carlo methods lead to massive rejection of samples that do not satisfy the linear integer constraints of the conditional distribution. We propose a network sampling approach to the Monte Carlo problem that eliminates rejection entirely. Its advantages over alternative saddlepoint and Markov Chain Monte Carlo approaches are also discussed.

Permutation tests for comparing marginal survival functions with clustered failure time data.

We propose a class of two-sample non-parametric permutation tests to compare the marginal survival distributions of two groups when the failure times are correlated within cluster, with clusters nested within each group. The permutation distribution effectively takes into account the correlation between failure times within a cluster. The method is able to handle data with clusters of either fixed or variable sizes. Moreover, this class of test statistics is sensitive to various alternatives. The size and power of the proposed tests are assessed by a series of simulation studies. The method is illustrated by application to data from the Hypertension Detection and Follow-up program trial.

Valid inference in random effects meta-analysis.

The standard approach to inference for random effects meta-analysis relies on approximating the null distribution of a test statistic by a standard normal distribution. This approximation is asymptotic on k, the number of studies, and can be substantially in error in medical meta-analyses, which often have only a few studies. This paper proposes permutation and ad hoc methods for testing with the random effects model. Under the group permutation method, we randomly switch the treatment and control group labels in each trial. This idea is similar to using a permutation distribution for a community intervention trial where communities are randomized in pairs. The permutation method theoretically controls the type I error rate for typical meta-analyses scenarios. We also suggest two ad hoc procedures. Our first suggestion is to use a t-reference distribution with k-1 degrees of freedom rather than a standard normal distribution for the usual random effects test statistic. We also investigate the use of a simple t-statistic on the reported treatment effects.

A Class of Permutation Tests of Bivariate Interchangeability

Abstract A set of permutation tests that are both exact and distribution-free are proposed to simultaneously detect differences in marginal locations and/or scales in bivariate data. The tests take advantage of the fact that when the marginal means and variances are equal, the pairwise differences are symmetrically distributed about 0 and are uncorrelated with the pairwise sums. Two statistics for detecting the marginal location and scale differences are combined in a quadratic form. A permutation distribution for this quadratic form follows from considering all 2 n conditionally equally likely sign changes on the differences. Several methods of estimating the covariance matrix of the quadratic form are examined, including conditional and unconditional (plug-in) approaches. These new tests are compared with the standard tests in the literature and, through simulation for several families of bivariate distributions, are found to compare quite favorably. This article also brings to light the largely overlooked likelihood ratio test for equal means and variances in the bivariate normal and shows its relationship to more recent approaches, including those presented here.