Exact Significance Levels Research Articles

Pedigree tests of transmission disequilibrium.

High-resolution mapping is essential for the positional cloning of complex disease genes. In outbred populations, linkage disequilibrium is expected to extend for short distances and could provide a powerful fine-mapping tool. Current family-based association tests use nuclear family members to define allelic transmission and controls, but ignore other types of relatives. Here we construct a general approach for scoring allelic transmission that accommodates families of any size and uses all available genotypic information. Family data allows for the construction of an expected genotype for every non-founder, and orthogonal deviates from this expectation are a measure of allelic transmission. These allelic transmission scores can be used to extend previously described tests of linkage disequilibrium for dichotomous or quantitative traits. Some of these tests are illustrated, together with a permutation framework for estimating exact significance levels. Simulation studies are used to investigate power and error rates of the approach. As a practical application, the method is used to investigate the relationship between circulating angiotensin-1 converting enzyme (ACE) levels and polymorphisms in the ACE gene using previously published data.

Some practical reasons for reconsidering the Kolmogorov‐Smirnov test

The Kolmogorov‐Smirnov test is a method for comparing the distributions of two independent groups that has virtually disappeared from applied research and introductory statistics books for the social sciences. The apparent reason is the perception that it has low power compared to methods for comparing means in particular and measures of location in general. However, extant studies comparing the power of the Kolmogorov‐Smirnov test to other methods for comparing means are limited to normal distributions having a common variance. This note points out that, even under a shift model, the Kolmogorov‐Smirnov test not only can have high power relative to methods for comparing robust measures of location, there are situations where it has higher power than methods for comparing robust measurement of location. Some additional features of the Kolmogorov‐Smirnov test are noted, and a simple S‐PLUS program for computing the exact significance level is provided. Data from a study on the effects of drinking alcohol are used to illustrate the potential advantage of the Kolmogorov‐Smirnov test.

A Note on Exact Analysis of Several 2 x 2 Tables

Yao and Tritchler (1993, Biometrics 49, 233-236) proposed an exact test for conditional independence in a series of 2 x 2 tables, and presented a fast Fourier transform (FFT) algorithm to compute exact significance levels for it. The purpose of this note is to provide a general perspective on computational methods applicable to this problem. We also assess the power and efficiency comparisons done by them.

Sensitivity analysis of non-parametric tests for censored data

The use of non-parametric tests for the two sample problem such as say the weighted logrank tests for censored data is based on the hypothesis that the probability of belonging to a groupe is the same for each individual. Our problem is : what can be said about the significance level of such a test if this hypothesis does not hold ? Using a new formulation of the weighted logrank tests statistics, it is shown that, in presence of such an heterogeneity, bounds on the exact (and unknown) significance level may be given. This results allows to lead the sensitivity analysis of those test statistics which are shown to be asymptotically normal.

Exact Inference for Kendall'sSand Spearman's ρ with Extension to Fisher's Exact Test inr×cContingency Tables

Abstract We formulate the problem of exact inference for Kendall's S and Spearman's D algebraically, using a general recursion formula developed by Smid for the score S with ties in both rankings. Analogous recursion formulas are shown to hold for the score D as well as for a log transform, F, of the score used in Fisher's exact test of independence in contingency tables. A new implementation of Mehta and Patel's network algorithm is then applied to obtain exact significance levels of either S or D for observations from both continuous and discrete distributions. A simple extension is made to obtain Fisher's exact test in r x c contingency tables. Observed CPU times for contingency table problems four to six of Mehta and Patel and problems four and five of Clarkson, Fan, and Joe are roughly 2/3 of those obtained using Clarkson's et al. implementation of the network algorithm. It is shown that a hierarchy, with F > S > D, holds regarding the rate of aggregation. An algorithm for rapid lexicographic enumera...

The Discrete Distribution Used for the Log‐rank Test Can Be Inaccurate

AbstractWhen the number of tumors is small, a significance level for the Cox‐Mantel (log‐rank) test Z is often computed using a discrete approximation to the permutation distribution. For j = 0,…, J let Nj(t) be the number of animals in group j alive and tumor‐free at the start of time t. Make a 2 × (1+J) table for each time t of the number of animals Rj(t) with newly palpated tumor out of the total Nj(t) at risk. There are a total of say K tables, one for each distinct time t with observed death or newly palpated tumor. The usual discrete approximation to the permutation distribution of Z is defined by taking tables to be independent with fixed margins Nj(t) and ΣRj(t) for all t.However, the Nj(t) are random variables for the actual permutation distribution of Z, resulting in dependence among the tables. Calculations for the exact permutation distribution are explained, and examples are given where the exact significance level differs substantially from the usual discrete approximation. The discrepancy arisis primarily because permutations with different Z‐scores under the exact distribution can be equal for the discrete approximation, inflating the approximate P‐value.

Methods for Exact Goodness-of-Fit Tests

Abstract Numerous goodness-of-fit tests with asymptotic chi-squared distributions have been proposed for discrete multivariate data, and there has been much discussion about using asymptotic results for computing critical values when there are small expected cell values. Although exact methods would be preferred in these situations, it generally is believed that such methods are computationally intractable. We propose methods for calculating exact distributions and significance levels for goodness-of-fit statistics that are computationally feasible over a wide range of models. In particular, the distribution for a simple multinomial model can be evaluated in polynomial time. For composite null hypotheses, we calculate the distribution conditional on the sufficient statistics for the nuisance parameters. We calculate the characteristic function of a distribution and invert the characteristic function using the fast Fourier transform (FFT). Our approach emphasizes the relationship between exact methods and probability formulas. Our technique, transforming the domain of the problem, is interesting for two reasons: First, algorithms that use the FFT and the convolution theorem are efficient for calculating the distribution of sums of independent statistics; and second, less storage is needed when working in the frequency domain than in the probability domain. The algorithms can be applied to general goodness-of-fit statistics and are parallelizable.

Methods for Exact Goodness-of-Fit Tests

Abstract Numerous goodness-of-fit tests with asymptotic chi-squared distributions have been proposed for discrete multivariate data, and there has been much discussion about using asymptotic results for computing critical values when there are small expected cell values. Although exact methods would be preferred in these situations, it generally is believed that such methods are computationally intractable. We propose methods for calculating exact distributions and significance levels for goodness-of-fit statistics that are computationally feasible over a wide range of models. In particular, the distribution for a simple multinomial model can be evaluated in polynomial time. For composite null hypotheses, we calculate the distribution conditional on the sufficient statistics for the nuisance parameters. We calculate the characteristic function of a distribution and invert the characteristic function using the fast Fourier transform (FFT). Our approach emphasizes the relationship between exact methods and ...

An Empirical Assessment of Saddlepoint Approximations for Testing a Logistic Regression Parameter

Saddlepoint methods provide quick and easy approximations to significance levels for conditional tests of logistic regression parameters. We evaluate the accuracies of saddlepoint approximations for three well-known conditional tests: Bartlett's test for no three-factor interaction in a 2 x 2 x 2 table, the test for trend in a series of probabilities, and the exact test of no association in stratified 2 x 2 tables with a common odds ratio. General recommendations are suggested regarding the use of saddlepoint approximations for exact conditional significance levels.

Constructing the exact significance level for a person fit statistic

This paper uses an extension of the network algorithm originally introduced by Mehta and Patel to construct exact tail probabilities for testing the general hypothesis that item responses are distributed according to the Rasch model. By assuming that item difficulties are known, the algorithm is applicable to the statistical tests either given the maximum likelihood ability estimate or conditioned on the total score. A simulation study indicates that the network algorithm is an efficient tool for computing the significance level of a person fit statistic based on test lengths of 30 items or less.

Testing the rank size rule hypothesis with an efficient estimator

The rank size rule postulates that the population sizes of a city system are distributed according to the Pareto distribution with parameter α = 1. If the sample is truncated the geometric mean is a sufficient statistic for the Pareto distribution. Since the geometric mean has a standard distribution, a parameter test on the rank size rule hypothesis with exact significance level becomes available. The test is performed for several countries. The results suggest that the textbook opinion about the validity of the Pareto law should be altered.

Importance Sampling for Estimating Exact Probabilities in Permutational Inference

Abstract This article discusses importance sampling as an alternative to conventional Monte Carlo sampling for estimating exact significance levels in a broad class of two-sample tests, including all of the linear rank tests (with or without censoring), homogeneity tests based on the chi-squared, hypergeometric, and likelihood ratio statistics, the Mantel—Haenszel trend test, and Zelen's test for a common odds ratio in several 2 × 2 contingency tables. Inference is based on randomly selecting 2 × k contingency tables from a reference set of all such tables with fixed marginals. Through a network algorithm, the tables are selected in proportion to their importance for reducing the variance of the estimated Monte Carlo p-value. Spectacular gains, up to four orders of magnitude, are achieved relative to conventional Monte Carlo sampling. The technique is illustrated on four real data sets.

The Distribution Function of Positive Definite Quadratic Forms in Normal Random Variables

Some existing representations for the cumulative distribution function of positive definite quadratic forms in normal random variables lead to inefficient computational algorithms. These inefficiencies are overcome with the derivation of some alternative recurrence relations. Some additional results concerning the distribution function and some extensions are also provided which make it possible to compute exact significance levels for certain test statistics proposed for the analysis of variance of unbalanced data.

NPSTAT: A FORTRAN-77 program to perform nonparametric variable-by-variable comparisons on two or more independent groups of data

NPSTAT compares the location (median), dispersion (variance), and overall shape of 2 or more groups of data (samples), using the Kruskal-Wallis and Van der Waerden tests (> 2 groups), Mann-Whitney and Kolmogorov-Smirnov tests (2 groups), and squared ranks test (2 or more groups). Exact or approximate significance levels of test statistics are calculated in all situations. Multivariate data are treated one variable at a time. NPSTAT reproduces results from textbooks despite previous inconsistencies with calculation and significance assessment methods. Commercial routines (e.g. in MINITAB, NAG) cover only 3 of the 5 tests programmed, but yield consistent results for these. NPSTAT is useful particularly for comparing nonnormally distributed data, multivariate data with missing values, and data measured only on an ordinal scale. It also can be used to assess outlying values.

Conditional Permutation Tests and the Propensity Score in Observational Studies

Abstract In observational studies, the distribution of treatment assignments is unknown, and therefore randomization tests are not generally applicable. However, permutation tests that condition on sample information about the treatment assignment mechanism can be applicable in observational studies, provided treatment assignment is strongly ignorable. These tests use the conditional distribution of the treatment assignments given a sufficient statistic for the unknown parameter of the propensity score. Several tests that are commonly used in observational studies are particular instances of this general procedure. Moreover, conditional permutation tests and covariance adjustment are closely related, in the sense that the conditional permutation distribution of the covariance adjusted difference leads to the same inferences as the conditional permutation distribution of the unadjusted difference of sample means. A backtrack algorithm is developed to permit efficient calculation of the exact conditional significance level, and two approximations are discussed. A clinical study of treatments for lung cancer is used to illustrate the technique. Conditional permutation tests extend previous large sample results on the propensity score by providing a basis for exact tests and confidence intervals in small observational studies when treatment assignment is strongly ignorable.

Exact Significance Testing to Establish Treatment Equivalence with Ordered Categorical Data

This communication concerns the problem of establishing the therapeutic equivalence of two treatments that are being compared on the basis of ordered categorical data. The problem is formulated as a significance test in which the null hypothesis specifies a treatment difference. An efficient numerical algorithm for computing the exact significance level is provided, along with a simple method for obtaining the asymptotic significance level. Both methods are applied to a clinical trial of a new agent versus an active control. Guidelines for when to use the exact procedure and when to rely on asymptotic theory are provided.

Significance tests and spurious correlation in regression models with autocorrelated errors

In this paper, we report some results on the exact significance level when the usual F-statistic is used in a linear regression model with autocorrelated disturbances. The exact tail area probabilities sometimes differ substantially from the nominal size used in an ‘F-test’ and from upper-bound probabilities derived by Kiviet (1979) which do not depend on the values of the regressors. A similar conclusion is also reached for the exact size of the significance tests for the spurious regressions considered by Granger and Newbold (1974, 1977). The results indicate once more that one has to be careful when using an algebraic F-test in the presence of autoregressive errors. However then too, the Durbin-Watson test is expected to indicate the presence of autocorrelation.

Significance levels and confidence intervals for permutation tests

A computational algorithm is given which calculates exact significance levels of a wide class of permutation tests in the one and two sample problems. This class includes the permutation test based on the means, locally most powerful permutation tests and linear rank tests. When a shift model is assumed confidence intervals can also be obtained. Approximate methods, based on asymptotic expansions, are also presented.

Multiple Runs Distributions: Recurrences and Critical Values

Abstract Most traditional applications of runs tests have entailed bivariate runs distributions. Extensions to more general runs tests for randomness have been hindered by the lack of exact critical values for distributions with more than two kinds of elements. Combinatorial recurrences that can be used to compute exact significance levels for small sample sizes are presented for runs distributions for ν ≥ 2 kinds of elements. Critical values for small sample sizes are tabled for ν = 2(1)6. More extensive tables are available from the author.

Multiple Runs Distributions: Recurrences and Critical Values

Abstract Most traditional applications of runs tests have entailed bivariate runs distributions. Extensions to more general runs tests for randomness have been hindered by the lack of exact critical values for distributions with more than two kinds of elements. Combinatorial recurrences that can be used to compute exact significance levels for small sample sizes are presented for runs distributions for ν ≥ 2 kinds of elements. Critical values for small sample sizes are tabled for ν = 2(1)6. More extensive tables are available from the author.