Choice Of Test Statistic Research Articles

A comparison of nonparametric methods for multivariate two-sample tests

Comparing two multivariate populations can be challenging when the distributional forms are unknown. In such a situation, parametric test procedures are not appropriate given that they require distributional assumptions. In this context, it may be useful to consider the application of nonparametric tests. Specifically, we investigate the application of NonParametric Combination (NPC) which is characterized by high flexibility in the choice of test statistic and is quite powerful in multivariate scenarios. Moreover, it has the advantage of being able to deal with low sample sizes which is a very common situation in real-world applications. The performance of NPC is compared with another nonparametric method well-known in the literature, namely the (depth-based) M-test. A simulation study is provided for the purposes of comparing the performance of the NPC and the data depth-based test. Finally, both test procedures are applied to a real case study.

A Study of Permutation Tests in the Context of a Problem in Primatology

Female baboons, some with infants, were observed and counts made of interactions in which females interacted with the infants of other females (so-called infant-handling). Independent of these observations, each baboon is assigned a dominance rank of “low,” “medium,”or “high.” Researchers hypothesized that females tend to handle infants of females ranked below them. The data form an array with row-labels being infant labels and columns being female labels. Entry (i, j) counts total infant handlings of infant i by female j. Each count corresponds to one of 9 combinations of female by infant/mother ranks, which induces a 3-by-3 table of total interactions. We use a permutation test to support the research hypothesis, where ranks are permuted at random. We also discuss statistical properties of our method such as choice of test statistic, power, and stability of results to individual observations. We discover that the data support a nuanced view of baboon interaction, where higher-ranked females prefer to handle down the hierarchy, while lower-ranked females must balance the desire to accede to the desires of the high-ranked females while protecting their infants from the potential risks involved in such interactions.

Incremental Model Fit Assessment in the Case of Categorical Data: Tucker–Lewis Index for Item Response Theory Modeling

The Tucker–Lewis index (TLI; Tucker & Lewis, 1973), also known as the non-normed fit index (NNFI; Bentler & Bonett, 1980), is one of the numerous incremental fit indices widely used in linear mean and covariance structure modeling, particularly in exploratory factor analysis, tools popular in prevention research. It augments information provided by other indices such as the root-mean-square error of approximation (RMSEA). In this paper, we develop and examine an analogous index for categorical item level data modeled with item response theory (IRT). The proposed Tucker–Lewis index for IRT (TLIRT) is based on Maydeu-Olivares and Joe's (2005) M_2 family of limited-information overall model fit statistics. The limited-information fit statistics have significantly better Chi-square approximation and power than traditional full-information Pearson or likelihood ratio statistics under realistic situations. Building on the incremental fit assessment principle, the TLIRT compares the fit of model under consideration along a spectrum of worst to best possible model fit scenarios. We examine the performance of the new index using simulated and empirical data. Results from a simulation study suggest that the new index behaves as theoretically expected, and it can offer additional insights about model fit not available from other sources. In addition, a more stringent cutoff value is perhaps needed than Hu and Bentler's (1999) traditional cutoff criterion with continuous variables. In the empirical data analysis, we use a data set from a measurement development project in support of cigarette smoking cessation research to illustrate the usefulness of the TLIRT. We noticed that had we only utilized the RMSEA index, we could have arrived at qualitatively different conclusions about model fit, depending on the choice of test statistics, an issue to which the TLIRT is relatively more immune.

Using randomization tests to assess treatment effects in multiple-group interrupted time series analysis.

Interrupted time series analysis (ITSA) is a popular evaluation methodology in which a single treatment unit's outcome is studied over time and the intervention is expected to "interrupt" the level and/or trend of the outcome, subsequent to its introduction. The internal validity of this analysis is strengthened considerably if the treated unit is contrasted with a comparable control group. However, multiple-group ITSA typically has small sample sizes, and parametric methods for multiple-group ITSA require strong assumptions that are unlikely to be met, possibly resulting in misleading P values. In this paper, randomization tests are introduced as a non-parametric, distribution-free option for computing exact P values. The effect of California's Proposition 99 (passed in 1988) for reducing cigarette sales is evaluated by comparing California (CA) to Montana (MT) and Idaho (ID)-the two best matched control states not exposed to any smoking reduction initiatives. Results from randomization tests are contrasted to those of interrupted time series analysis regression (ITSAREG)-a commonly used parametric approach for evaluating treatment effects in ITSA studies. Both approaches found ID and MT to be comparable to CA on their preintervention time series, and both approaches equally found CA to have statistically lower cigarette sales in the postintervention period (P<0.01). In these data, randomization tests computed P values comparable with ITSAREG, bolstering confidence in the intervention effect. Routinely including randomization tests as a complement, or alternative, to parametric methods is therefore beneficial because randomization tests are free of assumptions regarding sample size and distribution and are extremely flexible in the choice of test statistic.

DETECTING UNSPECIFIED STRUCTURE IN LOW-COUNT IMAGES

Unexpected structure in images of astronomical sources often presents itself upon visual inspection of the image, but such apparent structure may either correspond to true features in the source or be due to noise in the data. This paper presents a method for testing whether inferred structure in an image with Poisson noise represents a significant departure from a baseline (null) model of the image. To infer image structure, we conduct a Bayesian analysis of a full model that uses a multiscale component to allow flexible departures from the posited null model. As a test statistic, we use a tail probability of the posterior distribution under the full model. This choice of test statistic allows us to estimate a computationally efficient upper bound on a p-value that enables us to draw strong conclusions even when there are limited computational resources that can be devoted to simulations under the null model. We demonstrate the statistical performance of our method on simulated images. Applying our method to an X-ray image of the quasar 0730+257, we find significant evidence against the null model of a single point source and uniform background, lending support to the claim of an X-ray jet.

A new permutation technique to explore and control for spatial autocorrelation

Summary Permutation tests are important in ecology and evolution as they enable robust analysis of small sample sizes and control for various forms of dependencies among observations. A common source of dependence is spatial autocorrelation. Accounting for spatial autocorrelation is often crucial, because many ecological and evolutionary processes are spatially restricted, such as gene flow, dispersal, mate choice, inter‐ and intraspecific competition, mutualism and predation. Here we discuss various ways of controlling for spatial autocorrelation in permutation tests; we highlight their particular properties and assumptions and introduce a new permutation technique which explores and controls for spatial autocorrelation: the floating grid permutation technique (FGPT). The FGPT is a method to randomize observations with known geographical locations. Within the randomization process, the probability an observation is assigned to any of the spatial locations is a negative function of the distance between its original and assigned location. The slope of this function depends on a preset parameter, and by exploring its parameter space, non‐random ecological and evolutionary processes can be both assessed and controlled at multiple spatial scales. We show that the FGPT has acceptable type‐I‐error rates. We applied the FGPT to simulated univariate and bivariate data sets in which both negative and positive spatial autocorrelation were present. In comparison with a method that uses eigenvector decomposition to separate negative from positive spatial autocorrelation, the FGPT performed better for negative spatial autocorrelation alone, equal for positive spatial autocorrelation alone and equal or slightly worse for simultaneous negative and positive spatial autocorrelation. For the bivariate data, it performed equally to a bootstrapping technique in which sampling probabilities were weighted by distance. The FGPT benefits from a large flexibility for application to bivariate (e.g. dyadic interactions) and multivariate observations (e.g. genetic marker‐based relatedness measures) and has a large freedom in the choice of test statistic. It also has the potential to identify two spatial autocorrelation patterns, even if both result in positive spatial autocorrelation, given that they operate at different spatial scales. The Floating Grid Permutation Technique is available as the R‐package fgpt in CRAN.

Bahadur Efficiency of Sensitivity Analyses in Observational Studies

An observational study draws inferences about treatment effects when treatments are not randomly assigned, as they would be in a randomized experiment. The naive analysis of an observational study assumes that adjustments for measured covariates suffice to remove bias from nonrandom treatment assignment. A sensitivity analysis in an observational study determines the magnitude of bias from nonrandom treatment assignment that would need to be present to alter the qualitative conclusions of the naive analysis, say leading to the acceptance of a null hypothesis rejected in the naive analysis. Observational studies vary greatly in their sensitivity to unmeasured biases, but a poor choice of test statistic can lead to an exaggerated report of sensitivity to bias. The Bahadur efficiency of a sensitivity analysis is introduced, calculated, and connected to established concepts, such as the power of a sensitivity analysis and the design sensitivity. The Bahadur slope equals zero when the sensitivity parameter equals the design sensitivity, but the Bahadur slope permits more refined distinctions. Specifically, the Bahadur relative efficiency can also compare the relative performance of two test statistics at a value of the sensitivity parameter below the minimum of their design sensitivities. Adaptive procedures that combine several tests can achieve the best design sensitivity and the best Bahadur slope of their component tests. Ultimately, in sufficiently large sample sizes, design sensitivity is more important than efficiency for the power of a sensitivity analysis, and the exponential rate at which rate design sensitivity overtakes efficiency is characterized.

Greater power and computational efficiency for kernel-based association testing of sets of genetic variants.

Motivation: Set-based variance component tests have been identified as a way to increase power in association studies by aggregating weak individual effects. However, the choice of test statistic has been largely ignored even though it may play an important role in obtaining optimal power. We compared a standard statistical test—a score test—with a recently developed likelihood ratio (LR) test. Further, when correction for hidden structure is needed, or gene–gene interactions are sought, state-of-the art algorithms for both the score and LR tests can be computationally impractical. Thus we develop new computationally efficient methods.Results: After reviewing theoretical differences in performance between the score and LR tests, we find empirically on real data that the LR test generally has more power. In particular, on 15 of 17 real datasets, the LR test yielded at least as many associations as the score test—up to 23 more associations—whereas the score test yielded at most one more association than the LR test in the two remaining datasets. On synthetic data, we find that the LR test yielded up to 12% more associations, consistent with our results on real data, but also observe a regime of extremely small signal where the score test yielded up to 25% more associations than the LR test, consistent with theory. Finally, our computational speedups now enable (i) efficient LR testing when the background kernel is full rank, and (ii) efficient score testing when the background kernel changes with each test, as for gene–gene interaction tests. The latter yielded a factor of 2000 speedup on a cohort of size 13 500.Availability: Software available at http://research.microsoft.com/en-us/um/redmond/projects/MSCompBio/Fastlmm/.Contact: heckerma@microsoft.comSupplementary information: Supplementary data are available at Bioinformatics online.

Testing for Nonlinearity Using Multi-dimensional Information Entropy

The choice of test statistics has a significant impact on the nonlinearity detection of time series. This paper proposes a new test statistic - multi-dimensional information entropy for the nonlinearity test. After applying IAAFT algorithm to create surrogate data, five test statistic methods, which include time reversibility, higher order autocovariance, nonlinear prediction error, approximate entropy and multi-dimensional information entropy, are employed to test the nonlinearity of AR signal, Henon signal, Lorenz signal, ECG signal and EDA signal. By comparing the test results, it indicates that the multidimensional information entropy is an effective and stable nonlinear test statistic. The multi-dimensional information entropy has high noise immunity and more sensitive to nonlinear signal.

Exact unconditional testing procedures for comparing two independent Poisson rates

We consider seven exact unconditional testing procedures for comparing adjusted incidence rates between two groups from a Poisson process. Exact tests are always preferable due to the guarantee of test size in small to medium sample settings. Han [Comparing two independent incidence rates using conditional and unconditional exact tests. Pharm Stat. 2008;7(3):195–201] compared the performance of partial maximization p-values based on the Wald test statistic, the likelihood ratio test statistic, the score test statistic, and the conditional p-value. These four testing procedures do not perform consistently, as the results depend on the choice of test statistics for general alternatives. We consider the approach based on estimation and partial maximization, and compare these to the ones studied by Han (2008) for testing superiority. The procedures are compared with regard to the actual type I error rate and power under various conditions. An example from a biomedical research study is provided to illustrate the testing procedures. The approach based on partial maximization using the score test is recommended due to the comparable performance and computational advantage in large sample settings. Additionally, the approach based on estimation and partial maximization performs consistently for all the three test statistics, and is also recommended for use in practice.

Analysis of high‐throughput screening assays using cluster enrichment

In this paper, we describe the implementation and evaluation of a cluster-based enrichment strategy to call hits from a high-throughput screen using a typical cell-based assay of 160,000 chemical compounds. Our focus is on statistical properties of the prospective design choices throughout the analysis, including how to choose the number of clusters for optimal power, the choice of test statistic, the significance thresholds for clusters and the activity threshold for candidate hits, how to rank selected hits for carry-forward to the confirmation screen, and how to identify confirmed hits in a data-driven manner. Whereas previously the literature has focused on choice of test statistic or chemical descriptors, our studies suggest that cluster size is the more important design choice. We recommend clusters to be ranked by enrichment odds ratio, not by p-value. Our conceptually simple test statistic is seen to identify the same set of hits as more complex scoring methods proposed in the literature do. We prospectively confirm that such a cluster-based approach can outperform the naive top X approach and estimate that we improved confirmation rates by about 31.5% from 813 using the top X approach to 1187 using our cluster-based method.

An exact adaptive test with superior design sensitivity in an observational study of treatments for ovarian cancer

A sensitivity analysis in an observational study determines the magnitude of bias from nonrandom treatment assignment that would need to be present to alter the qualitative conclusions of a na\"{\i}ve analysis that presumes all biases were removed by matching or by other analytic adjustments. The power of a sensitivity analysis and the design sensitivity anticipate the outcome of a sensitivity analysis under an assumed model for the generation of the data. It is known that the power of a sensitivity analysis is affected by the choice of test statistic, and, in particular, that a statistic with good Pitman efficiency in a randomized experiment, such as Wilcoxon's signed rank statistic, may have low power in a sensitivity analysis and low design sensitivity when compared to other statistics. For instance, for an additive treatment effect and errors that are Normal or logistic or $t$-distributed with 3 degrees of freedom, Brown's combined quantile average test has Pitman efficiency close to that of Wilcoxon's test but has higher power in a sensitivity analysis, while a version of Noether's test has poor Pitman efficiency in a randomized experiment but much higher design sensitivity so it is vastly more powerful than Wilcoxon's statistic in a sensitivity analysis if the sample size is sufficiently large.

Comparison of Test Statistics for the Sequential Parallel Design

The sequential parallel design has been proposed to improve the efficiency of clinical trials in psychiatry. The design randomizes patients into three treatment groups. Each group has two phases. The three groups are (1) placebo in the first phase followed by placebo in the second phase, (2) placebo in the first phase followed by drug in the second phase, and (3) drug in the first phase followed by drug in the second phase. We consider the case of binary response data. The analysis of data in the second phase of the trial is restricted to placebo patients who failed to respond in the first phase of the trial. A crucial element in the choice of test statistics is the underlying assumptions of treatment effect in the two phases of the trial. We develop various likelihood-based statistics when the treatment effect is different in the two phases and for the special case in which the treatment effect is equal in the two phases. These statistics are compared in simulation studies to determine their underlying null and nonnull behavior. The score test under the assumption of equal treatment effects is seen to have advantages in terms of Type I error and robustness to assumptions.

Demand-Side Management and Energy Efficiency Revisited

The key finding of an influential paper that received the International Association for Energy Economists' Best Paper Award (2004) is that utilities have been overstating electricity savings and underestimating costs associated with energy efficiency demand side management (DSM) programs. This claim is based on point estimates of average DSM-related savings and costs implied by an econometric model of residential electricity demand. In this response we first argue that the choice of test statistics, by not weighting estimated savings and costs by utility electricity sales and DSM expenditures respectively, biases results in favor of rejecting the null hypothesis that utility-reported electricity savings reflect true values. We also note that utility estimates of average program savings and costs are rejected based on point estimates alone; no attempt is made to evaluate the uncertainty surrounding these estimates. We use the same data and econometric model to estimate the appropriate test statistics. We then construct nonparametric bootstrap confidence intervals. We fail to reject the average electricity savings and DSM program costs reported by utilities using both the weighted and unweighted test statistics. Our results suggest that the evidence for rejecting utility estimates of DSM savings and costs should be re-interpreted.

A Comparison of Procedures for Adaptive Choice of Location Tests in Flexible Two‐Stage Designs

AbstractAlthough linear rank statistics for the two‐sample problem are distribution free tests, their power depends on the distribution of the data. In the planning phase of an experiment, researchers are often uncertain about the shape of this distribution and so the choice of test statistic for the analysis and the determination of the required sample size are based on vague information. Adaptive designs with interim analysis can potentially overcome both problems. And in particular, adaptive tests based on a selector statistic are a solution to the first. We investigate whether adaptive tests can be usefully implemented in flexible two‐stage designs to gain power. In a simulation study, we compare several methods for choosing a test statistic for the second stage of an adaptive design based on interim data with the procedure that applies adaptive tests in both stages. We find that the latter is a sensible approach that leads to the best results in most situations considered here. The different methods are illustrated using a clinical trial example.

Resampling Procedures to Compare Two Survival Distributions in the Presence of Right-Censored Data

SUMMARY In this simulation study, the accuracy of the size of four classes of test procedures asymptotic normal test, permutation test, bootstrap hypothesis test, and bootstrap interval test were compared using the logrank and Prentice-Wilcoxon two sample test statistics. Sample sizes of 20 and 50 per group were used. Twenty observations per group yielded differences in the test procedures. When the Prentice-Wilcoxon test statistic was employed, the asymptotic test procedure was the most accurate; resampling procedures did not improve upon its accuracy. Using the logrank test statistic, the permutation or bootstrap hypothesis testing procedures were closest to the nominal significance level. The more popular asymptotic normal logrank test was consistently anticonservative. When the number of observations in each group was increased to 50, the test procedures were comparable. Power comparisons were also examined. The test with the greatest power was dependent upon the two survival patterns and the choice of test statistic.

Positive association in a two-way contingency table: a numerical study

In a doubly-ordered two-way contingency table the rows and the columns both define ordinal variables. The idea of positive association between those variables may be formalised in many different ways. Each definition is a composite alternative against which the null hypothesis (no association) may be tested. Asymptotic tests based on 8 possible test statistics are studied using Monte Carlo sampling. For some tests the exact size is shown to be very sensitive to the configuration of the marginal probabilities. Power comparisons show that the choice of test statistic must depend on the form of positive association thought to be appropriate to the problem in hand. The choice is only clear-cut when a relatively strong definition is adopted. All comparisons are made over a reference set based on multinomial sampling, but excluding tables with very small marginal totals. Some reasons are given for not using the orthodox reference set in which all marginal totals are held fixed.

Simultaneous test procedures and the choice of a test statistic in MANOVA.

Problems associated with the use of a preliminary multivariate analysis of variance (MANOVA) test in conjunction with univariate follow-up tests can be avoided if follow-up tests are carried out with a simultaneous test procedure (STP) derived from the MANOVA test statistic used for the overall test. It is argued that the choice of a MANOVA test statistic for such analyses should be based on the power and robustness of MANOVA STPS rather than on the properties of the corresponding overall tests. Monte Carlo data are presented which show that the STP based on the trace statistic V can be extremely conservative relative to the STP based on the largest root statistic R. The data suggest that the lack of robustness of the R statistic is unlikely to produce problems if the R STP is used to evaluate interpretable contrasts on linear combinations of variates of interest to the experimenter as opposed to contrasts for which coefficients referring to groups and variates are determined by the data. • When multivariate analysis of variance (MANOVA) is applied to data from a k group experiment (k > 2) with p dependent variables (p> 1), the analysis is usually regarded as a two-stage procedure. Initially an overall test of a general null hypothesis is carried out using one of the various MANOVA test statistics. If, and only if, this general hypothesis is rejected, follow-up tests are carried out in order to clarify the nature of the differences between groups. The properties of MANOVA test statistics and the properties of multiple comparison procedures for follow-up tests are usually treated as unrelated questions. Recent discussions of MANOVA test statistics, for example, make no reference to follow-up tests (Olson, 1974, 1976, 1979; Stevens, 1979). The commonly recommended practice of changing from a multivariate to a univariate test statistic for follow-up tests (Bock, 1975; Finn, 1974; Hummel & Sligo, 1971) is based on the assumptions that the only function of the initial overall test is to protect the second stage of analysis from inflation of the Type I error rate and that the efficiency of this protection does not depend in any way on the nature of the follow-up tests. As Ram