Unexpected failures of recommended tests in basic statistical analyses of ecological data

Z Kikvidze,J Moya-Laraño

doi:10.5194/we-8-67-2008

Z Kikvidze, J Moya-Laraño

Open Access

https://doi.org/10.5194/we-8-67-2008

Copy DOI

Abstract

Abstract. Ecologists, when analyzing the output of simple experiments, often have to compare statistical samples that simultaneously are of uneven size, unequal variance and distribute non-normally. Although there are special tests designed to address each of these unsuitable characteristics, it is unclear how their combination affects the tests. Here we compare the performance of recommended tests using generated data sets that simulate statistical samples typical in ecological research. We measured rates of type I and II errors, and found that common parametric tests such as ANOVA are quite robust to non-normality, uneven sample size, unequal variance, and their effect combined. ANOVA and randomization tests produced very similar results. At the same time, the t-test for unequal variance unexpectedly lost power with samples of uneven size. Also, non-parametric tests were strongly affected by unequal variance in large samples, yet non-parametric tests could complement parametric tests when testing samples of uneven size. Thus, we demonstrate that the robustness of each kind of test strongly depends on the combination of parameters (distribution, sample size, equality of variances). We conclude that manuals should be revised to offer more elaborate instructions for applying specific statistical tests.

Highlights

Ecologists, when analyzing the output of simple experiments, often have to compare statistical samples that simultaneously are of uneven size, unequal variance and distribute non-normally
We assess whether the special test devised for unequal variances is robust to distribution type and uneven sample size, and whether non-parametric and randomization tests are universally robust to the combinations of these unsuitable characteristics, as it is implicitly assumed
While with samples of equal variance the U-tests were slightly superior to ANOVA (Fig. 2), with unequal variance non-parametric test started to unexpectedly fail, producing very high of both type errors for large samples

Summary

Introduction

Ecologists, when analyzing the output of simple experiments, often have to compare statistical samples that simultaneously are of uneven size, unequal variance and distribute non-normally. Some recent manuals do not discuss tests for unequal variance at all (Quinn and Keough 2002, Gotelli and Ellison 2004) Another example of uncertainty is the use of non-parametric tests recommended for ‘low quality’ data such as samples of unknown distribution (Snedecor and Cochran 1980, Sokal and Rohlf 1995, Zar 1999). We assess whether the special test devised for unequal variances is robust to distribution type and uneven sample size, and whether non-parametric and randomization tests are universally robust to the combinations of these unsuitable characteristics, as it is implicitly assumed

Methods

Results

Conclusion