Putting the Mantel test back together again.

Abstract

The Mantel test has been widely used in ecology and evolution, but over the last two decades it has been frequently critiqued because results were inconsistent with expectations and there were issues with Type I (false-positive) and Type II (false-negative) error rates. Three-matrix extensions of the Mantel test have been challenged for similar reasons. Even the null hypotheses underlying the Mantel test have been questioned. As a result, use of the Mantel test and its variants has been discouraged or limited to special situations. Here, we examine Mantel test criticisms including the lack of agreement between traditional variable-based Pearson correlations (r) and observation-based Mantel correlations (rm ), and the unusual Type I and Type II error rates. We propose an alternate proximity measure that resolves these issues. We use simulations and examples to contrast Mantel results based on Euclidean distance, squared Euclidean distance, and the simple difference (Diff) with traditional bivariate Pearson correlations. We demonstrate that use of the simple difference in Mantel tests can resolve the underlying problems with poor agreement between bivariate Pearson and Mantel correlations, as well as appropriate Type I and Type II errors (i.e., where r = cor(x,y) and rm = cor(dx ,dy ), if dx = Diff(x) and dy = Diff(y), r = rm ). We also show that the simple difference can provide solutions to issues with partial Mantel tests and distance-based MANOVA. Because our results resolve many of the issues with Mantel tests, we hope that these findings will restore the popularity of the Mantel test.