Mantel test in population genetics.

José Alexandre F Diniz-Filho,Luis Mauricio Bini,Thiago F Rangel,Jacqueline S Lima,Mariana Pires de Campos Telles,Thannya N Soares,Ricardo Dobrovolski,Victor Lemes Landeiro

doi:10.1590/s1415-47572013000400002

Abstract

The comparison of genetic divergence or genetic distances, estimated by pairwise FST and related statistics, with geographical distances by Mantel test is one of the most popular approaches to evaluate spatial processes driving population structure. There have been, however, recent criticisms and discussions on the statistical performance of the Mantel test. Simultaneously, alternative frameworks for data analyses are being proposed. Here, we review the Mantel test and its variations, including Mantel correlograms and partial correlations and regressions. For illustrative purposes, we studied spatial genetic divergence among 25 populations of Dipteryx alata (“Baru”), a tree species endemic to the Cerrado, the Brazilian savannas, based on 8 microsatellite loci. We also applied alternative methods to analyze spatial patterns in this dataset, especially a multivariate generalization of Spatial Eigenfunction Analysis based on redundancy analysis. The different approaches resulted in similar estimates of the magnitude of spatial structure in the genetic data. Furthermore, the results were expected based on previous knowledge of the ecological and evolutionary processes underlying genetic variation in this species. Our review shows that a careful application and interpretation of Mantel tests, especially Mantel correlograms, can overcome some potential statistical problems and provide a simple and useful tool for multivariate analysis of spatial patterns of genetic divergence.

Highlights

The estimation of genetic divergence between individuals from different localities (“populations” hereafter) has been an important component of empirical studies in population genetics
We review the Mantel test and its extensions (Mantel correlogram, partial correlation and regression), discussing how it can be associated with theoretical models in population genetics
Despite recent discussions and criticisms, we believe that the Mantel test can be a powerful approach to analyze multivariate data, mainly if the ecological or evolutionary hypotheses are better expressed as pairwise distances or similarities, as pointed out by Legendre and Fortin (2010)

Summary

Introduction

The estimation of genetic divergence between individuals from different localities (“populations” hereafter) has been an important component of empirical studies in population genetics. That the Mantel correlations in both the first and last distance classes were not very high (i.e., -0.33), indicating that the spatial structure is not strong (remember that the overall Mantel test is 0.499, so that only about 24.9% (i.e. 0.4992) of the genetic divergence is explained by geographic distance - see below).

Results

Conclusion