The effect of a barrier to gene flow on patterns of geographic variation

Abstract

Explicit formulae are given for the effects of a barrier to gene flow on random fluctuations in allele frequency; these formulae can also be seen as generating functions for the distribution of coalescence times. The formulae are derived using a continuous diffusion approximation, which is accurate over all but very small spatial scales. The continuous approximation is confirmed by comparison with the exact solution to the stepping stone model. In both one and two spatial dimensions, the variance of fluctuations in allele frequencies increases near the barrier; when the barrier is very strong, the variance doubles. However, the effect on fluctuations close to the barrier is much greater when the population is spread over two spatial dimensions than when it occupies a linear, one-dimensional habitat: barriers of strength comparable with the dispersal range (B approximately equal to sigma) can have an appreciable effect in two dimensions, whereas only barriers with strength comparable with the characteristic scale (B approximately equal to L=sigma/sqrt{2mu}) are significant in one dimension (mu is the rate of mutation or long-range dispersal). Thus, in a two-dimensional population, barriers to gene flow can be detected through their effect on the spatial pattern of genetic marker alleles.