The distribution of F st and other genetic statistics for a class of population structure models

Abstract

We examine genetic statistics used in the study of structured populations. In a 1999 paper, Wakeley observed that the coalescent process associated with the finite island model can be decomposed into a scattering phase and a collecting phase. This decomposition becomes exact in the large population limit with the coalescent at the end of the scattering phase converging to the Ewens sampling formula and the coalescent during the collecting phase converging to the Kingman coalescent. In this paper we introduce a class of limiting models, which we refer to as G/KC models, that generalize Wakeley's decomposition. G in G/KC represents a completely general limit for the scattering phase, while KC represents a Kingman coalescent limit for the collecting phase. We show that both the island and two-dimensional stepping stone models converge to G/KC models in the large population limit. We then derive the distribution of the statistic F(st) for all G/KC models under a large sample limit for the cases of strong or weak mutation, thereby deriving the large population, large sample limiting distribution of F(st) for the island and two-dimensional stepping stone models as a special case of a general formula. Our methods allow us to take the large population and large sample limits simultaneously. In the context of large population, large sample limits, we show that the variance of F(st) in the presence of weak mutation collapses as O(1/log d) where d is the number of demes sampled. Further, we show that this O(1/log d) is caused by a heavy tail in the distribution of F(st). Our analysis of F(st) can be extended to an entire class of genetic statistics, and we use our approach to examine homozygosity measures. Our analysis uses coalescent based methods.