The Relationship Between FST and the Frequency of the Most Frequent Allele

Abstract

FST is frequently used as a summary of genetic differentiation among groups. It has been suggested that FST depends on the allele frequencies at a locus, as it exhibits a variety of peculiar properties related to genetic diversity: higher values for biallelic single-nucleotide polymorphisms (SNPs) than for multiallelic microsatellites, low values among high-diversity populations viewed as substantially distinct, and low values for populations that differ primarily in their profiles of rare alleles. A full mathematical understanding of the dependence of FST on allele frequencies, however, has been elusive. Here, we examine the relationship between FST and the frequency of the most frequent allele, demonstrating that the range of values that FST can take is restricted considerably by the allele-frequency distribution. For a two-population model, we derive strict bounds on FST as a function of the frequency M of the allele with highest mean frequency between the pair of populations. Using these bounds, we show that for a value of M chosen uniformly between 0 and 1 at a multiallelic locus whose number of alleles is left unspecified, the mean maximum FST is ∼0.3585. Further, FST is restricted to values much less than 1 when M is low or high, and the contribution to the maximum FST made by the most frequent allele is on average ∼0.4485. Using bounds on homozygosity that we have previously derived as functions of M, we describe strict bounds on FST in terms of the homozygosity of the total population, finding that the mean maximum FST given this homozygosity is 1 − ln 2 ≈ 0.3069. Our results provide a conceptual basis for understanding the dependence of FST on allele frequencies and genetic diversity and for interpreting the roles of these quantities in computations of FST from population-genetic data. Further, our analysis suggests that many unusual observations of FST, including the relatively low FST values in high-diversity human populations from Africa and the relatively low estimates of FST for microsatellites compared to SNPs, can be understood not as biological phenomena associated with different groups of populations or classes of markers but rather as consequences of the intrinsic mathematical dependence of FST on the properties of allele-frequency distributions.