Null Alleles and FIS × FST Correlations.

Abstract

Three published papers in this journal have considered the proposition that, under a Wahlund effect caused by population mixture, a positive correlation is expected between single-locus values of FIS for a sample from the mixture and FST between the populations contributing to the mixture. Two of the papers assumed unbiased samples to estimate FST but did not consider possible effects of null alleles; the other paper focused on effects of nulls but used biased samples that also included Wahlund effects to estimate FST. The result is an information gap regarding scenarios that include null alleles but have unbiased estimates of FST. Simulations were used to fill this information gap, with the following results: 1) converting ~10% of alleles to nulls substantially reduced apparent heterozygosity and substantially increased FIS, with few exceptions; 2) adding null alleles also increased FST at most loci, although the effect was much more modest; 3) null alleles generally degraded correlations between FIS and FST, but the relationship remained relatively strong for FST ≥ 0.06; and 4) null alleles had only a small effect on correlations between r2, a measure of linkage disequilibrium between pairs of loci, and the product of FST values for those loci. These results argue for some caution in interpreting FIS × FST correlations under conditions where null alleles might be common and suggest that two-locus analyses might provide more robust assessments of Wahlund effects.