If FST does not measure neutral genetic differentiation, then comparing it with QST is misleading. Or is it?

Abstract

The comparison between neutral genetic differentiation (F(ST) ) and quantitative genetic differentiation (Q(ST) ) is commonly used to test for signatures of selection in population divergence. However, there is an ongoing discussion about what F(ST) actually measures, even resulting in some alternative metrics to express neutral genetic differentiation. If there is a problem with F(ST) , this could have repercussions for its comparison with Q(ST) as well. We show that as the mutation rate of the neutral marker increases, F(ST) decreases: a higher within-population heterozygosity (He) yields a lower F(ST) value. However, the same is true for Q(ST) : a higher mutation rate for the underlying QTL also results in a lower Q(ST) estimate. The effect of mutation rate is equivalent in Q(ST) and F(ST) . Hence, the comparison between Q(ST) and F(ST) remains valid, if one uses neutral markers whose mutation rates are not too high compared to those of quantitative traits. Usage of highly variable neutral markers such as hypervariable microsatellites can lead to serious biases and the incorrect inference that divergent selection has acted on populations. Much of the discussion on F(ST) seems to stem from the misunderstanding that it measures the differentiation of populations, whereas it actually measures the fixation of alleles. In their capacity as measures of population differentiation, Hedrick's G'(ST) and Jost's D reach their maximum value of 1 when populations do not share alleles even when there remains variation within populations, which invalidates them for comparisons with Q(ST) .