Recombination, balancing selection and phylogenies in MHC and self-incompatibility genes.

Abstract

Using a coalescent model of multiallelic balancing selection with recombination, the genealogical process as a function of recombinational distance from a site under selection is investigated. We find that the shape of the phylogenetic tree is independent of the distance to the site under selection. Only the timescale changes from the value predicted by Takahata's allelic genealogy at the site under selection, converging with increasing recombination to the timescale of the neutral coalescent. However, if nucleotide sequences are simulated over a recombining region containing a site under balancing selection, a phylogenetic tree constructed while ignoring such recombination is strongly affected. This is true even for small rates of recombination. Published studies of multiallelic balancing selection, i.e., the major histocompatibility complex (MHC) of vertebrates, gametophytic and sporophytic self-incompatibility of plants, and incompatibility of fungi, all observe allelic genealogies with unexpected shapes. We conclude that small absolute levels of recombination are compatible with these observed distortions of the shape of the allelic genealogy, suggesting a possible cause of these observations. Furthermore, we illustrate that the variance in the coalescent with recombination process makes it difficult to locate sites under selection and to estimate the selection coefficient from levels of variability.