Bistability in two-locus models with selection, mutation, and recombination

Abstract

The evolutionary effect of recombination depends crucially on the epistatic interactions between linked loci. A paradigmatic case where recombination is known to be strongly disadvantageous is a two-locus fitness landscape displaying reciprocal sign epistasis with two fitness peaks of unequal height. Although this type of model has been studied since the 1960s, a full analytic understanding of the stationary states of mutation-selection balance was not achieved so far. Focusing on the bistability arising due to the recombination, we consider here the deterministic, haploid two-locus model with reversible mutations, selection and recombination. We find analytic formulae for the critical recombination probability r ( c ) above which two stable stationary solutions appear which are localized on each of the two fitness peaks. We also derive the stationary genotype frequencies in various parameter regimes. In particular, when the recombination rate is close to r ( c ) and the fitness difference between the two peaks is small, we obtain a compact description in terms of a cubic polynomial which is analogous to the Landau theory of physical phase transitions.