The star-shaped Λ-coalescent and Fleming–Viot process

Abstract

ABSTRACTThe star-shaped Λ-coalescent and corresponding Λ-Fleming–Viot process, where the Λ measure has a single atom at unity, are studied in this article. The transition functions and stationary distribution of the Λ-Fleming–Viot process are derived in a two-type model with mutation. The distribution of the number of non-mutant lines back in time in the star-shaped Λ-coalescent is found. Extensions are made to a model with d types, either with parent-independent mutation or general Markov mutation, and an infinitely-many-types model, when d → ∞. An eigenfunction expansion for the transition functions is found, which has polynomial right eigenfunctions and left eigenfunctions described by hyperfunctions. A further star-shaped model with general frequency-dependent change is considered and the stationary distribution in the Fleming–Viot process derived. This model includes a star-shaped Λ-Fleming–Viot process with mutation and selection. In a general Λ-coalescent explicit formulae for the transition functions and stationary distribution, when there is mutation, are unknown. However, in this article, explicit formulae are derived in the star-shaped coalescent.