Mathematics of coalescent gene trees

Abstract

Using the coalescent process continuously until all the lineages have become one, we get different possible gene trees. In this paper I had computed the probabilities of such different gene trees that occur depending on the way lineages can form. In particular I had shown the probability of any 3 sample gene tree is 1/3 and for four samples resembling caterpillar gene trees the probability is 1/18 and probability for any balanced gene tree with 4 samples is 1/9. I have also proved a theorem providing the number of ranked gene trees. This estimate will help us to determine the probability of any gene tree.