Counting 3-stack-sortable permutations

Abstract

We prove a “decomposition lemma” that allows us to count preimages of certain sets of permutations under West's stack-sorting map s. As a first application, we give a new proof of Zeilberger's formula for the number W2(n) of 2-stack-sortable permutations in Sn. Our proof generalizes, allowing us to find an algebraic equation satisfied by the generating function that counts 2-stack-sortable permutations according to length, number of descents, and number of peaks. This is also the first proof of this formula that generalizes to the setting of 3-stack-sortable permutations. Indeed, the same method allows us to obtain a recurrence relation for W3(n), the number of 3-stack-sortable permutations in Sn. Hence, we obtain the first polynomial-time algorithm for computing these numbers. We compute W3(n) for n≤174, vastly extending the 13 terms of this sequence that were known before. We also prove the first nontrivial lower bound for limn→∞⁡W3(n)1/n, showing that it is at least 8.659702. Invoking a result of Kremer, we also prove that limn→∞⁡Wt(n)1/n≥(t+1)2 for all t≥1, which we use to improve a result of Smith concerning a variant of the stack-sorting procedure. Our computations allow us to disprove a conjecture of Bóna, although we do not yet know for sure which one.In fact, we can refine our methods to obtain a recurrence for W3(n,k,p), the number of 3-stack-sortable permutations in Sn with k descents and p peaks. This allows us to gain a large amount of evidence supporting a real-rootedness conjecture of Bóna. Using part of the theory of valid hook configurations, we give a new proof of a γ-nonnegativity result of Brändén, which in turn implies an older result of Bóna. We then answer a question of the current author by producing a set A⊆S11 such that ∑σ∈s−1(A)xdes(σ) has nonreal roots. We interpret this as partial evidence against the same real-rootedness conjecture of Bóna that we found evidence supporting. Examining the parities of the numbers W3(n), we obtain strong evidence against yet another conjecture of Bóna. We end with some conjectures of our own.