Transition between characters of classical groups, decomposition of Gelfand-Tsetlin patterns and last passage percolation

Abstract

We study the combinatorial structure of the irreducible characters of the classical groups GLn(C), SO2n+1(C), Sp2n(C), SO2n(C) and the “non-classical” odd symplectic group Sp2n+1(C), finding new connections to the probabilistic model of Last Passage Percolation (LPP). Perturbing the expressions of these characters as generating functions of Gelfand-Tsetlin patterns, we produce two families of symmetric polynomials that interpolate between characters of Sp2n(C) and SO2n+1(C) and between characters of SO2n(C) and SO2n+1(C). We identify the first family as a one-parameter specialization of Koornwinder polynomials, for which we thus provide a novel combinatorial structure; on the other hand, the second family appears to be new. We next develop a method of Gelfand-Tsetlin pattern decomposition to establish identities between all these polynomials that, in the case of irreducible characters, can be viewed as branching rules. Through these formulas we connect orthogonal and symplectic characters, and more generally the interpolating polynomials, to LPP models with various symmetries, thus going beyond the link with classical Schur polynomials originally found by Baik and Rains (Duke Math. J., 2001). Taking the scaling limit of the LPP models, we finally provide an explanation of why the Tracy-Widom GOE and GSE distributions from random matrix theory admit formulations in terms of both Fredholm determinants and Fredholm Pfaffians.