Constant term formulas for refined enumerations of Gog and Magog trapezoids

Abstract

Gog and Magog trapezoids are certain arrays of positive integers that generalize alternating sign matrices (ASMs) and totally symmetric self-complementary plane partitions (TSSCPPs) respectively. Zeilberger used constant term formulas to prove that there is the same number of (n,k)-Gog trapezoids as there is of (n,k)-Magog trapezoids, thereby providing so far the only proof for a weak version of a conjecture by Mills, Robbins and Rumsey from 1986. About 20 years ago, Krattenthaler generalized Gog and Magog trapezoids and formulated an extension of their conjecture, and, recently, Biane and Cheballah generalized Gog trapezoids further and formulated a related conjecture. In this paper, we derive constant term formulas for various refined enumerations of generalized Gog trapezoids including those considered by Krattenthaler and by Biane and Cheballah. For this purpose we employ a result on the enumeration of truncated monotone triangles which is in turn based in the author's operator formula for the number of monotone triangles with prescribed bottom row. As a byproduct, we also generalize the operator formula for monotone triangles by including the inversion number and the complementary inversion number for ASMs. Constant term formulas as well as determinant formulas for the refined Magog trapezoid numbers that appear in Krattenthaler's conjecture are also deduced by using the classical approach based on non-intersecting lattice paths and the Lindström–Gessel–Viennot theorem. Finally, we review and partly extend a few existing tools that may be helpful in relating constant term formulas for Gogs to those for Magogs to eventually prove the above mentioned conjectures.