Combinatorics of Fulton’s essential set

Abstract

1. Introduction. An n-by-n permutation matrix can be represented by an nby-n array of squares with one dot in each row and column and all other squares empty. The diagram of a permutation matrix (defined in 1800 by Rothe) is obtained by shading every row from the dot and eastwards and shading every column from the dot and southwards. The essential set is the set of southeast corners of the connected components of the diagram. The essential set, together with a rank function, was introduced by Fulton [7] in a pioneering paper from 1992, to study the irreducible loci in spaces of matrices. Although we are not going to pursue this geometric issue much, but rather concentrate on the essential set per se, we need a short recollection of the background. Fulton studies varieties given by ideals of minors, subject to certain rank conditions, in a generic matrix. One concern is, given a prescribed rank function r(i, j), to determine if there exist matrices such that the rank of the upper-left i-by-j submatrices is r(i, j) for every position (i, j). He observes that if such matrices exist, then there is in particular some permutation matrix with this property, so it is enough to consider permutation matrices. More generally, he is interested in prescribing the ranks only for a few of the submatrices, such that all the other ranks follow from these: in other words, to find a subset of the upper-left submatrices of a permutation matrix such that the permutation is uniquely determined by their ranks. Fulton shows that the essential set of a permutation matrix is such a set; that is, the permutation is determined by its essential set and the ranks of the corresponding upper-left submatrices. In Section 2 we present more detailed definitions and the relevant results of Fulton. Our first result, in Section 3, is an elementary algorithm that retrieves a permutation from its ranked essential set. The algorithm is flexible, in that the input set may be bigger and include any other squares, as well as be smaller as long as it contains a certain of the essential set. By this flexibility, the algorithm ought to be useful as a practical tool. In general, the core is much smaller than the essential set, so this is a significant strengthening. Fulton used the essential sets mainly for vexillary (2143-avoiding) permutations. He gave a characterization of the ranked essential set for vexillary permutations, which may be expressed by three simple conditions, and he pointed out that it might be useful to have an analogous characterization of the ranked sets of squares that arise as ranked essential sets of all permutations. As an application of the algorithm, we are able to state such a characterization, which keeps