The closure ordering of adjoint nilpotent orbits in $\mathfrak{so}(p,q)$

Abstract

Let ${\mathcal O}$ be a nilpotent orbit in $\mathfrak{so}(p,q)$ under the adjoint action of the full orthogonal group ${\rm O}(p,q)$. Then the closure of ${\mathcal O}$ (with respect to the Euclidean topology) is a union of ${\mathcal O}$ and some nilpotent ${\rm O}(p,q)$-orbits of smaller dimensions. In an earlier work, the first author has determined which nilpotent ${\rm O}(p,q)$-orbits belong to this closure. The same problem for the action of the identity component ${\rm SO}(p,q)^0$ of ${\rm O}(p,q)$ on $\mathfrak{so}(p,q)$ is much harder and we propose a conjecture describing the closures of the nilpotent ${\rm SO}(p,q)^0$-orbits. The conjecture is proved when $\min(p,q)\le7$. Our method is indirect because we use the Kostant-Sekiguchi correspondence to translate the problem to that of describing the closures of the unstable orbits for the action of the complex group ${\rm SO}_p({\bf C})\times{\rm SO}_q({\bf C})$ on the space $M_{p,q}$ of complex $p\times q$ matrices with the action given by $(a,b)\cdot x=axb^{-1}$. The fact that the Kostant--Sekiguchi correspondence preserves the closure relation has been proved recently by Barbasch and Sepanski.