Modules for algebraic groups with finitely many orbits on totally singular 2-spaces

Abstract

This is the author's second paper treating the double coset problem for classical groups. Let G be an algebraic group over an algebraically closed field K. The double coset problem consists of classifying the pairs H,J of closed connected subgroups of G with finitely many (H,J)-double cosets in G. The critical setup occurs when H is reductive and J is a parabolic subgroup. Assume that G is a classical group, H is simple and J is a maximal parabolic Pk, the stabilizer of a totally singular k-space. We show that if there are finitely many (H,Pk)-double cosets in G, then the triple (G,H,k) belongs to a finite list of candidates. Most of these candidates have k=1 or k=2. The case k=1 was solved in [23] and here we deal with k=2. We solve this case by determining all faithful irreducible self-dual H-modules V, such that H has finitely many orbits on totally singular 2-spaces of V.