Nilpotent orbits and some small unitary representations of indefinite orthogonal groups

Abstract

For 2⩽ m⩽ l/2, let G be a simply connected Lie group with g 0= so(2m,2l−2m) as Lie algebra, let g= k⊕ p be the complexification of the usual Cartan decomposition, let K be the analytic subgroup with Lie algebra k∩ g 0 , and let U( g) be the universal enveloping algebra of g . This work examines the unitarity and K spectrum of representations in the “analytic continuation” of discrete series of G, relating these properties to orbits in the nilpotent radical of a certain parabolic subalgebra of g . The roots with respect to the usual compact Cartan subalgebra are all ± e i ± e j with 1⩽ i< j⩽ l. In the usual positive system of roots, the simple root e m − e m+1 is noncompact and the other simple roots are compact. Let q= l⊕ u be the parabolic subalgebra of g for which e m − e m+1 contributes to u and the other simple roots contribute to l , let L be the analytic subgroup of G with Lie algebra l∩ g 0 , let L C = Int g ( l) , let 2δ( u) be the sum of the roots contributing to u , and let q ̄ = l⊕ u ̄ be the parabolic subalgebra opposite to q . The members of u∩ p are nilpotent members of g . The group L C acts on u∩ p with finitely many orbits, and the topological closure of each orbit is an irreducible algebraic variety. If Y is one of these varieties, let R( Y) be the dual coordinate ring of Y; this is a quotient of the algebra of symmetric tensors on u∩ p that carries a fully reducible representation of L C . For s∈ Z , let λ s= ∑ k=1 m (−l+ s 2 )e k . Then λ s defines a one-dimensional ( l,L) module C λ s . Extend this to a ( q ̄ ,L) module by having u ̄ act by 0, and define N(λ s+2δ( u))=U( g)⊗ q ̄ C λ s+2δ( u) . Let N′(λ s+2δ( u)) be the unique irreducible quotient of N(λ s+2δ( u)) . The representations under study are π s=Π S(N(λ s+2δ( u))) and π s′=Π S(N′(λ s+2δ( u))) , where S= dim( u∩ k) and Π S is the Sth derived Bernstein functor. For s>2 l−2, it is known that π s = π s ′ and that π s ′ is in the discrete series. Enright, Parthsarathy, Wallach, and Wolf showed for m⩽ s⩽2 l−2 that π s = π s ′ and that π s ′ is still unitary. The present paper shows that π s ′ is unitary for 0⩽ s⩽ m−1 even though π s ≠ π s ′, and it relates the K spectrum of the representations π s ′ to the representation of L C on a suitable R( Y) with Y depending on s. Use of a branching formula of D. E. Littlewood allows one to obtain an explicit multiplicity formula for each K type in π s ′; the variety Y is indispensable in the proof. The chief tools involved are an idea of B. Gross and Wallach, a geometric interpretation of Littlewood's theorem, and some estimates of norms. It is shown further that the natural invariant Hermitian form on π s ′ does not make π s ′ unitary for s<0 and that the K spectrum of π s ′ in these cases is not related in the above way to the representation of L C on any R( Y). A final section of the paper treats in similar fashion the simply connected Lie group with Lie algebra g 0= so(2m,2l−2m+1) , 2⩽ m⩽ l/2.