Analysis of restrictions of unitary representations of a nilpotent Lie group

Abstract

Let G be a connected simply connected nilpotent Lie group, K an analytic subgroup of G and π an irreducible unitary representation of G. Let D π ( G ) K be the algebra of differential operators keeping invariant the space of C ∞ vectors of π and commuting with the action of K on that space. In this paper, we assume that the restriction of π to K has finite multiplicities and we show that D π ( G ) K is isomorphic to a subalgebra of the field of rational K-invariant functions on the co-adjoint orbit Ω ( π ) associated to π, and for some particular cases, that D π ( G ) K is even isomorphic to the algebra of polynomial K-invariant functions on Ω ( π ) . We prove also the Frobenius reciprocity for some restricted classes of nilpotent Lie groups, especially in the cases where K is normal or abelian.